[Boost-users] Tokenizer: request for another ctor for char_separator

19 Oct 2005

      Hello,

it seems to me that boost::char_separator needs to have another ctor that can 
accept delimiters that are string types.  For example, given a 
std::string("X\0Y\0\0Z", 6), it does not seem to be possible to use the 
current ctor of boost::char_separator so that '\0' can be used as a 
separator.  

It is possible to use boost::escaped_list_separator, since *it* takes a string 
type on construction, but on the other hand  boost::escaped_list_separator 
does not have an empty_token_policy.  In summary, I suggest adding another 
ctor to  boost::char_separator.  This will enable the parsing of the above 
string as 

   <X><Y><><Z>

or

   <X><Y><Z>

by the following:

#include <boost/tokenizer.hpp>
#include <iostream>
#include <string>

using namespace std;
using namespace boost;

int main()
{
  typedef tokenizer<boost::char_separator<char> > Tok;
  string str=string("X\0Y\0\0Z", 6);

  {
    char_separator<char> keep_empty(string(1,'\0'), string(),
				    boost::keep_empty_tokens);
    Tok tokens(str, keep_empty);
    for(Tok::iterator iter = tokens.begin(); iter != tokens.end(); ++iter)
      cout << '<' << *iter << '>';
    cout << endl;
  }
  {
    char_separator<char> skip_empty(string(1,'\0'), string(),
				    boost::drop_empty_tokens);
    Tok tokens(str, skip_empty);
    for(Tok::iterator iter = tokens.begin(); iter != tokens.end(); ++iter)
      cout << '<' << *iter << '>';
    cout << endl;
  }
}

This is achieved by a small change to class char_separator in 
token_functions.hpp included in the attachment.

--
Best Regards,
Fredrik Hedman