Re: [Boost-users] Tokenizer: Low value as separator

16 Oct 2005

      On second thought there seems to be a way without changing
the interface of the library: use and escaped_list_separator instead.  
A variation of your would then become

#include <boost/tokenizer.hpp>
#include <iostream>
#include <string>

using namespace std;
using namespace boost;

int main()
{
  string str=string("X\0Y\0\0Z", 6);

  typedef tokenizer<boost::escaped_list_separator<char> > Tknz;
  typedef Tknz::iterator Iter;

  escaped_list_separator<char> sep(string(), string(1,'\0'), string());
  Tknz tok(str, sep);

  for(Iter t = tok.begin(); t != tok.end(); ++t)
    cout << '<' << *t << '>';
  cout << endl;
}

This will give:

<X><Y><><Z>

There does not seem to be a hook for getting rid of empty tokens for
escaped_list_separator.
--
Regards,
Fredrik Hedman