Hello,
it seems to me that boost::char_separator needs to have another ctor that can
accept delimiters that are string types. For example, given a
std::string("X\0Y\0\0Z", 6), it does not seem to be possible to use the
current ctor of boost::char_separator so that '\0' can be used as a
separator.
It is possible to use boost::escaped_list_separator, since *it* takes a string
type on construction, but on the other hand boost::escaped_list_separator
does not have an empty_token_policy. In summary, I suggest adding another
ctor to boost::char_separator. This will enable the parsing of the above
string as
<X><Y><><Z>
or
<X><Y><Z>
by the following:
#include
#include <iostream>
#include <string>
using namespace std;
using namespace boost;
int main()
{
typedef tokenizer Tok;
string str=string("X\0Y\0\0Z", 6);
{
char_separator<char> keep_empty(string(1,'\0'), string(),
boost::keep_empty_tokens);
Tok tokens(str, keep_empty);
for(Tok::iterator iter = tokens.begin(); iter != tokens.end(); ++iter)
cout << '<' << *iter << '>';
cout << endl;
}
{
char_separator<char> skip_empty(string(1,'\0'), string(),
boost::drop_empty_tokens);
Tok tokens(str, skip_empty);
for(Tok::iterator iter = tokens.begin(); iter != tokens.end(); ++iter)
cout << '<' << *iter << '>';
cout << endl;
}
}
This is achieved by a small change to class char_separator in
token_functions.hpp included in the attachment.
--
Best Regards,
Fredrik Hedman
