
11 Jun
2005
11 Jun
'05
5:52 p.m.
I have used my own C++ tokenizer in the past, but I would like to use Boost's instead. The predominant use of tokenizing for me is to split on white space, but Boost's default is to use white space AND punctuation. Is there any possibility to have either the default changed, or another TokenizerFunction added such as ws_separator, or something similar? I know I can use boost::char_separator<char> sep(" \n\t"); (but do I need to add "\v" to the char set?) but I would rather have something like boost::ws_separator sep; and, better, make the ws_separator be the default TokenizerFunction for tokenizer. Thanks for listening. Tom Browder