Re: [Boost-users] Tokenizer Question
At 10:52 2005-06-11, Tom Browder wrote:
I have used my own C++ tokenizer in the past, but I would like to use Boost's instead.
The predominant use of tokenizing for me is to split on white space,
for tokenizing on whitespace, simple stream input (>>) to a std::string suffices. IMO, it's hardly worth troubling yourself with a tokenizer for whitespace.
but Boost's default is to use white space AND punctuation. Is there any possibility to have either the default changed, or another TokenizerFunction added such as ws_separator, or something similar?
I know I can use
boost::char_separator<char> sep(" \n\t");
(but do I need to add "\v" to the char set?)
but I would rather have something like
boost::ws_separator sep;
and, better, make the ws_separator be the default TokenizerFunction for tokenizer.
Thanks for listening.
Tom Browder
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Victor A. Wagner Jr. http://rudbek.com The five most dangerous words in the English language: "There oughta be a law"
for tokenizing on whitespace, simple stream input (>>) to a std::string suffices.
My own tokenizer does just that--and puts the tokens into a deque.
IMO, it's hardly worth troubling yourself with a tokenizer for whitespace.
Well, not really. When parsing line-oriented output and semi-known structured lines it's handy to be able to sometimes work with a line's tokens as if they were in a vector or deque. In fact, I was going to add a suggestion that the tokenizer also have the [] operator so that the individual tokens could be addressed as tok[1], etc. -Tom
participants (2)
-
Tom Browder -
Victor A. Wagner Jr.