
Dear All I'm using the boost tokenizer to tokenise lines similar to this one: first_initial |23|1| This gives me: first initial 23 1 but what I actually want is: first_initial 23 1 Does anyone know how to get it to ignore the underscore? Thanks Paul

Hi Paul, By default the char_delimiters_separator (which is the default TokenizerFunction for token_iterator and tokenizer) splits on any punctuation, which I assume includes '_'. To explicitly specify the splitting characters, you pass a string of the separator characters into the constructor, like this: char_delimiters_separator<char> sep(false, " |"); // create token iterator or tokenizer using sep Cheers, Jeremy On 11/23/01 10:21 AM, "pjgrenyer@iee.org" <pjgrenyer@iee.org> wrote:
Dear All
I'm using the boost tokenizer to tokenise lines similar to this one:
first_initial |23|1|
This gives me:
first initial 23 1
but what I actually want is:
first_initial 23 1
Does anyone know how to get it to ignore the underscore?
Thanks Paul
-- Jeremy Siek http://www.osl.iu.edu/~jsiek Ph.D. Student, Indiana Univ. B'ton email: jsiek@osl.iu.edu C++ Booster (http://www.boost.org) office phone: (812) 855-3608

Jeremy Many thanks for this! Paul Paul Grenyer pjgrenyer@iee.org ----- Original Message ----- From: Jeremy Siek To: Boost-Users@yahoogroups.com Sent: Friday, November 23, 2001 8:53 PM Subject: Re: [Boost-Users] boost tokenizer Hi Paul, By default the char_delimiters_separator (which is the default TokenizerFunction for token_iterator and tokenizer) splits on any punctuation, which I assume includes '_'. To explicitly specify the splitting characters, you pass a string of the separator characters into the constructor, like this: char_delimiters_separator<char> sep(false, " |"); // create token iterator or tokenizer using sep Cheers, Jeremy On 11/23/01 10:21 AM, "pjgrenyer@iee.org" <pjgrenyer@iee.org> wrote:
Dear All
I'm using the boost tokenizer to tokenise lines similar to this one:
first_initial |23|1|
This gives me:
first initial 23 1
but what I actually want is:
first_initial 23 1
Does anyone know how to get it to ignore the underscore?
Thanks Paul
-- Jeremy Siek http://www.osl.iu.edu/~jsiek Ph.D. Student, Indiana Univ. B'ton email: jsiek@osl.iu.edu C++ Booster (http://www.boost.org) office phone: (812) 855-3608 Yahoo! Groups Sponsor Info: <http://www.boost.org> Wiki: <http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl> Unsubscribe: <mailto:boost-users-unsubscribe@yahoogroups.com> Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service. [Non-text portions of this message have been removed]
participants (3)
-
Jeremy Siek
-
Paul Grenyer
-
pjgrenyer@iee.org