I have a question about boost::tokenizer. I have a Semicolon separated string, something like: "All;;;;;;Bus;;;;;;BRT;;;;;;All" When I use boost tokenizer, I get "All", "Bus", "BRT",and "All". I want to get the the empty string between the semicolons, seems like the tokenizer takes consecutive semicolons as a single token; is there any way I can change this to represent different strings? boost::char_separator<char> sep(";"); typedef boost::tokenizer<boost::char_separator<char> > tokenizer; tokenizer Id_tokens(IdString, sep); for(tokenizer::iterator Id_tok_iter= Id_tokens.begin(), End_Id_tok_iter = Id_tokens.end(); Id_tok_Iter != End_Id_tok_iter) { //Do Something } Another question, is there some way to assign a different string once a tokenizer is constructed? tokenizer Id_tokens(IdString,sep); Id_tokens.assign(NewString,sep); //Is this possible with the boost tokenizer? I couldn't find any examples to this end in the places I searched. Thank you, Vijay
In the constructor of the char_separator, you need to specify keep_empty_tokens. See http://www.boost.org/libs/tokenizer/char_separator.htm On Thu, 4 Apr 2002, macrodisk wrote: macrod> I have a question about boost::tokenizer. I have a Semicolon separated macrod> string, something like: macrod> "All;;;;;;Bus;;;;;;BRT;;;;;;All" macrod> When I use boost tokenizer, I get "All", "Bus", "BRT",and "All". I macrod> want to get the the empty string between the semicolons, seems like macrod> the tokenizer takes consecutive semicolons as a single token; is macrod> there any way I can change this to represent different strings? macrod> macrod> boost::char_separator<char> sep(";"); macrod> typedef boost::tokenizer<boost::char_separator<char> > macrod> tokenizer; macrod> tokenizer Id_tokens(IdString, sep); macrod> for(tokenizer::iterator Id_tok_iter= Id_tokens.begin(), macrod> End_Id_tok_iter = macrod> Id_tokens.end(); macrod> Id_tok_Iter != End_Id_tok_iter) macrod> { macrod> //Do Something macrod> } macrod> macrod> Another question, is there some way to assign a different string once macrod> a tokenizer is constructed? macrod> macrod> tokenizer Id_tokens(IdString,sep); macrod> Id_tokens.assign(NewString,sep); //Is this possible with the boost macrod> tokenizer? macrod> macrod> I couldn't find any examples to this end in the places I searched. macrod> macrod> Thank you, macrod> macrod> Vijay macrod> macrod> macrod> macrod> macrod> macrod> Info: <http://www.boost.org> macrod> Wiki: <http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl> macrod> Unsubscribe: <mailto:boost-users-unsubscribe@yahoogroups.com> macrod> macrod> macrod> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ macrod> macrod> ---------------------------------------------------------------------- Jeremy Siek http://php.indiana.edu/~jsiek/ Ph.D. Student, Indiana Univ. B'ton email: jsiek@osl.iu.edu C++ Booster (http://www.boost.org) office phone: (812) 855-3608 ----------------------------------------------------------------------
Thank you for your reply. The following code gives me the error in Borland compiler: boost::char_separator<char> sep(";", boost::keep_empty_tokens); [C++ Error] CVLZOut1.cpp(23): E2285 Could not find a match for 'boost::char_separator<char,char_traits<char> >::char_separator (char *,boost::empty_token_policy)' but this works fine: boost::char_separator<char> sep(";", "|", boost::keep_empty_tokens); I had expected the first version to work too, since boost::char_separator<char> sep(";"); works fine and I just want to add keep_empty_tokens to this. Thanks again, Vijay --- In Boost-Users@y..., Jeremy Siek <jsiek@c...> wrote:
In the constructor of the char_separator, you need to specify keep_empty_tokens. See
Hi Vijay, On Fri, 5 Apr 2002, macrodisk wrote: macrod> Thank you for your reply. The following code gives me the error in macrod> Borland compiler: macrod> macrod> boost::char_separator<char> sep(";", boost::keep_empty_tokens); macrod> macrod> [C++ Error] CVLZOut1.cpp(23): E2285 Could not find a match macrod> for 'boost::char_separator<char,char_traits<char> >::char_separator macrod> (char *,boost::empty_token_policy)' That is right, it should give an error. The above is incorrect usage. macrod> but this works fine: macrod> boost::char_separator<char> sep(";", "|", boost::keep_empty_tokens); macrod> macrod> I had expected the first version to work too, since macrod> boost::char_separator<char> sep(";"); macrod> works fine and I just want to add keep_empty_tokens to this. Sorry. The way default arguments work in C++ is not the most ideal. There is no particularly good way to handle this kind of thing. Ciao, Jeremy ---------------------------------------------------------------------- Jeremy Siek http://php.indiana.edu/~jsiek/ Ph.D. Student, Indiana Univ. B'ton email: jsiek@osl.iu.edu C++ Booster (http://www.boost.org) office phone: (812) 855-3608 ----------------------------------------------------------------------
One minor thing to point out: macrodisk wrote:
boost::char_separator<char> sep(";", boost::keep_empty_tokens); boost::char_separator<char> sep(";", "|", boost::keep_empty_tokens);
Even if C++ default arguments worked as you liked (which they don't), those two are not equivalent. The actual default argument for kept_delims is 0 (at least in 1.27, perhaps it's different in older versions since the documentation states the default is ""). What you should be using is: boost::char_separator<char> sep(";", 0, boost::keep_empty_tokens); moo ------------------------------------------------------------ R Samuel Klatchko - Principal Software Jester Brightmail Inc - rsk@brightmail.com
participants (3)
-
Jeremy Siek
-
macrodisk
-
Samuel Klatchko