[Tokenizer] No way to produce exactly one empty token

Is there any input to boost::tokenizer that will produce exactly one empty token? Consider the following code: typedef boost::char_separator<char> Separator; typedef boost::tokenizer<Separator> Tokenizer; Separator sep("|" , "", keep_empty_tokens); Tokenizer tokenizer1("", sep); Tokenizer tokenizer2("|", sep); The first case, where an empty string is tokenized, produces no tokens. The second case, where a string containing just a separator character is split, produces two empty tokens. It seems odd to me that there’s no input that will produce just one empty token. Joe Gottman

A simple answer is no. Maybe a similar issue provides another answer. A text file is a sequence of lines separated by newlines. Does this imply that text files terminated in the following manner; "People demand freedom of speech\nto make up for the freedom of thought\nwhich they avoid\n" actually contains a trailing, empty line? Some applications treat these terminations as trailing empty lines, others ignore the ambiguity and a precious subset treat them as fatal errors. It's a bummer when you get those fatal errors on a compiler upgrade and you have hundreds of source files. The boost::tokeniser had to make a policy decision which doesnt seem to match your expectation? At least it didn't throw :-) Apologies if you just wanted the code workaround. ----- Original Message ----- From: "Joe Gottman" <jgottman@carolina.rr.com> To: <boost@lists.boost.org> Sent: Wednesday, February 20, 2008 4:01 PM Subject: [boost] [Tokenizer] No way to produce exactly one empty token Is there any input to boost::tokenizer that will produce exactly one empty token? Consider the following code: typedef boost::char_separator<char> Separator; typedef boost::tokenizer<Separator> Tokenizer; Separator sep("|" , "", keep_empty_tokens); Tokenizer tokenizer1("", sep); Tokenizer tokenizer2("|", sep); The first case, where an empty string is tokenized, produces no tokens. The second case, where a string containing just a separator character is split, produces two empty tokens. It seems odd to me that there’s no input that will produce just one empty token. Joe Gottman _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
participants (2)
-
Joe Gottman
-
Scott Woods