[boost] Re: Tokenizer doc problems/confusion (?)

David Abrahams <dave@boost-consulting.com> writes:

"John R. Bandela" <jbandela@ufl.edu> writes:

...

Thanks for the e-mail.

The problem the char_separators not giving you the correct words and counts is a bug. It was caused by the latest changes to token_functions to speed up tokenizing non-input iterators. The version in the boost 1.31 should work (it is the one prior to the changes). I have also just fixed it in the CVS.

Thanks! All the docs still show char_delimiters_separator as the default, though.

Should we consider creating some regression tests for the tokenizer? Robert