Robert Zeh <razeh@archelon-us.com> writes:
David Abrahams <dave@boost-consulting.com> writes:
"John R. Bandela" <jbandela@ufl.edu> writes:
Thanks for the e-mail.
The problem the char_separators not giving you the correct words and counts is a bug. It was caused by the latest changes to token_functions to speed up tokenizing non-input iterators. The version in the boost 1.31 should work (it is the one prior to the changes). I have also just fixed it in the CVS.
Thanks! All the docs still show char_delimiters_separator as the default, though.
Should we consider creating some regression tests for the tokenizer?
In status/Jamfile: test-suite tokenizer : [ run libs/tokenizer/examples.cpp <lib>../libs/test/build/boost_test_exec_monitor ] [ run libs/tokenizer/simple_example_1.cpp ] [ run libs/tokenizer/simple_example_2.cpp ] [ run libs/tokenizer/simple_example_3.cpp ] [ run libs/tokenizer/simple_example_4.cpp ] [ run libs/tokenizer/simple_example_5.cpp ] That first one, at any rate, is actually a regression test AFAICT. -- Dave Abrahams Boost Consulting www.boost-consulting.com