tokenizer currently fails with tests example (regression) and simple_example_3 and simple_example_4 (expected) under msvc-stlport: http://tinyurl.com/3mkvx http://tinyurl.com/44pbx http://tinyurl.com/5xd8z I've tracked down the failure to a problem of MSVC 6.0 with templatized memfuns of an imported class (thanks to Francois Dumont): http://www.stlport.com/dcforum/DCForumID6/1763.html The following workaround in boost/token_functions.hpp solves the problem for all tests. As the patch is strictly msvc-stlport-specific, I don't think it can have nasty side effects somewhere else. OK to commit? Joaquín M López Muñoz Telefónica, Investigación y Desarrollo 11a12,13

// 30 Sep 2004 Joaquín M López Muñoz // Workaround for a problem with string::assign in msvc-stlport 38a41 #include <boost/detail/workaround.hpp> 210a214,224

#if BOOST_WORKAROUND(BOOST_MSVC, == 1200) &&\ BOOST_WORKAROUND(__SGI_STL_PORT, < 0x500) &&\ defined(_STLP_DEBUG) // Problem with string::assign for msvc-stlport in debug mode: the // linker is not able to import the templatized version of this memfun. // See http://www.stlport.com/dcforum/DCForumID6/1763.html for details.

t = Token(); while(b != e) t += *b++; #else 211a226,227 #endif