[boost] [tokenizer and token_iterator] request for policy or redesign

Hi John, I vaguely recall that the standard committee rejected the tokenizer proposal because of performance problems. A thing that would boost performance a lot would be to allow the user to specify a "token match type". So I have some stupid questions: 1. Why is the three template paramters not defined as template < class TokenizerFunc = char_delimiters_separator<char>, class Type = std::string class Iterator = typename range_const_iterator<Type>::type

class tokenizer ?? 2. what is the requirement exactly on the Type parameter? 3. would it not be possible to allow typedef boost::tokenizer< fun, boost::sub_range<string> > tokenizer; ? I believe this would give a huge speedup. br Thorsten