[Boost-users] boost::char_separator enhancement (or, how do I do it myself)

I'm teaching myself C++, and want to tokenize my stream, using boost::tokenizer. I want to be able to optionally tokenize with or without including the delimiter character - this is supported. However: the delimiter character, if included, becomes its own token. I want an option to have the delimiter character be part of the previous token (which is what makes sense for the processing I am doing on my files). ??? ta zen