Two tokenizer questions 1) I do not understand the constructor parameters to the char_delimits_separator class which is the default TokenizerFunction model. I assume the constructor would enable one to specify the delimiters for each token. I assume that the "returnable" parameter is a string literal of delimiters which are also returned as tokens and that "nonreturnable" is a string literal of delimiters which are not returned as tokens. I assume that the intersection of these string literals consists of all the delimiters. Given these assumptions, what is the purpose of the bool "return_delims" first parameter ? If "returnable" specifies a non-empty string literal, then why shouldn't these be the delimiters returned as tokens irregardless of the setting of "return_delims". If "returnable" specifies an empty string literal, then no delimiters will be returned as tokens once again irregardless of the setting of "return_delims". Would someone please explain why "return_delims" exists as a parameter in conjunction with the meaning of the "returnable" parameter ? It seems utterly redundant and unnecessary. 2) How does one create one own's TokenizerFunction to be plugged into the tokenizer class template as the first template type ? The documentation on TokenizerFunction concept leaves me utterly confused on a practical level. I assume I am creating a template class here despite the name which suggests it is a function, but the documentation never explains what it is. Maybe someone from Boost or John Bandela should consider updating it.