RE: [Boost-users] Can boost::tokenizer tokenize 2byte character string?

Gennadiy Rozental wrote:

If it 2byte why are you using boost::char_separator<char>?

Shouldn't you be using boost::char_separator?

I think Lee Joo-Young is talking about multibyte strings in an encoding such as GB2312 and Shift_JIS that uses 1 or 2 bytes per character rather than wide strings that use a fixed 2 bytes per character on certain platforms (or rather, used to until UTF-16 came along). Ben.