On Sunday 16 October 2005 03.58, me22 wrote:
The input string contains null characters. I want to use null character as the seperator. The following code produces only "X". Why "Y" and "Z" are discarded and how to fix this code?
string str="X\0Y\0Z";
Take a closer look at that line. The tokenizer library isn't the problem. Yes and no. Fixing the above to
str=string("X\0Y\0Z", 5); will only partly solve the problem since the char_separator ctor takes a 'const Char*' char_separator<char> sep("\0"); and passing a "\0" will treated as an empty string when the ctor initializes the private member m_dropped_delims. It is not clear to me how to best work around this. Kind of a feature of the interface plus the fact that c-strings are terminated by '\0';) One solution to support passing '\0' as a separator could be to add another ctor to char_separator that can accept a 'const string &' for kept and dropped delimiters; something like class char_separator { public: typedef std::basic_string<Char,Traits> string_type; //... explicit char_separator(const string_type & dropped_delims, const string_type & kept_delims = string_type(), empty_token_policy empty_tokens = drop_empty_tokens) : m_dropped_delims(dropped_delims), m_kept_delims(kept_delims), m_use_ispunct(false), m_use_isspace(false), m_empty_tokens(empty_tokens), m_output_done(false) { } //... seems to work when char_separator is used as char_separator<char> sep(string(1,'\0')); Tok tokens(str, sep); -- Regards, Fredrik Hedman