But what I want to get is "my", "birthday is 1976", "15" and "17". In other words, I want the tokenizer either can give us the remaining of
the
string or can change the delimiters dynamically. Is there a way to do this?
You need to use two tokenizers, just like in the example you provided. The only thing you need to do differently is retrieve an iterator to the current position in the input string after the first tokenizer iterator (beg) has been dereferenced, and pass the iterator range [current,end) to the constructor of the tokenizer next rather than the whole string (s). You can retrieve the current iterator from a token_iterator by calling the iterator's member function base(). Here's your example again, modified to do just that: std::string s = "my birthday is 1976, 15/17"; boost::tokenizer<> tok(s); boost::tokenizer<>::iterator beg=tok.begin(); // here *beg will give us "my" typedef boost::tokenizer<boost::char_separator<char> > Boost_char_tokenizer; boost::char_separator<char> sep(",/"); Boost_char_tokenizer next(beg.base(),s.end(), sep); Boost_char_tokenizer::iterator tok_iter = next.begin(); for(; tok_iter!=next.end(); ++tok_iter){ std::cout << *tok_iter << std::endl; } // this will give us " birthday is 1976", " 15" and "17" Note that the only difference in this version is the construction of next: Boost_char_tokenizer next(beg.base(),s.end(), sep); Bjorn