The boost Tokenizer package: split into a vector of vectors of tokens

Can the boost Tokenizer package split a string into into a series of serieses of tokens (a vector of vectors of tokens)? ==================== For instance, string s = "ab cd \nxy z\n123"; vector<vector<string> > v; Output should be as follows: v[0][0] = "ab"; v[0][1] = "cd"; v[1][0] = "xy"; v[1][1] = "z"; v[2][0] = "123"; ==================== http://boost.org/libs/tokenizer/introduc.htm doesn't contain such a sample. Alex Vinokur email: alex DOT vinokur AT gmail DOT com http://mathforum.org/library/view/10978.html http://sourceforge.net/users/alexvn

Hi, You might be able to do it using the string_algo find_iterator. #include <boost/algorithm/string.hpp> using namespace std; using namespace boost; ... typedef find_iterator<string::iterator> string_find_iterator; string_find_iterator It=make_find_iterator(str1, token_finder(is_space())); for(int i=0; i<maxi; i++) for(int j=0; j<maxj; j++) { if(It!=string_find_iterator()) { v[i][j]=*It; } else { break; } } Regards, Pavol On Fri, Dec 23, 2005 at 07:30:48PM +0200, Alex Vinokur wrote:
Can the boost Tokenizer package split a string into into a series of serieses of tokens (a vector of vectors of tokens)?
==================== For instance, string s = "ab cd \nxy z\n123";
vector<vector<string> > v;
Output should be as follows: v[0][0] = "ab"; v[0][1] = "cd";
v[1][0] = "xy"; v[1][1] = "z";
v[2][0] = "123"; ====================
http://boost.org/libs/tokenizer/introduc.htm doesn't contain such a sample.
Alex Vinokur email: alex DOT vinokur AT gmail DOT com http://mathforum.org/library/view/10978.html http://sourceforge.net/users/alexvn
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Just after the sending I have realize a typo. Instead of this: typedef find_iterator<string::iterator> string_find_iterator; use this: typedef split_iterator<string::iterator> string_find_iterator; On Fri, Dec 23, 2005 at 07:04:08PM +0100, Pavol Droba wrote:
Hi,
You might be able to do it using the string_algo find_iterator.
#include <boost/algorithm/string.hpp>
using namespace std; using namespace boost;
...
typedef find_iterator<string::iterator> string_find_iterator; string_find_iterator It=make_find_iterator(str1, token_finder(is_space()));
for(int i=0; i<maxi; i++) for(int j=0; j<maxj; j++) { if(It!=string_find_iterator()) { v[i][j]=*It; } else { break; } }
Regards, Pavol
On Fri, Dec 23, 2005 at 07:30:48PM +0200, Alex Vinokur wrote:
Can the boost Tokenizer package split a string into into a series of serieses of tokens (a vector of vectors of tokens)?
==================== For instance, string s = "ab cd \nxy z\n123";
vector<vector<string> > v;
Output should be as follows: v[0][0] = "ab"; v[0][1] = "cd";
v[1][0] = "xy"; v[1][1] = "z";
v[2][0] = "123"; ====================
http://boost.org/libs/tokenizer/introduc.htm doesn't contain such a sample.
Alex Vinokur email: alex DOT vinokur AT gmail DOT com http://mathforum.org/library/view/10978.html http://sourceforge.net/users/alexvn
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

"Pavol Droba" <droba@topmail.sk> wrote in message news:20051223180408.GP15407@lenin.felcer.sk...
Hi,
You might be able to do it using the string_algo find_iterator.
#include <boost/algorithm/string.hpp>
using namespace std; using namespace boost;
...
[snip] typedef split_iterator<string::iterator> string_find_iterator; ------------------------------------------------------------------------------
string_find_iterator It=make_find_iterator(str1, token_finder(is_space()));
That line produces the following error message: error: conversion from `boost::algorithm::find_iterator<__gnu_cxx::__normal_iterator<char*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >' to non-scalar type `boost::algorithm::split_iterator<__gnu_cxx::__normal_iterator<char*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >' requested g++ - 3.4.4-1 boost - 1.33.1-1 ------------------------------------------------------------------------------
for(int i=0; i<maxi; i++) for(int j=0; j<maxj; j++) { if(It!=string_find_iterator()) { v[i][j]=*It; } else { break; } }
[snip] Regards, -- Alex Vinokur email: alex DOT vinokur AT gmail DOT com http://mathforum.org/library/view/10978.html http://sourceforge.net/users/alexvn

On Sat, Dec 24, 2005 at 09:06:51AM +0200, Alex Vinokur wrote:
"Pavol Droba" <droba@topmail.sk> wrote in message news:20051223180408.GP15407@lenin.felcer.sk...
Hi,
You might be able to do it using the string_algo find_iterator.
#include <boost/algorithm/string.hpp>
using namespace std; using namespace boost;
...
[snip] typedef split_iterator<string::iterator> string_find_iterator; ------------------------------------------------------------------------------
string_find_iterator It=make_find_iterator(str1, token_finder(is_space()));
That line produces the following error message:
error: conversion from `boost::algorithm::find_iterator<__gnu_cxx::__normal_iterator<char*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >' to non-scalar type `boost::algorithm::split_iterator<__gnu_cxx::__normal_iterator<char*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >' requested
Sorry, I have fixed one error, but the other line was tied to it. the second line should be string_find_iterator It=make_split_iterator(str1, token_finder(is_space())); Regards, Pavol

Hi Alex,
Can the boost Tokenizer package split a string into into a series of serieses of tokens (a vector of vectors of tokens)?
As per the current implementation of tokenizer it does not provide this feature. It does not store the tokenized values in a container. It provides an iterator concept to iterate through the tokenized values. Thank you, Nitin Motgi On 12/23/05, Alex Vinokur <alexvn@connect.to> wrote:
Can the boost Tokenizer package split a string into into a series of serieses of tokens (a vector of vectors of tokens)?
==================== For instance, string s = "ab cd \nxy z\n123";
vector<vector<string> > v;
Output should be as follows: v[0][0] = "ab"; v[0][1] = "cd";
v[1][0] = "xy"; v[1][1] = "z";
v[2][0] = "123"; ====================
http://boost.org/libs/tokenizer/introduc.htm doesn't contain such a sample.
Alex Vinokur email: alex DOT vinokur AT gmail DOT com http://mathforum.org/library/view/10978.html http://sourceforge.net/users/alexvn
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- Nitin Motgi NITIN . DOT. MOTGI .AT. GMAIL .DOT. COM
participants (3)
-
Alex Vinokur
-
nitin motgi
-
Pavol Droba