Andrey Sverdlichenko
Saturday, July 26, 2003, 4:08:52 AM, you wrote:
JdG> Is there any advantage in using both tokenizer and spirit? I'm not a JdG> tokenizer expert, but it seems that what you are trying to achieve can JdG> be done by spirit alone:
JdG> parse(first, last, uint_p[append(numbers)], space_p);
It was just a sample. Before this numbers there is command line that i prefer to parse with tokenizer (i really have reasons for this) and after i split this string there is no way to convert tokenizer::iterator to string::iterator.
Hi, Can we move this discussion over to Spirit's ML? Spirit-general mailing list Spirit-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/spirit-general As I said, I am not a tokenizer expert. Anyway, perhaps what you want to do is to make the tokenizer output your *lexer*. Please check out the C++ lexer (written by JCAB). It provides a really good example on how to use a lexer with Spirit. The idea is to pass a *token stream* to Spirit instead of the raw character stream. For that to work, you'll probably have to do some work with the tokenizer output to assign individual tokens to the parsed input. For instance, say we have an input: "123 456 a big brown fox" your token stream might look like: struct token { char ID; // int, string... etc. Iter first; Iter last; }; You'll probably need Spirit to parse the individual lexical tokens from the tokenizer output. Pseudocode: vector <token> for each tokenizer output if parse(tokenizer[i], lex_rule).full vector.push_back(token(tokenizer[i])) else lexical error!!! Then, pass in your vector of tokens to Spirit. Be sure to have ==, != and < operators to/from your token::ID so you can write (Example): char const INT_TOK= 1; char const STR_TOK = 2; r = ch_p(INT_TOK) | STR_TOK; start_ = *r; HTH. If you have further questions, let's continue the discussion in Spirit's ML. Regards, -- Joel de Guzman joel at boost-consulting.com http://www.boost-consulting.com http://spirit.sf.net