[tokenizer] tokenizing by strings rather than chars
Hi, I am facing a problem, and I wonder if Boost.Tokenizer, or any other Boost library solves it already. I want to parse numbers separated by "new-lines", however, on my windows system, a "new-line" is represented by two consecutive characters: 13, 10 (or in other words by string "\n\r"). So, only if I encounter a sequence of these two chars do I want to cut the token. boost::char_separator appears to be a bad choice, because it only "cuts" on single chars. Is there a solution to that problem in Boost? Regards, &rzej
Andrzej Krzemienski wrote:
I want to parse numbers separated by "new-lines", however, on my windows system, a "new-line" is represented by two consecutive characters: 13, 10 (or in other words by string "\n\r"). So, only if I encounter a sequence of these two chars do I want to cut the token.
Spirit.Qi would do that handily. Something like this should work: namespace qi = boost::spirit::qi; std::vector<int> data; qi::phrase_parse(input.begin(), input.end(), qi::int_, qi::eol, data); _____ Rob Stewart robert.stewart@sig.com Software Engineer using std::disclaimer; Dev Tools & Components Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
On 05/27/2011 10:06 AM, Stewart, Robert wrote:
Andrzej Krzemienski wrote:
I want to parse numbers separated by "new-lines", however, on my windows system, a "new-line" is represented by two consecutive characters: 13, 10 (or in other words by string "\n\r"). So, only if I encounter a sequence of these two chars do I want to cut the token.
Spirit.Qi would do that handily. Something like this should work:
namespace qi = boost::spirit::qi; std::vector<int> data; qi::phrase_parse(input.begin(), input.end(), qi::int_, qi::eol, data);
_____
Small correction (missing the kleen): qi::phrase_parse(input.begin(), input.end(), *qi::int_, qi::eol, data); -- Michael Caisse Object Modeling Designs www.objectmodelingdesigns.com
Michael Caisse wrote:
On 05/27/2011 10:06 AM, Stewart, Robert wrote:
Andrzej Krzemienski wrote:
I want to parse numbers separated by "new-lines", however, on my windows system, a "new-line" is represented by two consecutive characters: 13, 10 (or in other words by string "\n\r"). So, only if I encounter a sequence of these two chars do I want to cut the token.
Spirit.Qi would do that handily. Something like this should work:
namespace qi = boost::spirit::qi; std::vector<int> data; qi::phrase_parse(input.begin(), input.end(), qi::int_, qi::eol, data);
Small correction (missing the kleen):
qi::phrase_parse(input.begin(), input.end(), *qi::int_, qi::eol, data);
Well, I suppose it would be useful to parse more than one! :) Thanks Michael. _____ Rob Stewart robert.stewart@sig.com Software Engineer using std::disclaimer; Dev Tools & Components Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
participants (3)
-
Andrzej Krzemienski -
Michael Caisse -
Stewart, Robert