[spirit][qi] Dynamically combine Boost.Spirit.Qi rules at runtime

Hi, I recently tried to combine an arbitrary number of rules as alternatives in Boost.Spirit.Qi. Since rules are implemented as objects it seemed feasible to me. My motivation was to make certain parts of my grammar easily extensible. Unfortunately, I ran into some problems, so I wrote a simple program (see attachment for the complete code) demonstrating the unexpected behavior. Here's the key part: // "Dynamic" version - Does not compile! :( /* std::vector<rule_t> rules; rules.push_back(qi::char_('a')[qi::_r1 = px::val(0)]); rules.push_back(qi::char_('b')[qi::_r1 = px::val(1)]); rules.push_back(qi::char_('c')[qi::_r1 = px::val(2)]); std::vector<rule_t>::iterator i(rules.begin()), last(rules.end()); rule_t grammar; grammar = (*i)(qi::_r1); //[no_op] for(++i; i!=last; ++i) { grammar = grammar.copy() | (*i)(qi::_r1); //[no_op] } */ // "Dynamic" version - Kind of works! :-/ std::vector<rule_t> rules; rules.push_back(qi::char_('a')[qi::_r1 = px::val(0)]); rules.push_back(qi::char_('b')[qi::_r1 = px::val(1)]); rules.push_back(qi::char_('c')[qi::_r1 = px::val(2)]); std::vector<rule_t>::iterator i(rules.begin()), last(rules.end()); qi::rule<iterator_t, int()> temp; temp = (*i)(qi::_val); //[no_op] for(++i; i!=last; ++i) { temp = temp.copy() | (*i)(qi::_val); //[no_op] } rule_t grammar; grammar = temp[qi::_r1 = qi::_1]; Now, I suppose that the first version (commented out) doesn't compile, because I'm not passing the inherited attribute down to grammar.copy(). I would love to know if there's a better way of doing this than the workaround that is the second version. The second version actually compiles and it seems to do the right thing. However, once I attach a simple semantic action as indicated in the code using comments (see "[no_op]"), the behavior becomes really weird. Rather than printing 0,1,2 as before, the program prints 0,0,2. So I'm wondering, is what I'm trying to accomplish resulting in undefined behavior? Is this a bug? Or, quite possibly, am I just using something (e.g. semantic actions?) the wrong way? Thanks for reading, Daniel

I recently tried to combine an arbitrary number of rules as alternatives in Boost.Spirit.Qi. Since rules are implemented as objects it seemed feasible to me. My motivation was to make certain parts of my grammar easily extensible. Unfortunately, I ran into some problems, so I wrote a simple program (see attachment for the complete code) demonstrating the unexpected behavior. Here's the key part:
// "Dynamic" version - Does not compile! :( /* std::vector<rule_t> rules;
rules.push_back(qi::char_('a')[qi::_r1 = px::val(0)]); rules.push_back(qi::char_('b')[qi::_r1 = px::val(1)]); rules.push_back(qi::char_('c')[qi::_r1 = px::val(2)]);
std::vector<rule_t>::iterator i(rules.begin()), last(rules.end());
rule_t grammar;
grammar = (*i)(qi::_r1); //[no_op]
for(++i; i!=last; ++i) { grammar = grammar.copy() | (*i)(qi::_r1); //[no_op] } */
// "Dynamic" version - Kind of works! :-/
std::vector<rule_t> rules;
rules.push_back(qi::char_('a')[qi::_r1 = px::val(0)]); rules.push_back(qi::char_('b')[qi::_r1 = px::val(1)]); rules.push_back(qi::char_('c')[qi::_r1 = px::val(2)]);
std::vector<rule_t>::iterator i(rules.begin()), last(rules.end());
qi::rule<iterator_t, int()> temp;
temp = (*i)(qi::_val); //[no_op]
for(++i; i!=last; ++i) { temp = temp.copy() | (*i)(qi::_val); //[no_op] }
rule_t grammar;
grammar = temp[qi::_r1 = qi::_1];
Now, I suppose that the first version (commented out) doesn't compile, because I'm not passing the inherited attribute down to grammar.copy(). I would love to know if there's a better way of doing this than the workaround that is the second version.
The second version actually compiles and it seems to do the right thing. However, once I attach a simple semantic action as indicated in the code using comments (see "[no_op]"), the behavior becomes really weird. Rather than printing 0,1,2 as before, the program prints 0,0,2. So I'm wondering, is what I'm trying to accomplish resulting in undefined behavior? Is this a bug? Or, quite possibly, am I just using something (e.g. semantic actions?) the wrong way?
Just as a heads up: I saw your question on stackoverflow and I'm currently trying to find a solution for you. Regards Hartmut --------------- http://boost-spirit.com

Thank you! I posted on stackoverflow first to make sure it's not some simple beginners mistake, but since I didn't get an answer I figured it was worth a shot here. :) Looking forward to seeing what you come up with. Daniel On 5/7/2011 11:42 PM, Hartmut Kaiser wrote:
I recently tried to combine an arbitrary number of rules as alternatives in Boost.Spirit.Qi. Since rules are implemented as objects it seemed feasible to me. My motivation was to make certain parts of my grammar easily extensible. Unfortunately, I ran into some problems, so I wrote a simple program (see attachment for the complete code) demonstrating the unexpected behavior. Here's the key part:
// "Dynamic" version - Does not compile! :( /* std::vector<rule_t> rules;
rules.push_back(qi::char_('a')[qi::_r1 = px::val(0)]); rules.push_back(qi::char_('b')[qi::_r1 = px::val(1)]); rules.push_back(qi::char_('c')[qi::_r1 = px::val(2)]);
std::vector<rule_t>::iterator i(rules.begin()), last(rules.end());
rule_t grammar;
grammar = (*i)(qi::_r1); //[no_op]
for(++i; i!=last; ++i) { grammar = grammar.copy() | (*i)(qi::_r1); //[no_op] } */
// "Dynamic" version - Kind of works! :-/
std::vector<rule_t> rules;
rules.push_back(qi::char_('a')[qi::_r1 = px::val(0)]); rules.push_back(qi::char_('b')[qi::_r1 = px::val(1)]); rules.push_back(qi::char_('c')[qi::_r1 = px::val(2)]);
std::vector<rule_t>::iterator i(rules.begin()), last(rules.end());
qi::rule<iterator_t, int()> temp;
temp = (*i)(qi::_val); //[no_op]
for(++i; i!=last; ++i) { temp = temp.copy() | (*i)(qi::_val); //[no_op] }
rule_t grammar;
grammar = temp[qi::_r1 = qi::_1];
Now, I suppose that the first version (commented out) doesn't compile, because I'm not passing the inherited attribute down to grammar.copy(). I would love to know if there's a better way of doing this than the workaround that is the second version.
The second version actually compiles and it seems to do the right thing. However, once I attach a simple semantic action as indicated in the code using comments (see "[no_op]"), the behavior becomes really weird. Rather than printing 0,1,2 as before, the program prints 0,0,2. So I'm wondering, is what I'm trying to accomplish resulting in undefined behavior? Is this a bug? Or, quite possibly, am I just using something (e.g. semantic actions?) the wrong way?
Just as a heads up: I saw your question on stackoverflow and I'm currently trying to find a solution for you.
Regards Hartmut --------------- http://boost-spirit.com
_______________________________________________ Unsubscribe& other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Just to let everybody know, I found a slightly contorted version that seems to work. It uses a phoenix function to keep adding rules until there are no more left. This is the phoenix function: struct get_next_impl { template<typename F> struct result { typedef rule_t type; }; std::vector<rule_t>::iterator& i; std::vector<rule_t>::iterator& j; get_next_impl(std::vector<rule_t>::iterator& _i, std::vector<rule_t>::iterator& _j) : i(_i), j(_j) {} rule_t operator()(px::function<get_next_impl>& get_next) const { rule_t next; if(i==j) return qi::eps(false); return (*i++)(qi::_r1)[no_op] | (qi::eps[px::ref(next) = get_next(px::ref(get_next))] >> next(qi::_r1)); } }; And here's the example adapted to use it: std::vector<rule_t> rules; rules.push_back(qi::char_('a')[qi::_r1 = px::val(0)]); rules.push_back(qi::char_('b')[qi::_r1 = px::val(1)]); rules.push_back(qi::char_('c')[qi::_r1 = px::val(2)]); std::vector<rule_t>::iterator i(rules.begin()), last(rules.end()); px::function<get_next_impl> get_next = get_next_impl(i, last); rule_t grammar; rule_t next; grammar = qi::eps[px::ref(next) = get_next(px::ref(get_next))] >> next(qi::_r1); So far I haven't experienced any unexpected behavior. Best regards, Daniel On 5/7/2011 11:55 PM, Daniel F. wrote:
Thank you! I posted on stackoverflow first to make sure it's not some simple beginners mistake, but since I didn't get an answer I figured it was worth a shot here. :)
Looking forward to seeing what you come up with.
Daniel
On 5/7/2011 11:42 PM, Hartmut Kaiser wrote:
I recently tried to combine an arbitrary number of rules as alternatives in Boost.Spirit.Qi. Since rules are implemented as objects it seemed feasible to me. My motivation was to make certain parts of my grammar easily extensible. Unfortunately, I ran into some problems, so I wrote a simple program (see attachment for the complete code) demonstrating the unexpected behavior. Here's the key part:
// "Dynamic" version - Does not compile! :( /* std::vector<rule_t> rules;
rules.push_back(qi::char_('a')[qi::_r1 = px::val(0)]); rules.push_back(qi::char_('b')[qi::_r1 = px::val(1)]); rules.push_back(qi::char_('c')[qi::_r1 = px::val(2)]);
std::vector<rule_t>::iterator i(rules.begin()), last(rules.end());
rule_t grammar;
grammar = (*i)(qi::_r1); //[no_op]
for(++i; i!=last; ++i) { grammar = grammar.copy() | (*i)(qi::_r1); //[no_op] } */
// "Dynamic" version - Kind of works! :-/
std::vector<rule_t> rules;
rules.push_back(qi::char_('a')[qi::_r1 = px::val(0)]); rules.push_back(qi::char_('b')[qi::_r1 = px::val(1)]); rules.push_back(qi::char_('c')[qi::_r1 = px::val(2)]);
std::vector<rule_t>::iterator i(rules.begin()), last(rules.end());
qi::rule<iterator_t, int()> temp;
temp = (*i)(qi::_val); //[no_op]
for(++i; i!=last; ++i) { temp = temp.copy() | (*i)(qi::_val); //[no_op] }
rule_t grammar;
grammar = temp[qi::_r1 = qi::_1];
Now, I suppose that the first version (commented out) doesn't compile, because I'm not passing the inherited attribute down to grammar.copy(). I would love to know if there's a better way of doing this than the workaround that is the second version.
The second version actually compiles and it seems to do the right thing. However, once I attach a simple semantic action as indicated in the code using comments (see "[no_op]"), the behavior becomes really weird. Rather than printing 0,1,2 as before, the program prints 0,0,2. So I'm wondering, is what I'm trying to accomplish resulting in undefined behavior? Is this a bug? Or, quite possibly, am I just using something (e.g. semantic actions?) the wrong way?
Just as a heads up: I saw your question on stackoverflow and I'm currently trying to find a solution for you.
Regards Hartmut --------------- http://boost-spirit.com
_______________________________________________ Unsubscribe& other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

2011/5/11 Daniel F. <the.source@web.de>
Just to let everybody know, I found a slightly contorted version that seems to work. It uses a phoenix function to keep adding rules until there are no more left. This is the phoenix function:
struct get_next_impl { template<typename F> struct result { typedef rule_t type; };
std::vector<rule_t>::iterator& i; std::vector<rule_t>::iterator& j;
get_next_impl(std::vector<rule_t>::iterator& _i, std::vector<rule_t>::iterator& _j) : i(_i), j(_j) {}
rule_t operator()(px::function<get_next_impl>& get_next) const { rule_t next;
if(i==j) return qi::eps(false);
return (*i++)(qi::_r1)[no_op] | (qi::eps[px::ref(next) = get_next(px::ref(get_next))] >> next(qi::_r1)); } };
And here's the example adapted to use it:
std::vector<rule_t> rules;
rules.push_back(qi::char_('a')[qi::_r1 = px::val(0)]); rules.push_back(qi::char_('b')[qi::_r1 = px::val(1)]); rules.push_back(qi::char_('c')[qi::_r1 = px::val(2)]);
std::vector<rule_t>::iterator i(rules.begin()), last(rules.end());
px::function<get_next_impl> get_next = get_next_impl(i, last);
rule_t grammar; rule_t next;
grammar = qi::eps[px::ref(next) = get_next(px::ref(get_next))] >> next(qi::_r1);
So far I haven't experienced any unexpected behavior.
It's a interesting idea, I never thought like that before. But I doubt that if the "rule_t next;" living inside the function's scope could ever be passed out.

On 05/07/2011 10:37 PM, Daniel F. wrote:
Hi,
I recently tried to combine an arbitrary number of rules as alternatives in Boost.Spirit.Qi. Since rules are implemented as objects it seemed feasible to me. My motivation was to make certain parts of my grammar easily extensible. Unfortunately, I ran into some problems, so I wrote a simple program (see attachment for the complete code) demonstrating the unexpected behavior. Here's the key part:
Hi, I did something similar last year, writing a runtime-configurable date/time parser from small building blocks like day-, month- and year parsers. I used a very small portion of it to create a tutorial section for Spirit, but I guess Hartmut and Joel got lost in other work, so it never made it anywhere. Maybe it will be useful for you though? Please see the attachment and let me know :-) In this example, parsers are combined as a sequence, but combining them as alternatives is equally simple, of course. Regards, Roland

Hello Roland, that looks like an interesting way of doing it, I will keep it in mind. The reason why I'm not entirely convinced is that it requires to build another layer of grammar/rules and parsing machinery on top of Boost.Spirit.Qi. It seems to me that such a task should be possible to solve within the confines of Boost.Spirit.Qi using the rule composition functionality that is already there. Best regards, Daniel On 5/11/2011 8:37 PM, Roland Bock wrote:
On 05/07/2011 10:37 PM, Daniel F. wrote:
Hi,
I recently tried to combine an arbitrary number of rules as alternatives in Boost.Spirit.Qi. Since rules are implemented as objects it seemed feasible to me. My motivation was to make certain parts of my grammar easily extensible. Unfortunately, I ran into some problems, so I wrote a simple program (see attachment for the complete code) demonstrating the unexpected behavior. Here's the key part:
Hi,
I did something similar last year, writing a runtime-configurable date/time parser from small building blocks like day-, month- and year parsers. I used a very small portion of it to create a tutorial section for Spirit, but I guess Hartmut and Joel got lost in other work, so it never made it anywhere.
Maybe it will be useful for you though? Please see the attachment and let me know :-)
In this example, parsers are combined as a sequence, but combining them as alternatives is equally simple, of course.
Regards,
Roland
_______________________________________________ Unsubscribe& other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

On 05/11/2011 09:13 PM, Daniel F. wrote:
Hello Roland,
that looks like an interesting way of doing it, I will keep it in mind. The reason why I'm not entirely convinced is that it requires to build another layer of grammar/rules and parsing machinery on top of Boost.Spirit.Qi. It seems to me that such a task should be possible to solve within the confines of Boost.Spirit.Qi using the rule composition functionality that is already there.
Best regards, Daniel
Hi Daniel, agreed, it would be much nicer to use Qi mechanisms for this purpose, only. Regards, Roland
participants (4)
-
Daniel F.
-
Hartmut Kaiser
-
Roland Bock
-
TONGARI