[wave] Some bugs (posted here, 'coz SF bug-tracking is currently down)
Hello, I've built a spirit grammer to process the token sequence of wave. 1. Scope missing in macros. The first problem arose when I tried to write the wave token equivalent of chlit: struct token_lit : public boost::spirit::char_parser<token_lit> { token_lit(boost::wave::token_id id_) : id(id_) {} template <typename T> bool test(T id_) const { return BASE_TOKEN(id_) == BASE_TOKEN(id); } boost::wave::token_id id; }; The code above doesn't compile, because BASE_TOKEN uses constants from boost::wave without explicit scope. Lines to fix: token_ids.hpp(37-40,305,309). 2. Newline token reports bad position. For debug purposes I reported the match positions in the grammer actions using the file_position interface of the wave token type. The results were quite strange, sometimes pointing into the middle of a keyword, so I decided to take a smaller step and modified the wave example (the one in the documentation) just to see what happens, as follows: while (first != last) { std::cout << "value: \"" << (*first).get_value() << "\" pos: " << (*first).get_position ().get_file () << "(" << (*first).get_position ().get_line () << ") @" << (*first).get_position ().get_column () << std::endl; ++first; } Attached you can find some very small inputs and their corresponding outputs. 2.1. A file containing only whites and comments produced no tokens at all 2.2. Position of the enter token reports the new line number but the old column 2.3. The #line directive affects the position in a very strange way Please clarify if some of these are not errors. Oops, I almost forgot, I am using a fresh verison from the CVS repository already containing the fix for the SF bug #1274584. Best regards, József Mihalicza
Mihalicza, Jozsef wrote:
I've built a spirit grammer to process the token sequence of wave.
1. Scope missing in macros. The first problem arose when I tried to write the wave token equivalent of chlit: struct token_lit : public boost::spirit::char_parser<token_lit> { token_lit(boost::wave::token_id id_) : id(id_) {} template <typename T> bool test(T id_) const { return BASE_TOKEN(id_) == BASE_TOKEN(id); } boost::wave::token_id id; }; The code above doesn't compile, because BASE_TOKEN uses constants from boost::wave without explicit scope. Lines to fix: token_ids.hpp(37-40,305,309).
That's fixed now (Boost CVS HEAD and BOOST_RC_1_33_0 branches). The wave code itself always uses "using namespace boost::wave;" directives. For this reason this bug never got recognised.
2. Newline token reports bad position. For debug purposes I reported the match positions in the grammer actions using the file_position interface of the wave token type. The results were quite strange, sometimes pointing into the middle of a keyword, so I decided to take a smaller step and modified the wave example (the one in the documentation) just to see what happens, as follows: while (first != last) { std::cout << "value: \"" << (*first).get_value() << "\" pos: " << (*first).get_position ().get_file () << "(" << (*first).get_position ().get_line () << ") @" << (*first).get_position ().get_column () << std::endl; ++first; } Attached you can find some very small inputs and their corresponding outputs. 2.1. A file containing only whites and comments produced no tokens at all 2.2. Position of the enter token reports the new line number but the old column 2.3. The #line directive affects the position in a very strange way
Please clarify if some of these are not errors.
Oops, I almost forgot, I am using a fresh verison from the CVS repository already containing the fix for the SF bug #1274584.
Working on that, stay tuned... Thanks for reporting these problems! Regards Hartmut
Mihalicza, Jozsef wrote:
2.1. A file containing only whites and comments produced no tokens at all
That's not an error. By default Wave does not emit comment tokens. Whitespace tokens are emmitted only if necessary. To get the comment tokens as well you'll have to enable the 'preserve comments' mode which can be done by calling: ctx.set_language(boost::wave::enable_preserve_comments(ctx.get_language())); after the creation of the context object, but before calling ctx.begin(). BTW: I've now enabled the preserved mode in the cpp_tokens sample, which does exactly what you wanted to have when adding your token tracing code to the quick_start application from the docs. This should help you in correcting your code accordingly. But please note, this sample uses the Slex lexer by default. I'd like to get notified if you get different behaviour from this sample when compared to the output generated by the Re2C lexer.
2.2. Position of the enter token reports the new line number but the old column
The newline now reports the correct line number. This was a problem when using the Re2C generated lexer only. The Slex generated lexer did not show this error.
2.3. The #line directive affects the position in a very strange way
This problem seems to be fixed now as well. Please verify if it works for you. Generally the line numbers inside the generated tokens were somewhat bogus, which has been fixed now. Please consider any irregularities in this regard as bugs. Thanks for reporting the problems! Regards Hartmut
participants (2)
-
Hartmut Kaiser -
Mihalicza, Jozsef