Hartmut Kaiser wrote:
Yes, I agree. Wave should be rewritten (and hopefully will be rewritten) in a layered way cleanly implementing every of the mandated translation phases each on top of the previous. But I'm inclined to expose iterator interfaces, not generators - to allow for rewinds, error handling etc.
Yes !! Here is another use case worth considering (while we are at it ;-) ): Imagine a C++ parser interfacing with such a backtracking lexer. It often needs to parse tentatively, rolling back to a previous state if the tentative was not successfull. However, some parts of the tentative may have been successfull nevertheless, such as the parsing of a nested-name-specifier. It would be very nice if the parser could somehow mark up such a token sequence by injecting synthetic tokens back into the lexer so it doesn't need to parse them again. A similar use case is described in the 'Decorating tokens...' paper (http://www.cs.clemson.edu/~malloy/papers/papers.html) where the parsing requires some (non strictly layered) interaction between parser, symbol lookup, and lexer. Just some food for thought... Regards, Stefan