Andreas,
I changed the real_positions example to use a new token type carrying both, the original and the corrected positions. This makes it a bit more tricky, though, because now you have to explicitely instantiate some of the internal Wave library template types for this new token type. These explicit instantiations are normally done in the Wave library code (for the default token type).
I think this probably is the optimal solution and it is not difficult to instantiate the templates in my library although I expect it to slow down compilation considerably (?).
These need to be recompiled only if you have changed the token class, which should happen rarely.
Note: the new token type is essentially a copy of the default token type which has added the second position instance. For simplicity reasons I removed the class allocators, though. You'll have to re-add these, if needed. Please look at the file cpp_lex_token.hpp for reference).
In which instances would I need to use these class allocators? What are they used for in Wave with the standard token type?
Yes, the default token type in Wave uses a boost::pool based class allocator to improve performance. Please compare the real_positions_token and the default Wave token type in the file cpp_lex_token.hpp.
Yes. It was very easy to work with your code so that should be no problem. Are the real_positions set before or after the tokens are send to the preprocessing hooks?
The corrected position is set inside the generated_token() pp hook. IIUC correctly you're asking, whether the corrected token position will be available in other hooks as well. The answer is no. The generated_token pp hook is call at the very last point before the preprocessed token gets returned to the calling application. I see no other way to do this, because only at this point I know everything about the final token sequence. Any ideas?
An idea would be to allow calling of generated_token() from one of the other hooks. For instance, if you in rescanned_macro() know when the first macro has been fully texpanded then you can call generated_token() at that point on every token provided to rescanned_macro(). The reason why I say first macro is that a macro in it's formal definition can call another macro, so first macro calls second macro etc.
The suggestion I made above can probably also be considered an optimization (?) because you could also imagine recomputing the correct positions on every call to rescanned_macro. Do you see any reason for this not working?
If the generated_token() gets call for instance in the rescanned_macro() hook you won't be able to track the resulting position of the token because it's impossible to tell, at which expansion level inside a macro the current token got generated. Do I miss something? Regards Hartmut