Ovanes,
I have a question and may be small interface change to the token_type. Currently there is no "legal" way to check if a token type instance is empty, than using token_id operator and comparing it with T_EOI (which is not really documented, available in the header and default ctor of impl::token_data only).
Why did I need it:
I have a state machine class which preserves the look ahead. Now when I construct the state machine, the look ahead is default constructed. In particular states I have to distinguish if there is look ahead available or not and take some actions accordingly. Now I would need to rely on the following comparison:
BASEID_FROM_TOKEN(T_EOI)==tok.operator token_id();
Instead it would be more flexible to write tok.empty(). I am not sure if this convinces you, but at least T_EOI should be documented (I am still using old documentation 1.33.1). The docs should also state that empty token has this type. On the other hand I don't know where else T_EOI is used and how to distinguishe this case then. I would appreciate the empty member function, which can greatly ease the life ;)
I added an 'is_eoi()' function to the token types and will change the documentation soon. It's in the Boost CVS::HEAD. HTH Regards Hartmut