Dave Handley wrote:
Joel de Guzman wrote:
Some more things to note: * why use lexeme_d on the lexer stage? * the comment rule can be rewritten as: '#' >> *(anychar_p - eol_p) >> eol_p; * stringLiteral can be rewriten similarly. * word can be rewritten using chsets.
I used lexeme_d because the VRML grammar uses whitespace to separate ints and floats. Without the lexeme_d, ints could merge into one.
I smell something wrong here. Lexers work at the character level and process white space appropriately. The added skip-parser will be another source of slowdown. White-space skipping should be in the lexer, not outside it.
Joel de Guzman wrote:
Sure, no problem. But watch out, Spirit-2 will have a speed improvement in the 10X-20X range over classic Spirit ;-) I already have some test cases, and this does not touch on predictive parsing and lexing.
Joel, I would be really keen to try running my VRML test case through Spirit-2. Given that I've now got the Spirit-1 parser running at about 6-7 seconds, if you get a 10x speed improvement on this file, then it could make the case for separate lexing much harder to support! I may also need a bigger file to start doing test cases on - millisecond test timings don't make for accuracy in my view :-)
See the smiley ;-) ... I was just teasing :-) I do have some rather informal test cases though, on a toy parser. Seriously, well, it's rather early to say anything really conclusive. I'm not yet sure how it would turn out in real world cases outside the toy room. I do intend to do lots of benchmarks and optimization. That's an area that's largely left for later. The initial thrust was not on speed, but on the interface and usability. We've certainly learned a lot from the start when Spirit debuted. Now it's time to rock and roll!
Joel de Guzman wrote:
Anyway, as I said, I'm very interested with your lexer. Would you be interested in a collaboration?
Yes, we would be very interested in a collaboration.
Very cool! I'll keep in touch as things progress in the Spirit-2 world. Lexing + predictive parsing: that's where I'm heading. Regards, -- Joel de Guzman http://www.boost-consulting.com http://spirit.sf.net