[spirit] how to get_position() from a multi_pass<position_iterator <const char*> >
Hi,
I use a position_iterator
I use a position_iterator
for parsing with boost::spirit. Because I need the regex_p feature, I have to wrap this iterator using a multi_pass iterator.
I call pt_parse and get a
boost::spirit::tree_parse_info
>, ...> parse_info. Now I have problems calling the get_position() method on "parse_info.stop".
I guess this is the case because multi_pass does not store my iterator but wraps it using boost::iterator which loses the position features of position_iterators and only keeps the const char* pointer.
Do you have an advice for me?
It's difficult to tell from here as you didn't provide too much information. Would you mind posting a small example exposing your problem? Regards Hartmut
On Thu, Aug 20, 2009 at 22:42, Hartmut Kaiser
I use a position_iterator
for parsing with boost::spirit. Because I need the regex_p feature, I have to wrap this iterator using a multi_pass iterator.
I call pt_parse and get a
boost::spirit::tree_parse_info
>, ...> parse_info. Now I have problems calling the get_position() method on "parse_info.stop".
I guess this is the case because multi_pass does not store my iterator but wraps it using boost::iterator which loses the position features of position_iterators and only keeps the const char* pointer.
Do you have an advice for me?
It's difficult to tell from here as you didn't provide too much information. Would you mind posting a small example exposing your problem?
I have attached a minimal example. If you do not define MULTIPASSDBG, it builds but you are not allowed to use regex_p in the parser. If you define it, you are allowed to use regex_p, but the get_position does no longer compile and so the iterator becomes a bit useless. I read something about a "base()" method for wrapping iterators, but could not find one in the multi_pass sources. Thanks and regards, Peter
I use a position_iterator
for parsing with boost::spirit. Because I need the regex_p feature, I have to wrap this iterator using a multi_pass iterator.
I call pt_parse and get a
boost::spirit::tree_parse_info
>, ...> parse_info. Now I have problems calling the get_position() method on "parse_info.stop".
I guess this is the case because multi_pass does not store my iterator but wraps it using boost::iterator which loses the position features of position_iterators and only keeps the const char*
On Thu, Aug 20, 2009 at 22:42, Hartmut Kaiser
wrote: pointer. Do you have an advice for me?
It's difficult to tell from here as you didn't provide too much information. Would you mind posting a small example exposing your problem?
I have attached a minimal example.
If you do not define MULTIPASSDBG, it builds but you are not allowed to use regex_p in the parser.
If you define it, you are allowed to use regex_p, but the get_position does no longer compile and so the iterator becomes a bit useless.
I read something about a "base()" method for wrapping iterators, but could not find one in the multi_pass sources.
Ok, I understand. But I think your whole approach is not viable. Regex_p requires a bi-directional iterator, while multi_pass exposes an input_iterator only. So generally, even if it compiled it wouldn't work in the end. But to answer your original question, multi_pass doesn't allow to access the base iterator, which might be an oversight. I'll have a closer look and add it to the multi_pass iterator if possible. Regards Hartmut PS: I don't know how long you are using the Spirit.Classic AST/PT code already or whether this is a new project you just started to develop. In any case I strongly suggest to consider abandoning the Spirit.Classic tree parsers and switching to Spirit V2.1 (Boost SVN trunk). This is an interface breaking change, but overall and in the long run preferable. The Spirit.Classic tree code is known to be slow and buggy and I doubt it will be fixed. Spirit 2.1 is much faster, more stable and generally easier to use.
On Fri, Aug 21, 2009 at 23:37, Hartmut Kaiser
PS: I don't know how long you are using the Spirit.Classic AST/PT code already or whether this is a new project you just started to develop. In any case I strongly suggest to consider abandoning the Spirit.Classic tree parsers and switching to Spirit V2.1 (Boost SVN trunk). This is an interface breaking change, but overall and in the long run preferable. The Spirit.Classic tree code is known to be slow and buggy and I doubt it will be fixed. Spirit 2.1 is much faster, more stable and generally easier to use.
I know, I saw the announcement of V2 on the Website, but the documentation only is about the classic Spirit, so I used the old one. ( http://www.boost.org/doc/libs/1_39_0/libs/spirit/index.html automatically redirects to the classic documentation ) Once there is a documentation I will certainly consider switching to V2. But even classic is preferrable to flex/bison trouble with C++ ;) All the Best, Peter
AMDG Peter Schueller wrote:
On Fri, Aug 21, 2009 at 23:37, Hartmut Kaiser
wrote: PS: I don't know how long you are using the Spirit.Classic AST/PT code already or whether this is a new project you just started to develop. In any case I strongly suggest to consider abandoning the Spirit.Classic tree parsers and switching to Spirit V2.1 (Boost SVN trunk). This is an interface breaking change, but overall and in the long run preferable. The Spirit.Classic tree code is known to be slow and buggy and I doubt it will be fixed. Spirit 2.1 is much faster, more stable and generally easier to use.
I know, I saw the announcement of V2 on the Website, but the documentation only is about the classic Spirit, so I used the old one.
( http://www.boost.org/doc/libs/1_39_0/libs/spirit/index.html automatically redirects to the classic documentation )
Once there is a documentation I will certainly consider switching to V2.
There is documentation for V2. It's just hidden. http://www.boost.org/libs/spirit/doc/html In Christ, Steven Watanabe
On Fri, Aug 21, 2009 at 23:37, Hartmut Kaiser
wrote: PS: I don't know how long you are using the Spirit.Classic AST/PT code already or whether this is a new project you just started to develop. In any case I strongly suggest to consider abandoning the Spirit.Classic tree parsers and switching to Spirit V2.1 (Boost SVN
Peter Schueller wrote: trunk). This is an interface breaking change, but overall and in the long run preferable. The Spirit.Classic tree code is known to be slow and buggy and I doubt it will be fixed. Spirit 2.1 is much faster, more stable and generally easier to use.
I know, I saw the announcement of V2 on the Website, but the documentation only is about the classic Spirit, so I used the old one.
( http://www.boost.org/doc/libs/1_39_0/libs/spirit/index.html automatically redirects to the classic documentation )
Once there is a documentation I will certainly consider switching to V2.
There is documentation for V2. It's just hidden. http://www.boost.org/libs/spirit/doc/html
This is the documentation for Spirit V2.0. We will release Spirit V2.1 with Boost V1.41, so I would encourage to look here instead: http://tinyurl.com/ojalum (which is still work in progress, but nevertheless much more complete than the docs mentioned my Steven). Regards Hartmut
Steven Watanabe wrote:
AMDG
Peter Schueller wrote:
On Fri, Aug 21, 2009 at 23:37, Hartmut Kaiser
wrote: PS: I don't know how long you are using the Spirit.Classic AST/PT code already or whether this is a new project you just started to develop. In any case I strongly suggest to consider abandoning the Spirit.Classic tree parsers and switching to Spirit V2.1 (Boost SVN trunk). This is an interface breaking change, but overall and in the long run preferable. The Spirit.Classic tree code is known to be slow and buggy and I doubt it will be fixed. Spirit 2.1 is much faster, more stable and generally easier to use.
I know, I saw the announcement of V2 on the Website, but the documentation only is about the classic Spirit, so I used the old one.
( http://www.boost.org/doc/libs/1_39_0/libs/spirit/index.html automatically redirects to the classic documentation )
Once there is a documentation I will certainly consider switching to V2.
There is documentation for V2. It's just hidden. http://www.boost.org/libs/spirit/doc/html
And the point of hiding the Spirit 2.0 docs is.... ?
On Fri, Aug 21, 2009 at 23:37, Hartmut Kaiser
wrote: PS: I don't know how long you are using the Spirit.Classic AST/PT code already or whether this is a new project you just started to develop. In any case I strongly suggest to consider abandoning the Spirit.Classic tree parsers and switching to Spirit V2.1 (Boost SVN trunk). This is an interface breaking change, but overall and in
Peter Schueller wrote: the
long run preferable. The Spirit.Classic tree code is known to be slow and buggy and I doubt it will be fixed. Spirit 2.1 is much faster, more stable and generally easier to use.
I know, I saw the announcement of V2 on the Website, but the documentation only is about the classic Spirit, so I used the old one.
( http://www.boost.org/doc/libs/1_39_0/libs/spirit/index.html automatically redirects to the classic documentation )
Once there is a documentation I will certainly consider switching to V2.
There is documentation for V2. It's just hidden. http://www.boost.org/libs/spirit/doc/html
And the point of hiding the Spirit 2.0 docs is.... ?
We consider Spirit V2 to be a beta version and the docs themselves are not complete. This will be rectified with Boost V1.41, where the docs will be complete and Spirit V2.1 will officially replace Spirit.Classic (whose docs will be 'hidden' from that point on). Regards Hartmut
Hartmut Kaiser wrote:
On Fri, Aug 21, 2009 at 23:37, Hartmut Kaiser
wrote: PS: I don't know how long you are using the Spirit.Classic AST/PT code already or whether this is a new project you just started to develop. In any case I strongly suggest to consider abandoning the Spirit.Classic tree parsers and switching to Spirit V2.1 (Boost SVN trunk). This is an interface breaking change, but overall and in
Peter Schueller wrote: the
long run preferable. The Spirit.Classic tree code is known to be slow and buggy and I doubt it will be fixed. Spirit 2.1 is much faster, more stable and generally easier to use.
I know, I saw the announcement of V2 on the Website, but the documentation only is about the classic Spirit, so I used the old one. ( http://www.boost.org/doc/libs/1_39_0/libs/spirit/index.html automatically redirects to the classic documentation )
Once there is a documentation I will certainly consider switching to V2. There is documentation for V2. It's just hidden. http://www.boost.org/libs/spirit/doc/html And the point of hiding the Spirit 2.0 docs is.... ?
We consider Spirit V2 to be a beta version and the docs themselves are not complete. This will be rectified with Boost V1.41, where the docs will be complete and Spirit V2.1 will officially replace Spirit.Classic (whose docs will be 'hidden' from that point on).
I do not think it is wise to hide documentation when you want people to try a product in Beta. How is anybody supposed to find the docs when you suggest to them to try Spirit 2.0 or Spirit 2.1. Somewhere, on some web page centered around Spirit, you should actually have links to the 2.0 and 2.1 docs such as they are. If I search for Spirit, I find a Spirit home page at http://spirit.sourceforge.net/, but good luck in finding 2.0 or 2.1 docs from there. I can also find Spirit mentioned at the Boost home page at http://www.boost.org/, but again I will not find the 2.0 or 2.1 docs from there either. As far as the docs not being complete, and their eventually being complete when Boost 1.41 comes out, please excuse me for disagreeing that is the way to create Beta software. I realize how hard you and Joel have worked creating and updating Spirit, but I myself have just grown tired, after many years of programming, of implementations whose use is suggested but whose docs are never up to par. I just think is the wrong way to produce software, even very high quality and free software as Spirit is, to make programmers work harder than is necessary to understand how the software actually works in light of incomplete documentation. I will wait for 1.41 before I look at Spirit 2.1 to see if I might want to use it. I do not blame the OP if he feels the same, despite your suggestion to hime to look at the better, more functional, later Beta release.
Edward,
We consider Spirit V2 to be a beta version and the docs themselves are not complete. This will be rectified with Boost V1.41, where the docs will be complete and Spirit V2.1 will officially replace Spirit.Classic (whose docs will be 'hidden' from that point on).
I do not think it is wise to hide documentation when you want people to try a product in Beta. How is anybody supposed to find the docs when you suggest to them to try Spirit 2.0 or Spirit 2.1. Somewhere, on some web page centered around Spirit, you should actually have links to the 2.0 and 2.1 docs such as they are.
The main reason for hiding those docs was that these are very incomplete. If you compare them to the docs I pointed the OP to you'll see what I mean. We made sure everybody asking for Spirit V2.1 got directions on how to find the current docs. We have had really active discussions about the beta on the Spirit mailing list for over a year now. Many people participated. We wrote a lot of examples and hundreds of very illustrative tests for interested people to look at. But I agree, all this is no excuse for not publishing the links more prominently.
If I search for Spirit, I find a Spirit home page at http://spirit.sourceforge.net/, but good luck in finding 2.0 or 2.1 docs from there. I can also find Spirit mentioned at the Boost home page at http://www.boost.org/, but again I will not find the 2.0 or 2.1 docs from there either.
As said, I agree we could have done a better job of advertising the newer version. Spirit V2.1 is a huge library and we rewrote it thrice before it actually reached the current state. Writing documentation for it is an extremely time consuming thing (I'm working on it for 4 month daily 1-2h now). So please excuse if we didn't find the proper amount of time to adequately publishing what we did so far. Joel and I want to get the docs out and will concentrate on public relations as soon as this is done.
As far as the docs not being complete, and their eventually being complete when Boost 1.41 comes out, please excuse me for disagreeing that is the way to create Beta software.
Sorry, I didn't get your point. What do you mean here?
I realize how hard you and Joel have worked creating and updating Spirit, but I myself have just grown tired, after many years of programming, of implementations whose use is suggested but whose docs are never up to par.
I believe the current documentation, even if not 100% complete, is nevertheless a excellent starting point for everybody interested in Spirit. It provides sufficient information to get started and we try to be very responsive on the Spirit mailing list if additional questions arise. So I think it _is_ the proper way to suggest to use the upcoming Spirit V2.1 as it's so much better than the current official version.
I just think is the wrong way to produce software, even very high quality and free software as Spirit is, to make programmers work harder than is necessary to understand how the software actually works in light of incomplete documentation.
If you looked at the docs I posted the link to you would have seen their adequateness for learning about the new version. I really don't understand what you want to say with your rant. If pre-release software is not for you, just don't use it. We only want to avoid people start using the old Spirit in new projects because we know the new version will be released soon. Especially if they struggle with bugs in the old version which are already fixed in the newer one.
I will wait for 1.41 before I look at Spirit 2.1 to see if I might want to use it.
Please be our guest!
I do not blame the OP if he feels the same, despite your suggestion to hime to look at the better, more functional, later Beta release.
May I ask what did upset you in the first place? Did I offend you in any way? Regards Hartmut
Hartmut Kaiser wrote:
Edward,
We consider Spirit V2 to be a beta version and the docs themselves are not complete. This will be rectified with Boost V1.41, where the docs will be complete and Spirit V2.1 will officially replace Spirit.Classic (whose docs will be 'hidden' from that point on). I do not think it is wise to hide documentation when you want people to try a product in Beta. How is anybody supposed to find the docs when you suggest to them to try Spirit 2.0 or Spirit 2.1. Somewhere, on some web page centered around Spirit, you should actually have links to the 2.0 and 2.1 docs such as they are.
The main reason for hiding those docs was that these are very incomplete. If you compare them to the docs I pointed the OP to you'll see what I mean.
We made sure everybody asking for Spirit V2.1 got directions on how to find the current docs. We have had really active discussions about the beta on the Spirit mailing list for over a year now. Many people participated. We wrote a lot of examples and hundreds of very illustrative tests for interested people to look at. But I agree, all this is no excuse for not publishing the links more prominently.
If I search for Spirit, I find a Spirit home page at http://spirit.sourceforge.net/, but good luck in finding 2.0 or 2.1 docs from there. I can also find Spirit mentioned at the Boost home page at http://www.boost.org/, but again I will not find the 2.0 or 2.1 docs from there either.
As said, I agree we could have done a better job of advertising the newer version. Spirit V2.1 is a huge library and we rewrote it thrice before it actually reached the current state. Writing documentation for it is an extremely time consuming thing (I'm working on it for 4 month daily 1-2h now). So please excuse if we didn't find the proper amount of time to adequately publishing what we did so far. Joel and I want to get the docs out and will concentrate on public relations as soon as this is done.
As far as the docs not being complete, and their eventually being complete when Boost 1.41 comes out, please excuse me for disagreeing that is the way to create Beta software.
Sorry, I didn't get your point. What do you mean here?
I realize how hard you and Joel have worked creating and updating Spirit, but I myself have just grown tired, after many years of programming, of implementations whose use is suggested but whose docs are never up to par.
I believe the current documentation, even if not 100% complete, is nevertheless a excellent starting point for everybody interested in Spirit. It provides sufficient information to get started and we try to be very responsive on the Spirit mailing list if additional questions arise. So I think it _is_ the proper way to suggest to use the upcoming Spirit V2.1 as it's so much better than the current official version.
I just think is the wrong way to produce software, even very high quality and free software as Spirit is, to make programmers work harder than is necessary to understand how the software actually works in light of incomplete documentation.
If you looked at the docs I posted the link to you would have seen their adequateness for learning about the new version. I really don't understand what you want to say with your rant. If pre-release software is not for you, just don't use it. We only want to avoid people start using the old Spirit in new projects because we know the new version will be released soon. Especially if they struggle with bugs in the old version which are already fixed in the newer one.
I will wait for 1.41 before I look at Spirit 2.1 to see if I might want to use it.
Please be our guest!
I do not blame the OP if he feels the same, despite your suggestion to hime to look at the better, more functional, later Beta release.
May I ask what did upset you in the first place? Did I offend you in any way?
I object to your saying above that my comments are a "rant". I am not offended. I was merely arguing that supplying software for Boost users, even Beta versions, should be more open as to where the documentation resides and that documentation should be complete. You are perfectly right in that if I am not happy with the documentation of the new versions I should stick with the current release of Spirit, which I will do.
On Sat, Aug 22, 2009 at 10:09 PM, Edward Diener
I object to your saying above that my comments are a "rant".
I am not offended. I was merely arguing that supplying software for Boost users, even Beta versions, should be more open as to where the documentation resides and that documentation should be complete.
Is not all the documentation for all boost projects in their
respective lib directories? If you sync to Boost Trunk you will have
the documentation in the normal and expected place, what is wrong
this? It is accessible by the svn as well, and it is constantly being
updated. There seems nothing odd about this, it is accessible as any
other boost library documentation is.
On Sat, Aug 22, 2009 at 10:09 PM, Edward Diener
You are perfectly right in that if I am not happy with the documentation of the new versions I should stick with the current release of Spirit, which I will do.
You will be missing out on a lot, the old Spirit is, quite frankly, rather horrible in comparison. Spirit 2.1 is complete, the docs are almost complete, and the docs that matter are already complete, there is no reason not to be using it.
On Fri, Aug 21, 2009 at 23:37, Hartmut Kaiser
I use a position_iterator
for parsing with boost::spirit. Because I need the regex_p feature, I have to wrap this iterator using a multi_pass iterator.
I call pt_parse and get a
boost::spirit::tree_parse_info
>, ...> parse_info. Now I have problems calling the get_position() method on "parse_info.stop".
I guess this is the case because multi_pass does not store my iterator but wraps it using boost::iterator which loses the position features of position_iterators and only keeps the const char*
On Thu, Aug 20, 2009 at 22:42, Hartmut Kaiser
wrote: pointer. Do you have an advice for me?
It's difficult to tell from here as you didn't provide too much information. Would you mind posting a small example exposing your problem?
I have attached a minimal example.
If you do not define MULTIPASSDBG, it builds but you are not allowed to use regex_p in the parser.
If you define it, you are allowed to use regex_p, but the get_position does no longer compile and so the iterator becomes a bit useless.
I read something about a "base()" method for wrapping iterators, but could not find one in the multi_pass sources.
Ok, I understand. But I think your whole approach is not viable. Regex_p requires a bi-directional iterator, while multi_pass exposes an input_iterator Only. So generally, even if it compiled it wouldn't work in the end.
I see, obviously I did not read the documentation closely enough, I just found out that it states regex_p requires a bidirectional iterator and multi_pass exposes a forward iterator. Nevertheless the combination multi_pass/regex_p compiled and worked fine for me. regex_p even worked fine with only the position_iterator (until you changed the position_iterator iteration type) which made it no longer compile with regex_p. (I used regex_p in the first place because lexeme_p included leading whitespace (which is really a nuisance to me), regex_p does not create this issue).
But to answer your original question, multi_pass doesn't allow to access the base iterator, which might be an oversight. I'll have a closer look and add it to the multi_pass iterator if possible.
Thank you for this. I wonder if it would be possible to create some multi_pass like iterator which allows regex_p to be used with a position_iterator (both are very handy tools and it is quite of sad that they cannot be used together). Is there a solution to this in Spirit V2.1? Nevertheless thank you a lot for your work on spirit, it is great! :) Regards, Peter
Try the attached solution. I took the regular expression parser and adapted it to use the base iterator for position_iterators. It's a bit of a hack so it probably shouldn't be added to spirit. And I haven't tested it much. Other solutions would be to write a bidirectional position iterator or just manually work out the positions after parsing. Or maybe track the position in the lexer, but I think that'd be hard with spirit. Daniel
I read something about a "base()" method for wrapping iterators, but could not find one in the multi_pass sources.
Ok, I understand. But I think your whole approach is not viable. Regex_p requires a bi-directional iterator, while multi_pass exposes an input_iterator Only. So generally, even if it compiled it wouldn't work in the end.
I see, obviously I did not read the documentation closely enough, I just found out that it states regex_p requires a bidirectional iterator and multi_pass exposes a forward iterator.
Yeah, sorry, I meant to say forward iterator as well...
Nevertheless the combination multi_pass/regex_p compiled and worked fine for me.
I think it depends on the regex you used. Not all regex require bidirectional support from the iterator.
regex_p even worked fine with only the position_iterator (until you changed the position_iterator iteration type) which made it no longer compile with regex_p.
IFAIR, the position_iterator was always a forward iterator only as well (at least starting from Boost 1.31, i.e. since it's using V2 of the iterator library). Making the position iterator bidirectional should be possible, but for this you need to add backwards position tracking, which isn't implemented.
(I used regex_p in the first place because lexeme_p included leading whitespace (which is really a nuisance to me), regex_p does not create this issue).
Yeah, you seem to fight this error in the tree code by inventing all kind of sophisticated workarounds. I'm sorry for the trouble. But as said before, I doubt somebody will find the time to get back to this. At least not before Spirit 2.1 is finished.
But to answer your original question, multi_pass doesn't allow to access the base iterator, which might be an oversight. I'll have a closer look and add it to the multi_pass iterator if possible.
Thank you for this.
I wonder if it would be possible to create some multi_pass like iterator which allows regex_p to be used with a position_iterator (both are very handy tools and it is quite of sad that they cannot be used together).
Is there a solution to this in Spirit V2.1?
Well, for one Spirit 2.1 doesn't expose this bug you're working around. But even if the multi_pass there has been rewritten (is faster and has a smaller footprint) it exposes still a forward iterator only. That's mainly because this is sufficient for Spirit parsers. The position_iterator has been rewritten as well, but it's not finished yet. If you look through the Spirit maoling list archives you should be able to find it. It's not bidirectional still, but if you like you can add this functionality yourself and contribute it back. Spirit V2.1 has no regex component yet, and I think we won't add it again. Spirit now has a full-fledged lexer you can use to tokenize the input stream using regular expressions.
Nevertheless thank you a lot for your work on spirit, it is great! :)
You're welcome! Regards Hartmut PS: Could you post further questions with wrt Spirit on the Spirit mailing list? This way other people might be able to join the discussions.
participants (6)
-
Daniel James
-
Edward Diener
-
Hartmut Kaiser
-
OvermindDL1
-
Peter Schueller
-
Steven Watanabe