RFC: Spirit Porting

Bryce Lelbach

28 Oct 2010 28 Oct '10

6:25 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 This is possibly better suited for the Spirit mailing list; however, I would like to get as broad an audience as possible here. Recently, I ported the Boost.Serialization XML grammar from Spirit Classic to Spirit Qi. It was a pretty straightforward port, but I did expect an increase in the compile-time complexity of building the grammar. My reasons for this line of thinking were as follows: * Spirit 2.x is fully attributed, while Spirit 1.x is not. This increases both the amount of data that we're dealing with (every parser/generator component must have an attribute), as well as adding additional compile-time checks and transformations which much be applied to that data. * To make Spirit fully attributed, Spirit 2.x uses frameworks that Spirit Classic did not use; Boost.Proto and Boost.Fusion (Fusion was created for Spirit 2.x, I believe), as well as a newer version of Boost.Phoenix. To my knowledge, this new XML grammar is the first Boost component to use Spirit 2.x (Wave uses Spirit Classic). I believe this means this is the first time that software using Spirit 2.x has been placed under the scrutiny of Boost testing. Certainly, Spirit 2.x and the underlying libraries it uses undergo rigorous testing, but in my mind, testing a library and testing software that uses said library are two very different processes. My question stems partially from the thoughts of Serialization's maintainer, Robery Ramey: "Basically, my view is that we cannot ship something that doesn't work at least as well as the previous version." My question for you: Can software that uses Spirit Classic be ported to Spirit 2.x and retain the same level of compiler/platform compatibility without significant refactoring of the fundamental structure of the Spirit components of the software? Please do note that I believe that there is room in the Serialization XML grammar for refactoring, and I certainly don't mean to imply that the authors of Spirit, Proto, Fusion or Phoenix are at fault here in any way. - -- Bryce Lelbach aka wash http://groups.google.com/group/ariel_devel -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkzJF3UACgkQO/fqqIuE2t7j2wCgumHhaOfOEOvmZ/090cm5o7+x f9QAoOhPvph24tlpCnuTrnUdtZ6BGD6/ =Dfkl -----END PGP SIGNATURE-----

Show replies by date

Robert Ramey

28 Oct 28 Oct

5:15 p.m.

New subject: Spirit Porting

Bryce Lelbach wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

This is possibly better suited for the Spirit mailing list; however, I would like to get as broad an audience as possible here.

Recently, I ported the Boost.Serialization XML grammar from Spirit Classic to Spirit Qi. It was a pretty straightforward port, but I did expect an increase in the compile-time complexity of building the grammar. My reasons for this line of thinking were as follows:

* Spirit 2.x is fully attributed, while Spirit 1.x is not. This increases both the amount of data that we're dealing with (every parser/generator component must have an attribute), as well as adding additional compile-time checks and transformations which much be applied to that data. * To make Spirit fully attributed, Spirit 2.x uses frameworks that Spirit Classic did not use; Boost.Proto and Boost.Fusion (Fusion was created for Spirit 2.x, I believe), as well as a newer version of Boost.Phoenix.

To my knowledge, this new XML grammar is the first Boost component to use Spirit 2.x (Wave uses Spirit Classic). I believe this means this is the first time that software using Spirit 2.x has been placed under the scrutiny of Boost testing. Certainly, Spirit 2.x and the underlying libraries it uses undergo rigorous testing, but in my mind, testing a library and testing software that uses said library are two very different processes.

My question stems partially from the thoughts of Serialization's maintainer, Robery Ramey:

"Basically, my view is that we cannot ship something that doesn't work at least as well as the previous version."

My question for you:

Can software that uses Spirit Classic be ported to Spirit 2.x and retain the same level of compiler/platform compatibility without significant refactoring of the fundamental structure of the Spirit components of the software?

Please do note that I believe that there is room in the Serialization XML grammar for refactoring, and I certainly don't mean to imply that the authors of Spirit, Proto, Fusion or Phoenix are at fault here in any way.

I have encouraged and supported Bryce's undertaking on the assumption that the usage of the latest spirit would improve the xml_iarchive in every way including performance, maintainability and portability. This assumption was inspired and supported by the warnings that spirit classic was "deprecated" which seemed to suggest that it would be in our interest to convert to the latest spirit. The xml grammar used in the serialization library is very simple - what could go wrong? Well, It is starting to look like this assumption was wrong and I'm very disappointed. I feel the developer's of spirit have let Bryce down. I'm hoping the spirit developers can step up and follow through to realize the expections developed for this package. I'm hopeful that a couple of tweaks can reslove the issue - as I said the xml_archive grammar is a very simple one. Some of the issues seem simple as well - compile time with intel compilers, some hiccup deep in mpl which hangs up the IBM C++ compiler. And I'm more than hopeful that the spirit developer's can place a high priority on doing whatever is necessary to fix this. The serialization library is one of the highest profile users of spirit and an easy example and test case. Once, you put it out there, it's your responsability to see that it meets the expections you've created for it. I was very surprised (and disappointed) to find that Wave - a very convincing example of the capabilities of spirit does not use the latest spirit. Had I known that, I wouldn't have encouraged Bryce with his efforts. As I said, I'm hopeful that this is not too hard to fix - the spirit library tests are mostly OK (except for the IBM compiler which has to be resolved). Maybe you could start by adding a test which builds a grammar similar to the xml_archive grammar. Robert Ramey

Joel de Guzman

29 Oct 29 Oct

12:01 a.m.

New subject: Spirit Porting

On 10/29/2010 1:15 AM, Robert Ramey wrote:

...

I have encouraged and supported Bryce's undertaking on the assumption that the usage of the latest spirit would improve the xml_iarchive in every way including performance, maintainability and portability. This assumption was inspired and supported by the warnings that spirit classic was "deprecated" which seemed to suggest that it would be in our interest to convert to the latest spirit. The xml grammar used in the serialization library is very simple - what could go wrong? Well, It is starting to look like this assumption was wrong and I'm very disappointed. I feel the developer's of spirit have let Bryce down.

What are you saying? Are you even aware of the interactions between us regarding this? It seems that you haven't even been reading the Spirit mailing list about this and the threads surrounding it. I haven't seen you interact. And now you are saying that we are letting Bryce down? Accusatory words won't help at all, Robert.

...

I'm hoping the spirit developers can step up and follow through to realize the expections developed for this package.

You too, sir! 100% porting is never an easy task. What we are looking at are compiler quirks especially with lesser conforming compilers. The xml grammar used in the serialization library is *not* simple. You, of all people should know that. Bryce did an amazing job. It's the 1% that always kicks us in the end. That's why I am saying that we need another round to iron these out. ***These are not easy tasks*** I'm sure anyone working around ICEs should know. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net

Hartmut Kaiser

1:21 a.m.

New subject: Spirit Porting

...

I have encouraged and supported Bryce's undertaking on the assumption that the usage of the latest spirit would improve the xml_iarchive in every way including performance, maintainability and portability. This assumption was inspired and supported by the warnings that spirit classic was "deprecated" which seemed to suggest that it would be in our interest to convert to the latest spirit. The xml grammar used in the serialization library is very simple - what could go wrong? Well, It is starting to look like this assumption was wrong and I'm very disappointed. I feel the developer's of spirit have let Bryce down.

Robert, I believe you didn't mean what you wrote, as otherwise your words appear to be a bit too heavy handed. We are in permanent and active contact with Bryce (there is a lengthy mailing list discussion, and I am in contact with him on a daily basis on IRC). Nobody left Bryce down. Additionally, I believe he explicitly stated that in his original mail. BTW, the 'deprecated' warnings never claimed that Spirit.Classic as a whole was deprecated, but only the headers you've been using. We have no intent to remove Spirit.Classic from Boost. A simple change of the include directive to other header files as suggested by those warnings (and defining a pp constant during compilation) would have fixed the warnings altogether. That's something you could have done in 15 minutes if you only had cared to ask.

...

I'm hoping the spirit developers can step up and follow through to realize the expections developed for this package.

What expectations? What package?

...

I'm hopeful that a couple of tweaks can reslove the issue - as I said the xml_archive grammar is a very simple one. Some of the issues seem simple as well - compile time with intel compilers, some hiccup deep in mpl which hangs up the IBM C++ compiler. And I'm more than hopeful that the spirit developer's can place a high priority on doing whatever is necessary to fix this. The serialization library is one of the highest profile users of spirit and an easy example and test case. Once, you put it out there, it's your responsability to see that it meets the expections you've created for it.

Just forget the IBM compiler. It's not up to the task to compile any of Spirit V2 code. We talked to Michel Wong at BoostCon about this and he said that his compiler team resources are very limited and that he does not expect this to be fixed too soon.

...

I was very surprised (and disappointed) to find that Wave - a very convincing example of the capabilities of spirit does not use the latest spirit. Had I known that, I wouldn't have encouraged Bryce with his efforts.

Again, this sounds like a hasty assessment to me. You have no idea what it means to port something like Wave to a different underlying parser platform. It's not that somebody would be paying for doing so...

...

As I said, I'm hopeful that this is not too hard to fix - the spirit library tests are mostly OK (except for the IBM compiler which has to be resolved). Maybe you could start by adding a test which builds a grammar similar to the xml_archive grammar.

Working around compiler deficits or simply braindead compilers is never easy. However, that's exactly what you are demanding here so light-heartily. Bryce is working hard, harder than many people I know. If I was you I wouldn't push my luck, but instead be happy he is willing to help. Regards Hartmut --------------- http://boost-spirit.com

Bryce Lelbach

30 Oct 30 Oct

7:40 p.m.

New subject: Spirit Porting

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, 28 Oct 2010 09:15:45 -0800 "Robert Ramey" wrote:

...

I feel the developer's of spirit have let Bryce down. I'm hoping the spirit developers can step up and follow through to realize the exceptions developed for this package.

Again I want to stress that this is my problem. Spirit devs have their own library to maintain, and I bug them enough with questions/etc as it is. On Thu, 28 Oct 2010 09:15:45 -0800 "Robert Ramey" wrote:

...

I'm hopeful that a couple of tweaks can resolve the issue - as I said the xml_archive grammar is a very simple one.

I'm not going to get into a debate about the complexity of the XML Spirit grammar, because my time is rather limited at present. I'll say this; regardless of the scope (e.g. in general or in comparison to other Spirit grammars), the XML grammar isn't simple. A standards-compliant non-validating XML parser must implement appx 88 production rules. A validating parser has to go above and beyond that. XML is debatable simple on the user end of things, but it is not a simple language to implement (off the top of my head, some examples of simple languages: Lua, Jam, Scheme). We do not implementing a fully validating XML parser in Serialization (and there is no reason or need to), but that doesn't negate the underlying complexity of implementing the language. On Thu, 28 Oct 2010 09:15:45 -0800 "Robert Ramey" wrote:

...

Compile time with intel compilers,

I'm honestly more concerned about cleaning up warnings and improving compile time on VS/GCC. On Thu, 28 Oct 2010 09:15:45 -0800 "Robert Ramey" wrote:

...

[snip about IBM compiler issues]

If someone provides me with a licensed copy of IBM's compiler, I will look into the MPL thing and spend some time trying to resolve the issue with Serialization. Unfortunately, my understanding is that IBM's compiler is closed source, which will prevent me from actually fixing any of the compiler's problems instead of just writing workarounds. Because of this, I'm not going to commit large portions of my time to IBM's compiler, as I haven't heard any users asking for this and I'm not a fan of fruitlessly writing workarounds. On Thu, 28 Oct 2010 09:15:45 -0800 "Robert Ramey" wrote:

...

I was very surprised (and disappointed) to find that Wave - a very convincing example of the capabilities of spirit does not use the latest spirit.

It's on my long list of things that I'll never get to (porting it, that is) On Thu, 28 Oct 2010 09:15:45 -0800 "Robert Ramey" wrote:

...

As I said, I'm hopeful that this is not too hard to fix - the spirit library tests are mostly OK (except for the IBM compiler which has to be resolved).

The Spirit/Serialization examples, however, are not run through the test matrix, so we don't have a complete data set to analyze. Unit-tests are an important part of the testing process, however, they absolutely need to be accompanied by function-tests. Try compiling the Spirit examples with GCC and C++0x turned on. You'll run into a nasty namespace conflict brought on by the addition of std::ref (using declarations cause an ambiguity in one of the examples between std::ref and boost::phoenix::ref). I don't mean to pick on Spirit* - this may even be fixed now, as I think I may have reported it. My point is a number of Boost libraries, including Serialization, have function-tests in the form of examples that reveal minor to moderate bugs that are not detected by the Boost unit-tests. On Thu, 28 Oct 2010 20:21:45 -0500 "Hartmut Kaiser" wrote:

...

Just forget the IBM compiler. It's not up to the task to compile any of Spirit V2 code.

On Sat, 30 Oct 2010 08:34:34 +0800 Joel de Guzman wrote:

...

Alas, neither Hartmut nor I have access to the IBM compiler nor have any experience with it.

Ah, but we've got at least one Spirit user who is using Visual Age (Avi), and he was one of the first people to respond to this thread. As I have said, my time is limited, but if someone makes the resources available to me (aka legal copy of Visual Age + documentation), I will put some time into this (I don't think it's too farfetched for me to say that I'm a "compiler hacker"). Some time ago Bryce Lelbach wrote:

...

...
Can software that uses Spirit Classic be ported to Spirit 2.x and retain the same level of compiler/platform compatibility without significant refactoring of the fundamental structure of the Spirit components of the software?

Ramey, Hartmut and Joel, I really would like it if we could not have drama. I'm honored that all three of you put such value in my work. I made this post, though, to start a discussion on the above question. No one has answered it yet :( * Everyone should know I <3 Spirit - -- Bryce Lelbach aka wash http://groups.google.com/group/ariel_devel -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkzMdLUACgkQO/fqqIuE2t7qIwCg7i2S7QSdokMZhZqhAcPN//S4 tlUAoO/GX7sm6eMgXirR2FZZDF45WyVg =AlJG -----END PGP SIGNATURE-----

Eric Niebler

11:58 p.m.

New subject: Spirit Porting

On 10/30/2010 12:40 PM, Bryce Lelbach wrote:

...

On Thu, 28 Oct 2010 09:15:45 -0800 "Robert Ramey" wrote:

...
I feel the developer's of spirit have let Bryce down. I'm hoping the spirit developers can step up and follow through to realize the exceptions developed for this package.

Robert, I think you're being a bit dramatic. In the course of routine library maintenance it's pretty normal for compatibility with broken compilers to be lost. It can be sometimes be fixed, but it's not always worth the effort. If I made some change to BOOST_FOREACH that caused it to stop working on VC6 and someone complained, I'd tell them to stick with earlier releases. So, how to proceed? Someone who cares for this platform submits patches to the libraries involved: MPL, Proto, Fusion and Spirit2. Barring that, the options are: leave in the Spirit1 code, or else leave them both in and conditionally select the old or the new implementation. But expecting Spirit2 (and 3 and 4...) to continue working on every dang ol' broken compiler that Spirit1 ever worked on is unrealistic. (OK, vacpp isn't old, but it *is* quite broken.) That said, I *do* believe that if a new release breaks compatibility with old releases, there should be release notes to that effect.

...

Some time ago Bryce Lelbach wrote:

...
...
Can software that uses Spirit Classic be ported to Spirit 2.x and retain the same level of compiler/platform compatibility without significant refactoring of the fundamental structure of the Spirit components of the software?

Ramey, Hartmut and Joel, I really would like it if we could not have drama. I'm honored that all three of you put such value in my work. I made this post, though, to start a discussion on the above question. No one has answered it yet :(

I will: no. Not until someone who cares about vacpp does the work to make it so, and continues to do the work to keep it so. -- Eric Niebler BoostPro Computing http://www.boostpro.com

Joel de Guzman

31 Oct 31 Oct

1:07 a.m.

New subject: Spirit Porting

On 10/31/10 7:58 AM, Eric Niebler wrote:

...

On 10/30/2010 12:40 PM, Bryce Lelbach wrote:

...
On Thu, 28 Oct 2010 09:15:45 -0800 "Robert Ramey" wrote:

...
I feel the developer's of spirit have let Bryce down. I'm hoping the spirit developers can step up and follow through to realize the exceptions developed for this package.

Robert, I think you're being a bit dramatic. In the course of routine library maintenance it's pretty normal for compatibility with broken compilers to be lost. It can be sometimes be fixed, but it's not always worth the effort. If I made some change to BOOST_FOREACH that caused it to stop working on VC6 and someone complained, I'd tell them to stick with earlier releases.

So, how to proceed? Someone who cares for this platform submits patches to the libraries involved: MPL, Proto, Fusion and Spirit2. Barring that, the options are: leave in the Spirit1 code, or else leave them both in and conditionally select the old or the new implementation.

But expecting Spirit2 (and 3 and 4...) to continue working on every dang ol' broken compiler that Spirit1 ever worked on is unrealistic. (OK, vacpp isn't old, but it *is* quite broken.)

That said, I *do* believe that if a new release breaks compatibility with old releases, there should be release notes to that effect.

...
Some time ago Bryce Lelbach wrote:

...
...
Can software that uses Spirit Classic be ported to Spirit 2.x and retain the same level of compiler/platform compatibility without significant refactoring of the fundamental structure of the Spirit components of the software?

Ramey, Hartmut and Joel, I really would like it if we could not have drama. I'm honored that all three of you put such value in my work. I made this post, though, to start a discussion on the above question. No one has answered it yet :(

I will: no. Not until someone who cares about vacpp does the work to make it so, and continues to do the work to keep it so.

I thought that was implied in my "dramatic" reply to Robert's "dramatic" post. And my implied answer is NO. In my experience, that is simply not *practically* possible. Robert make it sound so easy. It is not! Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net

Joel de Guzman

1:46 a.m.

New subject: Spirit Porting

On 10/31/10 3:40 AM, Bryce Lelbach wrote:

...

...
[snip about IBM compiler issues]

If someone provides me with a licensed copy of IBM's compiler, I will look into the MPL thing and spend some time trying to resolve the issue with Serialization. Unfortunately, my understanding is that IBM's compiler is closed source, which will prevent me from actually fixing any of the compiler's problems instead of just writing workarounds. Because of this, I'm not going to commit large portions of my time to IBM's compiler, as I haven't heard any users asking for this and I'm not a fan of fruitlessly writing workarounds. ...

...

On Thu, 28 Oct 2010 20:21:45 -0500 "Hartmut Kaiser" wrote:

...
Just forget the IBM compiler. It's not up to the task to compile any of Spirit V2 code.

On Sat, 30 Oct 2010 08:34:34 +0800 Joel de Guzman wrote:

...
Alas, neither Hartmut nor I have access to the IBM compiler nor have any experience with it.

Ah, but we've got at least one Spirit user who is using Visual Age (Avi), and he was one of the first people to respond to this thread. As I have said, my time is limited, but if someone makes the resources available to me (aka legal copy of Visual Age + documentation), I will put some time into this (I don't think it's too farfetched for me to say that I'm a "compiler hacker").

Re: licensed copy of IBM's compiler: We also discussed that with Michael. Michael mentioned that he wanted to make a platform available for boosters. Perhaps it's time to ask again. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net

5228

Age (days ago)

5231

Last active (days ago)

List overview

Download

7 comments

5 participants

participants (5)

Bryce Lelbach
Eric Niebler
Hartmut Kaiser
Joel de Guzman
Robert Ramey