[serialization] Announcing eos portable archive
Dear boost fellows, during the last two weeks I finally had time to port our archives to recent boost versions up to 1.36. I also refactored a bit, put the archives into namespace eos and renamed them portable_[io]archive in order to prevent further mix-up with the portable binary archive from the examples. In addition I finally did some testing on x64 and found our code working without complaints. So I encourage anyone who wants to use binary serialization across different platforms with floating point support to have a look at it at the boost vault: http://www.boostpro.com/vault/index.php?directory=serialization Cheers, -- Christian Pfligersdorffer Software Engineering http://www.eos.info
I've just taken a short look at this implementation and it looks pretty good to me. I like the usage of BOOST_STATIC_ASSERT along with code comments that start with "If your compile fails here .... I also like the extensive comments/rationale right there in the code. a couple of minor nits. a) I would like to see the code formated slightly differently to make it more readable. Specifically I would like to see #if indented in the same way that the normal if( is. I realise that this is my particular taste so I don't want to make a huge argument for it. But aesthetically it would better match the other serialization code. Also consistent usage of 4 space indents for tabbing would be nice. b) The version of the portable binary library that is currently part of the package uses the endianness on the machine which create the archive. On reading the archive, the endienness is swapped around if and only if the the endienness of the reading machine is different than that contained in the archive. The endiennes of the archive is stored in the archive header. The idea is that the most likely reader of an archive is one using the same endienness of the creator. Also, Endienness of the archive being created can be overridden during archive construction. This is basically a way of keeping myself out of any endieness debate. c) I would like to see the complete test suite run on this. If you checkout the libs/serialization/test directory. You can see that there is a *.hpp file for each archvie type. By adding your own eos_portable_binary_archive.hpp and using a command line switch the the bjam invocation, you can run all the serialization tests on just your new archive. From personal experience I can tell you that this is an eye opening experience. d) it seems that its portability is not universal which is OK by me as it seems to have a pretty wide coverage. But more information on this would be useful. e) Note that the current text archives don't support serialization of NaN of various types. They just throw exceptions. So I have no problems with you're doing the same thing. However, not everyone agrees with me on this so expect some complaint. The issue of portably serializings NaN, should be addresses in all portable archives as a group - as it is essentially the same problem. Oh - then there is a documentation page. Good Luck with this. Robert Ramey Pfligersdorffer, Christian wrote:
Dear boost fellows,
during the last two weeks I finally had time to port our archives to recent boost versions up to 1.36. I also refactored a bit, put the archives into namespace eos and renamed them portable_[io]archive in order to prevent further mix-up with the portable binary archive from the examples.
In addition I finally did some testing on x64 and found our code working without complaints. So I encourage anyone who wants to use binary serialization across different platforms with floating point support to have a look at it at the boost vault:
http://www.boostpro.com/vault/index.php?directory=serialization
Cheers,
Good morning everybody, due to an upload error yesterday the vault version stayed the old one 'till one minute ago. So if you wondered what I was writing about please download version 3 now. Thank you Robert for having a look at my implementation and sharing your thoughts! I shall go through them one by one. Robert Ramey on Thursday, September 25, 2008 8:40 PM:
I've just taken a short look at this implementation and it looks pretty good to me. I like the usage of BOOST_STATIC_ASSERT along with code comments that start with "If your compile fails here .... I also like the extensive comments/rationale right there in the code.
Thank you! Code that is generic, minimalistic and tricky like that (and like most boost code too) really is not to be understood without rigorous commenting it. At least that is my feeling and I want to still understand my own code in a year or two ;-)
a) I would like to see the code formated slightly differently to make it more readable. Specifically I would like to see #if indented in the same way that the normal if( is. I realise that this is my particular taste so I don't want to make a huge argument for it. But aesthetically it would better match the other serialization code. Also consistent usage of 4 space indents for tabbing would be nice.
If you have a look at the now updated version 3 you'll find that I circumvented most of the #ifs that IMHO cripple code and make it even less readable. So this should be fine! (If not: my heart does not depend on that, neither on the indention ;)
b) The version of the portable binary library that is currently part of the package uses the endianness on the machine which create the archive. On reading the archive, the endienness is swapped around if and only if the the endienness of the reading machine is different than that contained in the archive. The endiennes of the archive is stored in the archive header. The idea is that the most likely reader of an archive is one using the same endienness of the creator. Also, Endienness of the archive being created can be overridden during archive construction. This is basically a way of keeping myself out of any endieness debate.
I see. The issue also came up in the lengthy cross-platform binary serialization a few weeks ago. I personally understand one would like to choose the endianness but what great benefits does it bear? We chose little endian because it is most widely used and was convenient to implement. I guess it would be a minor modification to allow for both but I don't deem it neccessary.
c) I would like to see the complete test suite run on this. If you checkout the libs/serialization/test directory. You can see that there is a *.hpp file for each archvie type. By adding your own eos_portable_binary_archive.hpp and using a command line switch the the bjam invocation, you can run all the serialization tests on just your new archive. From personal experience I can tell you that this is an eye opening experience.
I shall definitely do that! So far I have my own test cases which are mainly concerned with portability but you're right: the tests for one platform should also succeed.
d) it seems that its portability is not universal which is OK by me as it seems to have a pretty wide coverage. But more information on this would be useful.
As the archives usage spreads I hope to get word from users whether it works for them or not. In the latter case I guess I'm more likely to receive word :) Anyway, I need more information too, simply because I do not have a computer museum at hand.
e) Note that the current text archives don't support serialization of NaN of various types. They just throw exceptions. So I have no problems with you're doing the same thing. However, not everyone agrees with me on this so expect some complaint. The issue of portably serializings NaN, should be addresses in all portable archives as a group - as it is essentially the same problem.
Oh, I did not recognise that. So there is already an exception type for that case I can use. This I will change with the next version.
Oh - then there is a documentation page.
Hmm, are you considering to incorporate the archives into the official distribution? I am not unwilling to do what is neccessary as long as it does not take too much time. You must understand that my employer pays me for satisfying his need and not that of the boost community. So, if there is a lot of discussion about making it universally appliable and fit for every scenario I prefer to stay it an addition. If you and a few other boost "vips" :-) consider it basically fit the way it is I'd be happy to contribute. I guess one major issue is dependency on integer and fp_utilities. Trusting in the acceptance of those libraries into boost I did not hesitate to use them but integer is still not part of boost and fp_utilities have been completely redesigned for inclusion in boost::math v1.35 breaking my code. So still both libraries have to be added i.o. to use my archives. We'll see what comes from it... -- Christian Pfligersdorffer Software Engineering http://www.eos.info
Pfligersdorffer, Christian wrote: [..]
b) The version of the portable binary library that is currently part of the package uses the endianness on the machine which create the archive. On reading the archive, the endienness is swapped around if and only if the the endienness of the reading machine is different than that contained in the archive. The endiennes of the archive is stored in the archive header. The idea is that the most likely reader of an archive is one using the same endienness of the creator. Also, Endienness of the archive being created can be overridden during archive construction. This is basically a way of keeping myself out of any endieness debate.
I see. The issue also came up in the lengthy cross-platform binary serialization a few weeks ago. I personally understand one would like to choose the endianness but what great benefits does it bear? We chose little endian because it is most widely used and was convenient to implement. I guess it would be a minor modification to allow for both but I don't deem it neccessary.
IMO taking care of the endianness should be an important aspect of a portable binary library. Just my 2 cents, Daniel
Daniel Krügler wrote:
Pfligersdorffer, Christian wrote: [..]
b) The version of the portable binary library that is currently part of the package uses the endianness on the machine which create the archive. On reading the archive, the endienness is swapped around if and only if the the endienness of the reading machine is different than that contained in the archive. The endiennes of the archive is stored in the archive header. The idea is that the most likely reader of an archive is one using the same endienness of the creator. Also, Endienness of the archive being created can be overridden during archive construction. This is basically a way of keeping myself out of any endieness debate.
I see. The issue also came up in the lengthy cross-platform binary serialization a few weeks ago. I personally understand one would like to choose the endianness but what great benefits does it bear? We chose little endian because it is most widely used and was convenient to implement. I guess it would be a minor modification to allow for both but I don't deem it neccessary.
IMO taking care of the endianness should be an important aspect of a portable binary library.
Just my 2 cents,
Make that 4 cents. :-) I agree with Robert's approach on this one. I'd go even further though and make the endian-ness part of the archive type. Or at least a required constructor parameter separate from the other flags. Some of our engineers were bitten when the endian-ness defaulted to the build machine's endian-ness. Lot's of problems with Mac OS universal binaries. Jeff
Oh - then there is a documentation page.
Hmm, are you considering to incorporate the archives into the official distribution? I am not unwilling to do what is neccessary as long as it does not take too much time. You must understand that my employer pays me for satisfying his need and not that of the boost community. So, if there is a lot of discussion about making it universally appliable and fit for every scenario I prefer to stay it an addition. If you and a few other boost "vips" :-) consider it basically fit the way it is I'd be happy to contribute.
Regardless of the eventual future of this, you should make the page. Its not hard to do. If this ever get migrated into the official distribution the page could then be incorporated into the official documentation. Robert Ramey
participants (4)
-
Daniel Krügler
-
Jeff Flinn
-
Pfligersdorffer, Christian
-
Robert Ramey