Re: [boost] Using Serialization for binary marshalling

Matthew Vogt wrote:
Is anyone else interested in using Robert Ramey's serialization library for the purpose of marshalling data into pre-determined binary formats?
Interest has been expressed for XDR in the past. Basically, I see this as a very positive thing and want to do everything I can to support your efforts. Your code supports save and load for two new formats (XDR and CDR). This is a valuable contribution. I'm also pleased that the serialization library has been deemed (at least by one person) to be worthy of investment of such an effort ( 1500 lines of code !). I took a little time to scan your attachments. My examination was cursory and feel free to correct any thing I didn't get right. Here are my comments. 1) I wonder why you derived ordered_oarchive from basic_binary_oprimitive<Archive, OStream> . It doesn't seem that any of the functionality basic_binary_oprimitive actually used. I presume the same goes for ordered_iarchive . 2) there's some stuff in boost that addresses alignment in a guaranteed? Portable manner that may relevant here. Sse #include <boost/aligned_storage.hpp> . BTW - the best way to make your code portable without cluttering up with #ifdef etc... is to use more boost stuff - let other people clutter up their code with #ifdef all over the place. 3) I'm curious about the override for the saving of vector. It seems to me equivalent to the one included with the system. And if vector requires special treatment, what about the list, et al. This doesn't seem necessary to me. Perhaps the fact that I did treat strings specially was misleading. I considered this a special case. I don't think vectors are in this context. 4) Missing license, authorship etc. Also add more comments and description and explanation of motivation for things that might not be obvious. This is a work of literature and art that will live for a long time - make it worthy. 5) I took great pains to organize code so that users of the library would not be including things that were strictly implementation. This is the origin of the *.ipp files. You might want to consider the same. Any template that does have a serializable type as a parameter (ie only archive) is a candidate for this treatment. In fact I would like to see a more "formal" separation of interface and implementation. 6) Get the bjam system running if you haven't already. It would be quite easy to add your archives to the set tested. Currenty, the test system runs about 30-40 tests on each archive type. It's a torture test ( and still I'm finding bugs). I think it would take you maybe an hour to tweak the bjam script and the code in serialization/test/test_tools.hpp to run all the tests on your own archive classes - ( maybe I should add information on doing this to the manual - it never occurred to me that someone would get to this point so soon) This is an absolute must if you want to have your addition seriously considered. 7) I'm just a little queasy about using the term marshalling. In fact any of the archive implementations could be used for marshalling - not just a binary one (XML is the favorite flavor of the month). So I think it's sort of misleading. I think it is very helpful that your naming scheme xdr_oarchive, cdr_oarchive, leverages on the existing one. 8) in my native binary archive, I used a template parameter for the stream class. As it stands, I don't test or use the corresponding archive for wide character streams because the WEOF would need special treatment. I haven't spent any time on this. Perhaps you've addressed this in some way. I didn't look in that much detail. I suspect that's enough to keep you motivated. Good luck and thanks for working on and with the library. Robert Ramey

Robert Ramey <ramey <at> rrsd.com> writes:
Basically, I see this as a very positive thing and want to do everything I can to support your efforts. Your code supports save and load for two new formats (XDR and CDR). This is a valuable contribution. I'm also pleased that the serialization library has been deemed (at least by one person) to be worthy of investment of such an effort ( 1500 lines of code !).
Actually, XDR and CDR are of secondary interest for me; I'm interested in being able to save to proprietary formats (which often means that the applications involved never got around to specifying a standard format...). There must be thousands of ad-hoc binary formats in use today.
1) I wonder why you derived ordered_oarchive from basic_binary_oprimitive<Archive, OStream> . It doesn't seem that any of the functionality basic_binary_oprimitive actually used. I presume the same goes for ordered_iarchive .
I'm using the save_binary / load_binary functions from basic_binary_oprimitive and basic_binary_iprimitive.
2) there's some stuff in boost that addresses alignment in a guaranteed? Portable manner that may relevant here. Sse #include <boost/aligned_storage.hpp> . BTW - the best way to make your code portable without cluttering up with #ifdef etc... is to use more boost stuff - let other people clutter up their code with #ifdef all over the place.
Yes, but the aligned_storage template helps with platform-specific alignment within the machine. I don't see how it helps with platform-independent alignment within the content of the archive...
3) I'm curious about the override for the saving of vector. It seems to me equivalent to the one included with the system. And if vector requires special treatment, what about the list, et al. This doesn't seem necessary to me. Perhaps the fact that I did treat strings specially was misleading. I considered this a special case. I don't think vectors are in this context.
I need to override the serialization of vector, because the vector must be serialized with a known policy to yield a required layout in the archive. The fact that this serialization happens (at this point in time) to be exactly the same as the default implementation is not relevant - that can be changed at any time, but the CDR, XDR and other binary formats must not change. [ Points 4-6 conceded ]
7) I'm just a little queasy about using the term marshalling. In fact any of the archive implementations could be used for marshalling - not just a binary one (XML is the favorite flavor of the month). So I think it's sort of misleading. I think it is very helpful that your naming scheme xdr_oarchive, cdr_oarchive, leverages on the existing one.
Yes, I see what you're saying. I'll have a think about this - but the term 'marshalling' is not particularly prevalent in the code. Even if the term does have broad application, I think I am using it in the traditional sense.
8) in my native binary archive, I used a template parameter for the stream class. As it stands, I don't test or use the corresponding archive for wide character streams because the WEOF would need special treatment. I haven't spent any time on this. Perhaps you've addressed this in some way.
No, I was hoping you'd have it all sorted out - I've never used a stream specialised with anything but char. In fact, it doesn't really make sense to me, to produce binary output as anything but a byte stream - I was just following your lead, really :)
I didn't look in that much detail. I suspect that's enough to keep you motivated. Good luck and thanks for working on and with the library.
Yes, thanks for your comments. Working with the library is a pleasure. Matt
participants (2)
-
Matthew Vogt
-
Robert Ramey