On Sun, 18 Apr 2004 09:38:59 -0700, Robert Ramey wrote
Matthew Vogt writes:
Actually, XDR and CDR are of secondary interest for me;
For me too, but these have be much requested and finishing these off in a polished way will be significant contribution to the library for many users.
I'm interested in being able to save to proprietary formats (which often means that the applications involved never got around to specifying a standard format...). There must be thousands of ad-hoc binary formats in use today.
Hmmm - I'm a little wary of this. Though I'm not sure I know what you mean. Some persons interested in the library have hoped it can be used to generate some specific format. But that format doesn't accommodated all the information required to rebuild and arbitrary C++ data structure so in order to do one ends up coupling the serialization of classes to the archive format - just exactly what the serialization library is designed to avoid. Actually, My current thinking is to add a section to the documentation ( I love my documentation ! Many people have contributed to it by careful reading and criticism) which suggests a transition path from a proprietary format to usage of the serialization library. This transition would be basically
a) make a program which loads all your "old" data in the "old" way. b) serialize the structures.
So I'm skeptical of trying to adjust to "old" proprietary formats with the serialization library.
Well, I believe you are taking too narrow a view here. These 'old' proprietary formats may not be old or replaceable. They are just part of the landscape on many projects. For example, I recently worked on a project where there was a communications protocol between a server and remote computers that are on the other end of radio link. The communications protocols between the server and the remote computers was a binary format that was rather limited and specific -- about 4 message types. The physical level protocol isn't a standard TCP or UDP b/c the hardware doesn't support this sort of protocol (for good reason). There are little details like a maximum message size which means that continuation packets need to be generated. When messages come to the server from the remote computers it has to modify the message slightly then transmit the data to other servers and clients over TCP. Bottom line -- it was very handy to have a different message format on the TCP side as compared to the proprietary side. Of course in the servers and clients we had c++ objects to represent the messages. Trivially we created classes that had data members for each field in the message protocol. We had a serialization framework and we wrote serialization code for each message type. We serialized the fields in the order that the protocol specified. Then we had 2 archive types -- one to read/write the binary protocol format and one to read/write the internal format. This design works great. There is a clear mapping from the protocol specification to the message classes and all the details of the 'physical' protocol are in a single archive class. Of course if you sent some sort of message down pipe that wasn't of the correct type to go to the remote computers, the serialization part of the framework won't stop you. But we prevented that in other layers of the architecture. In this case, our server to client format could easily use one of the standard archives provided by the library. But the external protocol could not -- hence the need for the ability to write custom archive types that read/write proprietary protocols. As for the design coupling issue, there really isn't a problem. The archive implementation is tied to the serialization library as are the serialized classes. The fact that an arbitrary c++ type won't work with the specialized archive isn't important because other elements of the design can easily prevent this from happening. I believe that Matthew has now demonstrated that archive extension is possible, which I believe is one of the major changes from the first review. That said, any documentation of this process would clearly be a plus. In a typical proprietary archives writing of the version and type numbers will need to be shut off. So I can imagine documenting these sorts of issues. Jeff