On Aug 29, 2008, at 12:41 PM, troy d. straszheim wrote:
Rainer Deyke wrote:
Johan Råde wrote:
I see three options for dealing with the endianness issue: 1. make all archives big-endian 2. make all archives little-endian 3. use the native format when saving, and put an endianness flag in the archive.
1 is inefficient when moving data between little-endian platforms 2 is inefficient when moving data between big-endian platforms So 3 should be most efficient. Is there an easy way of storing an endianness-flag in an archive? I don't think performance should be the overriding concern, especially since byte-shuffling is very fast.
But it isn't fast. If the necessity of bitshuffling makes it impossible to serialize, say a vector<double> via the optimized array handling, you could easily be talking about a factor of 10 in speed. Showstopper for us, at least. I suppose you could copy the entire buffer, flip all the bits at once, then serialize the flipped buffer, but this also has significant cost, too much for scientific applications where another option exists.
Have you looked at HDF5? It supports metadata, parallel IO, arbitrary data structures, etc. and is designed for HPC applications. It also has native binary format support and will automatically provide cross- platform binary compatibility. It will likely require a bit more code instrumentation than boost serialization, but may be worth it if performance is key...