James Sutherland wrote:
On Aug 29, 2008, at 12:41 PM, troy d. straszheim wrote:
But it isn't fast. If the necessity of bitshuffling makes it impossible to serialize, say a vector<double> via the optimized array handling, you could easily be talking about a factor of 10 in speed. Showstopper for us, at least. I suppose you could copy the entire buffer, flip all the bits at once, then serialize the flipped buffer, but this also has significant cost, too much for scientific applications where another option exists.
Have you looked at HDF5? It supports metadata, parallel IO, arbitrary data structures, etc. and is designed for HPC applications. It also has native binary format support and will automatically provide cross-platform binary compatibility.
We do use HDF5... With boost.python and pytables plus boost::serializaton backed c++ datastructures one can quite flexibly provide converter/extractor/reducer utilities that get you from a boost::serialization portable binary format to a more 'analysis friendly' hdf5 format. Works great. -t