On Fri, May 30, 2008 at 10:02:51AM -0400, Jason Sachs wrote:
Alternatively, why not use files instead?
I am using an HDF5 file to log data as a physical store, but it doesn't handle concurrent access + I am looking for a fast way (in addition to the log file) to share some acquired data between processes.
Could you maybe use a raw memory-mapped file instead, and convert it to HDF5 off-line? You would use memory mapping to share data, and ordinary file write calls to append data to the file. (I understood that you're working on 32bits.. then you'd have to do some "windowing" over a possibly file > 2GB). This would at least solve your swap problem because the file itself *is* the backing store for its own mapping (well, if the mapping is shared so that modifications don't create anonymous COW pages which in turn need swap) - no additional swap needed.