On Fri, May 30, 2008 at 12:25:25PM -0400, Jason Sachs wrote:
Could you maybe use a raw memory-mapped file instead, and convert it to HDF5 off-line?
well, technically yes, but for robustness reasons I want to decouple
ok, i just wanted to know more about your problem.
Boost::interprocess to make the segment size equal to M. If I do this then my resource usage in the page file (or on disk if I use a memory-mapped file) is N*M which is much higher than I need. (I
Which is much higher than you need on the average. I fail to see why having a 10GB, or even a 20GB, swap-file is a problem for you. Too much of a hassle to configure it on all workstations?
of these going on at once (though usually just one or two). On my own computer I have increased my max swap file size from 3GB to 7GB (so the hard limit is somewhat adjustable), though it didn't take effect until I restarted my PC. I'm going to be using my programs on several computers + it seems silly to have to go to this extent.
Ok, and you'll run your job on a machine with e.g. 1GB of swap[*], and this particular instance will need 4GB of swap. What will happen when the allocation fails? Note that growing the SHM segment in small chunks will not help you with insufficient virtual memory, so you might as well allocate M*N at once and exit immediately if the memory is not available. [*] I'm using "swap" somewhat imprecisely to refer to total virtual memory (RAM + swap). Next-best solution: use binary search to find the maximum size you can allocate and use that instead of M*N. Neither way is particularly friendly towards other processes on the machine (I assumed that you were running the jobs on dedicated machines), but is least painful. Which is least expensive: your time spent developing multi- chunk SHM management or just allocating a big chunk and reconfiguring all computers *once*? (I'm sorry, I'm very pragmatic, and I don't seem to have enough info to really understand why you're making such a fuss over the swap size issue. I'm afraid I can't offer you any further suggestions, since I consider this a non-problem unless you have further constraints.)