Another thing that just occurred to me: allocate huge SHM segment but don't commit it immediately; let the os allocate the swap on-demand. Now, when you run out of swap, the OS should signal the condition somehow (structured exception handling? something else?). You can catch the condition and take appropriate action, whatever that might be.

See ticket #1975: http://svn.boost.org/trac/boost/ticket/1975