[interprocess] Memory mapped file based allocator
Hi, I am looking for a memory mapped file based allocator that does not map the whole file into memory. The size may be huge and I would not want to map all the contents into process memory. Since my read and write access are sequential, I would like the mapping to happen on demand (based on access violation trying to read an unmapped region of memory). Is there something like that in Interprocess/Boost (since the implementation needs to work both on M$ and GNU/Linux)? regards, dhruva Bollywood news, movie reviews, film trailers and more! Go to http://in.movies.yahoo.com/
dhruva wrote:
Hi, I am looking for a memory mapped file based allocator that does not map the whole file into memory. The size may be huge and I would not want to map all the contents into process memory. Since my read and write access are sequential, I would like the mapping to happen on demand (based on access violation trying to read an unmapped region of memory). Is there something like that in Interprocess/Boost (since the implementation needs to work both on M$ and GNU/Linux)?
No there is not on-demand mapping based on access violation. That would be really difficult to implement.
regards, dhruva
Regards, Ion
On Mon, Mar 9, 2009 at 6:34 AM, dhruva <dhruva@ymail.com> wrote:
Hi, I am looking for a memory mapped file based allocator that does not map the whole file into memory. The size may be huge and I would not want to map all the contents into process memory. Since my read and write access are sequential, I would like the mapping to happen on demand (based on access violation trying to read an unmapped region of memory). Is there something like that in Interprocess/Boost (since the implementation needs to work both on M$ and GNU/Linux)?
regards, dhruva
One solution (which I've used with files of gigabyte size) that you could consider is writing a class to encapsulate memory mapped files that exposes the file contents only through a custom iterator class. These could work together to map and unmap portions of the file as the iterator position crosses 'page' boundaries. This would give you something close to on-demand paging. Stuart Dootson
Hello, ----- Original Message ----
From: Stuart Dootson <stuart.dootson@gmail.com>
I am looking for a memory mapped file based allocator that does not map the whole file into memory. The size may be huge and I would not want to map all the
One solution (which I've used with files of gigabyte size) that you could consider is writing a class to encapsulate memory mapped files that exposes the file contents only through a custom iterator class. These could work together to map and unmap portions of the file as the iterator position crosses 'page' boundaries. This would give you something close to on-demand paging.
Thank you for the suggestion. I have implemented it on windows using SEH (structured exception handling). Is there something similar in the UNIX world. Adding a signal handler will be for the whole process, I need to check if there is a way to add a signal handler for a specific thread that is doing the read/write and perform the unmap and map sequence to keep moving the read/write window. This would be a great feature to have it tucked somewhere in BOOST. Reading/writing large files is quite common. If I succeed in putting in a solution that works on M$ and NIX world, I will post it. -dhruva Connect with friends all over the world. Get Yahoo! India Messenger at http://in.messenger.yahoo.com/?wm=n/
On Wed, Mar 11, 2009 at 3:41 AM, dhruva <dhruva@ymail.com> wrote:
Hello,
----- Original Message ----
From: Stuart Dootson <stuart.dootson@gmail.com>
I am looking for a memory mapped file based allocator that does not map the whole file into memory. The size may be huge and I would not want to map all the
One solution (which I've used with files of gigabyte size) that you could consider is writing a class to encapsulate memory mapped files that exposes the file contents only through a custom iterator class. These could work together to map and unmap portions of the file as the iterator position crosses 'page' boundaries. This would give you something close to on-demand paging.
Thank you for the suggestion. I have implemented it on windows using SEH (structured exception handling). Is there something similar in the UNIX world. Adding a signal handler will be for the whole process, I need to check if there is a way to add a signal handler for a specific thread that is doing the read/write and perform the unmap and map sequence to keep moving the read/write window. This would be a great feature to have it tucked somewhere in BOOST. Reading/writing large files is quite common. If I succeed in putting in a solution that works on M$ and NIX world, I will post it.
-dhruva
You don't need to use exceptions at all - in the iterator's 'dereference' method (see the iterator_facade documentation in Boost.Iterator for what that means), check if the item that the iterator references is in the currently mapped region. If not, map a region that does contain that item. Stuart Dootson
On Wed, Mar 11, 2009 at 1:20 AM, Stuart Dootson <stuart.dootson@gmail.com> wrote:
On Wed, Mar 11, 2009 at 3:41 AM, dhruva <dhruva@ymail.com> wrote:
----- Original Message ----
From: Stuart Dootson <stuart.dootson@gmail.com>
I am looking for a memory mapped file based allocator that does not map the whole file into memory. The size may be huge and I would not want to map all the
One solution (which I've used with files of gigabyte size) that you could consider is writing a class to encapsulate memory mapped files that exposes the file contents only through a custom iterator class. These could work together to map and unmap portions of the file as the iterator position crosses 'page' boundaries. This would give you something close to on-demand paging.
Thank you for the suggestion. I have implemented it on windows using SEH (structured exception handling). Is there something similar in the UNIX world. Adding a signal handler will be for the whole process, I need to check if there is a way to add a signal handler for a specific thread that is doing the read/write and perform the unmap and map sequence to keep moving the read/write window. This would be a great feature to have it tucked somewhere in BOOST. Reading/writing large files is quite common. If I succeed in putting in a solution that works on M$ and NIX world, I will post it.
-dhruva
You don't need to use exceptions at all - in the iterator's 'dereference' method (see the iterator_facade documentation in Boost.Iterator for what that means), check if the item that the iterator references is in the currently mapped region. If not, map a region that does contain that item.
I suppose the motivation for using structured exceptions is that you don't have to do this check, which is presumably faster. But structured exception handling itself introduces overhead (compared to "regular" exception handling) so usually it is not a good idea to require it. Emil Dotchevski Reverge Studios, Inc. http://www.revergestudios.com/reblog/index.php?n=ReCode
participants (4)
-
dhruva -
Emil Dotchevski -
Ion Gaztañaga -
Stuart Dootson