I just got Boost.Interprocess working, and it's really an amazing library. Ion, it's obvious you've put a lot of time into this library, and the result is really nice. However, I've had these issues so far: (1) Is there a way to get a managed_shared_memory segment to tell me allocation statistics? What I'd like is something that can tell me. (a) Segment size (b) Allocated memory (c) Most importantly, free memory (2) It's been brought up before, but there seriously needs to be a way to grow managed shared memory segments. I understand that this is really tricky to implement, and probably just as tricky to use safely, but it's an important feature, in my opinion. I guess the main difficulties here are the potential change of base, and atomicity? (3) Regarding priv_open_or_create() in managed_open_or_create_impl.hpp, this implementation creates some small annoyances for me by throwing. I don't have any idea how to tell if POSIX O_CREAT actually created anything or not. However, I'm using Windows, and on Windows, you always can tell, as far as I know, both for file creation and named shared memory object creation. Would it be difficult to avoid throwing for the Windows case?
Hi,
On 5/31/07, Aaron W. LaFramboise
(2) It's been brought up before, but there seriously needs to be a way to grow managed shared memory segments. I understand that this is really tricky to implement, and probably just as tricky to use safely, but it's an important feature, in my opinion. I guess the main difficulties here are the potential change of base, and atomicity?
I have a similar requirement. Something like a pool of shared memory blocks that can work as a stream. Since it is impossible to always grow a reserved chunk of memory at the same base address, you will have to create a pool (or the crude realloc type which beats scalability and performance). IMHO, it is ideal to develop a pool of shared memory each with their own base addresses and implement all the intelligence of traversing, seek and functionalities using an ITERATOR. Iterator can become an abstract interface to the underlying memory. I have implemented a class of my own (not along the BOOST lines) as a proof of concept and left it incomplete (needs debugging and fixing the traversing part - Iterator). with best regards, dhruva -- Dhruva Krishnamurthy Contents reflect my personal views only!
Hi Aaron, Aaron W. LaFramboise wrote:
I just got Boost.Interprocess working, and it's really an amazing library. Ion, it's obvious you've put a lot of time into this library, and the result is really nice.
Thanks for the props!
However, I've had these issues so far:
Let's see...
(1) Is there a way to get a managed_shared_memory segment to tell me allocation statistics? What I'd like is something that can tell me. (a) Segment size
You have "get_size()". Maybe it's not present in the documentation.
(b) Allocated memory (c) Most importantly, free memory
I would need to add that information to memory allocation algorithms, but I don't see them problematic. I'm just finishing some new tweaks for the library so I can try to add this ASAP. Just remember that knowing that you have more free memory than the object you want to create does not guarantee success due to: --> fragmentation --> meta-data to be added (name or the object, number of objects) --> alignment issues. Anyway, I think they can be useful for the programmer.
(2) It's been brought up before, but there seriously needs to be a way to grow managed shared memory segments. I understand that this is really tricky to implement, and probably just as tricky to use safely, but it's an important feature, in my opinion. I guess the main difficulties here are the potential change of base, and atomicity?
I can imagine 2 styles to do that: 1- Expand the segment (realloc-like). I don't know if this is supported portable by OS-s. 2- Build a multi-segment shared memory (using a new smart pointer that can point between different segments mapped in different addresses in each process). The problem is that if process A extends the /creates a new shared memory, _all_ processes attached to that memory must remap the memory segments immediatelly _atomically_. Otherwise, process A modifies an object (say, a vector) that it's shared with other processes but it's not reachable by other processes because they haven't remapped their segments. Really _difficult_. I've been thinking about a cooperative approach, so that processes, before using shared resources, must acquire a lock that would grow the memory if necessary. But does not seem very confortable might have performance issues and it's error-prone. Still, the multi-segment approach can be useful for a single-process approach when using memory mapped files. A process can create a giant growable and persistent memory database creating more files and mapping them. But multi-processing makes growing segments really hard. If someone has any idea or wants to help on this...
(3) Regarding priv_open_or_create() in managed_open_or_create_impl.hpp, this implementation creates some small annoyances for me by throwing. I don't have any idea how to tell if POSIX O_CREAT actually created anything or not. However, I'm using Windows, and on Windows, you always can tell, as far as I know, both for file creation and named shared memory object creation. Would it be difficult to avoid throwing for the Windows case?
When Shmem was reviewed reviewers wanted to avoid the two phase construction and use exceptions. With O_CREAT there is no portable way (if I'm wrong, please correct me) to know if the file/segment was created or opened, so I wanted to be portable. I'm afraid that you will need to use create_only or open_only in a loop (brute-force approach) to be sure if the segment was created or opened. Believe me, it's not that ugly ;-) Now, seriously, I find that's a missing feature but I can't think a portable way to do that unless I do the brute-force trick (which might be the answer). Let me think about it. Regards, Ion
Ion Gaztañaga wrote:
Still, the multi-segment approach can be useful for a single-process approach when using memory mapped files. A process can create a giant growable and persistent memory database creating more files and mapping them. But multi-processing makes growing segments really hard. If someone has any idea or wants to help on this...
In a multi-segment implementation, would an object like a vector be able to span multiple segments? I think not, but I'm not sure. If general growing of managed segments is too hard, maybe an easier problem could be solved, like making shared memory containers that have special support for growing. But I guess you still will need locking, because any time a container is grown, it potentially invalidates everything shared in every process.
When Shmem was reviewed reviewers wanted to avoid the two phase construction and use exceptions. With O_CREAT there is no portable way (if I'm wrong, please correct me) to know if the file/segment was
I agree with you about O_CREAT. But what about on Windows, with CreateFileMapping? "If the object exists before the function call, the function returns a handle to the existing object (with its current size, not the specified size), and GetLastError returns ERROR_ALREADY_EXISTS." It seems the API has a little bit of a POSIX bias. :)
Aaron W. LaFramboise wrote:
In a multi-segment implementation, would an object like a vector be able to span multiple segments? I think not, but I'm not sure.
No, because the memory must be contiguous. But other structures like deque/map-set/list could be implemented.
If general growing of managed segments is too hard, maybe an easier problem could be solved, like making shared memory containers that have special support for growing.
I think you reach the same problem. You need to allocate more pages from the OS and map them atomically in every process sharing those pages.
I agree with you about O_CREAT. But what about on Windows, with CreateFileMapping? "If the object exists before the function call, the function returns a handle to the existing object (with its current size, not the specified size), and GetLastError returns ERROR_ALREADY_EXISTS."
It seems the API has a little bit of a POSIX bias. :)
It's not that I like POSIX, it's just it was easier to emulate POSIX for Windows than doing the inverse. Shmem had a Windows approach but shared memory lifetime was a real issue, so I changed to a POSIX-like approach. I've tried to be portable, but maybe I should add extensions for systems where the programmer can get more information. Thanks for your comments, Ion
participants (3)
-
Aaron W. LaFramboise -
dhruva -
Ion Gaztañaga