Testing interest for a shared memory/STL library proposal

Hello to all, My name is Ion Gaztañaga, a telecommunication engineer working in railway industry R&D . I've been lately working on a CUJ article about placing STL containers (and any other object) in shared memory using smart pointers instead of raw pointers as allocator::pointer. The article has been accepted for a future issue and is currently in a editing/reviewing phase. Working in this article, I've created a library/framework, developing some classes that I think can be useful for a new boost library. I would like to know if there is any interested in boost for developing such classes, based or not in my article ideas or code. If there is any interest, I would be glad to present a boostified version of the library/framework for a review proccess. Currently, the framework uses these ideas: -> Relative pointers (smart pointers) for placing them in shared memory. -> OS basic mechanism abstractions: shared-memory creation, process-shared mutex, named semaphore -> Shared memory allocator for dynamic allocation/deallocation of parts of a shared memory segment. -> Named shared memory allocation manager, so that any process can look for an object in shared memory. -> Pooled shared memory allocator for node containers. -> Some STL-like container examples that can be placed in shared memory for systems with no smart-pointer allocator::pointer supporting STLs -> others... The framework works with Win32/Visual.NET 2003, QNX6.x/gcc-2.95.2,Mandrake Linux10.1/gcc-3.4.1 and partially (relative pointers and raw shmemory allocation) with Win32/VisualC++ 6.0. I've put a special full-length article in html and the source code in the file http://ice.prohosting.com/newfunk/boost_shmemSTL.zip, so that anyone can take a look. Please, you don't need to waste much time in the article, since I just want to know if contained ideas are interesting for boost. Please remember that some classes such as scoped-locks, etc... are in boost but I wanted the framework to be independent of any other library for this article. Of course, this would change in an hypotetic boostified version, since this is only to get an idea. Please, feel free to comment/critisize. Regards, Ion

Hi Ion, | I would like to know if there is any interested in boost for developing | such classes, based or not in my article ideas or code. If there is any | interest, I would be glad to present a boostified version of the | library/framework for a review proccess. Currently, the framework uses these | ideas:
From what I can gather from the docs, then the required changed to the standard library is quite small; maybe you should consider submitting a defect report that says where the standard must be changed in order to support this style of allocators and allocation?
Rewriting all container classes seems to be a quite big task, so maybe its better to change the standard. -Thorsten

From what I can gather from the docs, then the required changed to the standard library is quite small; maybe you should consider submitting a defect report that says where the standard must be changed in order to support this style of allocators and allocation?
Rewriting all container classes seems to be a quite big task, so maybe its better to change the standard.
Yes, I agree that would be nice and it is the way to go. Some implementations, such as Dinkum STL use allocator::pointer as pointer type and they have very minor problems compiling this approach. What I want to make clear is that I don't want to reinvent the STL for boost. I was talking to boostify only helper classes so that I could for example write in one process: //----------------------------------------------------------------- //Create shared memory allocator segment shm_named_alloc segment segment.create("/MyShMem", 10000); //Create an array of ints named "MyInts" in shared memory segment "/MyShaMem" int *ptr = segment.named_alloc<int>("MyInts", 10/*allocate an array*/, 0/*ctor parameter*/) //Fill ints for(int i = 0, i < 10){ ptr[i] = i; } std::fill(ptr, ptr+10, 1); And in other process write: //----------------------------------------------------------------- shm_named_alloc segment //Open allocator segment.open("/MyShMem", 10000); //Find "MyInts" int *ptr = segment.named_find<int>("MyInts") //sort std::sort(ptr, ptr+10); //Delete "MyInts" array from segment segment.named_delete<int>("MyInts"); //----------------------------------------------------------------- To get this, I propose a library with the following classes: -> Relative pointer (a new smart pointer). -> shared memory creation class, process shared mutex, process shared reentrant mutex, named semaphore -> shared memory manager class (uses shared memory segment as a pool for allowing malloc/free like operations) -> named shared memory manager (so that the user can store any class associated with a name in the shared memory segment) -> a STL allocator providing shared memory allocator for STL-like containers If someone wants to create its own container can use these elements but I don't plan to repeat all standard containers for boost. But I think that placing classes in shared memory can be interesting for efficient inter-process communication. Maybe I've not explained very well my goals. My article is about STL containers, but my proposal is for placing any object in shared memory and provide some allocators (normal and pooled) so that users can create their own containers if they need that. I hope this explanation can correct my previous post. Regards, Ion

Ion Gaztañaga wrote: [snip]
example write in one process:
[snip]
And in other process write:
[snip] This is great! I would find this kind of functionality extremely useful. My requirements are (currently) a little simpler that yours (the "clients" of the shared memory segment have read-only access for reading pre-cached data). Like many others I've grown tired of having to do this sort of thing manually! Michael

My requirements are (currently) a little simpler that yours (the "clients" of the shared memory segment have read-only access for reading pre-cached data). Like many others I've grown tired of having to do this sort of thing manually!
Michael
I've also talked with Francis Andre who is also very interested in this library for his work. Although so far only two people consider this library interesting, I will present a proposal so that people can see what the library can do. I aim for a simple, small library, targeted for Visual 2003 .NET and gcc 3.4.1 for the moment. If the library is accepted, I will start a port to other systems. So if anyone is not against this, I will present a proposal shortly. First I have to read deeply all steps required for a first version. I am also tired of doing shared memory stuff by hand. Regards, Ion

"Ion Gaztañaga" <ion_g_m@terra.es> wrote in message news:004601c4c80f$0a166140$02001aac@amd1ghzwinxp...
library for his work. Although so far only two people consider this library interesting, I will present a proposal so that people can see what the library can do.
I'm interested, too. I just don't have time to think about it. ;-)
Regards,
Ion
Jonathan

Hi Ion.
Although so far only two people consider this library interesting.
Actually I'd be most interested in such a library. I at the moment use my own solution for the same problem. I ran into the same trouble you describe in your documentation but did not get a chance to dig in so deep as I had a real-world problem to solve in a real-world time frame :-))). I did create a SharedMemory object which works, together with my SharedMemoryId objects which are actually offset based pointers that can be stored inside the SharedMemory area. (Can't tell you how much I hate it when I get sloppy and store a standard C++ reference in there only to find it is not usable from another process. :-))) My SharedMemoryId objects (actually called RepositoryId and WriteableRepositoryId as the my model is pretty domain specific) use an offset from the start of their SharedMemory area and when you use them you need to specify which SharedMemory object they relate to. I like your design with an offset from their own address much better as that would allow my * and -> operators to not rely on one global SharedMemory object (which was enough in my case). And I guess having a larger generic ptr that can be 0, and a smaller non-null one is a small-enough price to pay (much better than my relying on a global shared memory object when using standard C++ dereferencing operators). I did not implement any smart memory allocation schemes as my shared memory space was only allocated and never needed to be deallocated. I use a placement new operator (as in 'new (someSharedMemoryArea) int(5)') which just stores the object on the next available location in the shared memory area, and the memory allocation information is not shared so only one user allocates the memory after which it allows read-only access only. Unfortunately this also prevents me from directly using any 'smarter' data structures there, like strings or vectors, that sometimes need to reallocate their storage space. As for the standard C++ library containers - I tried the allocator approach and ran into the same allocator problems you encountered. In the end I just needed a list structure so I rolled out my own and made it an STL compatible container so that it can be used with standard C++ library algorithms. Hope you see you're not alone out there... like I thought I was when I did this... :-). If you need any help please just holler... I'll see what I can do... When I get the time I'll try to use your library in my project so you'll have another real-world test user application available (Windows/MS Visual Studio .NET 2003, and will be ported over to *nix/gcc some time next year - so if the library is portable its a major bonus as otherwise I'd have to rewrite my own). Many thanks for all the effort you put in this so far. Certainly looks promising. Best regards, Jurko Gospodnetic

I would like to know if there is any interested in boost for developing such classes, based or not in my article ideas or code. If there is any interest, I would be glad to present a boostified version of the library/framework for a review proccess. Currently, the framework uses these ideas:
Yes, me too, I am very interested by such library if it could provide, in addition to relative pointer, a STLAllocator that is working with "raw pointer" allocator scheme. By the way, there will be a small additional problem to solve for the library to be complete: the reference of shared objects placed in the shared memory with virtual functions. As such, the smart pointer will have to update the virtual table on each reference to the shared object before returing back the "raw pointer" on the object to the caller!! (unless the underlying OS garantee to map each instance of a shared library at the same base address of each process...Ugh...) Francis
Hello to all,
My name is Ion Gaztañaga, a telecommunication engineer working in railway industry R&D . I've been lately working on a CUJ article about placing STL containers (and any other object) in shared memory using smart pointers instead of raw pointers as allocator::pointer.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
participants (6)
-
Francis ANDRE
-
Ion Gaztañaga
-
Jonathan Turkanis
-
Jurko Gospodnetic
-
Michael van der Westhuizen
-
Thorsten Ottosen