[shmem-interprocess] Shmem 0.93 and update on Interprocess
Hi to all, I've just uploaded to Boost Vault a new Shmem bugfix version (0.93) with bugfixes to make Shmem MinGw friendly (props to Jan Stetka). Library and documentation download: http://boost-consulting.com/vault/index.php?&direction=0&order=&directory=Memory Online documentation: http://ice.prohosting.com/newfunk/boost/libs/shmem/ //////////////////////////////////////////////////////////// Update on Interprocess: -> Due to Shmem problems with shared memory/named mutex/name semaphore lifetime between POSIX/Windows. I've decided to change Shmem approach (emulation of Windows lifetime behavior in POSIX with reference counting) to a POSIX lifetime (explicit unlinking shared memory). This means that I have to rework all main classes to achieve POSIX persistence semantics in Windows even when a process crashes. This will require using memory mapped files to emulate shared memory (with some tricks to avoid performance problems with file-memory synchronization if it's possible). I think that this approach will be much more portable than the Shmem approach. -> I'm also experimenting with a new allocator with expand-in place capabilities to allow realloc-like features in shared memory containers. The first results are encouraging both in size savings and speed improvements. This expand in place features are based on a paper I plan to present as a ISO C++ proposal based on N1953. You can find the current draft here, if you want to make some comments: http://ice.prohosting.com/newfunk/proposals/nXXXX%20Improving%20STL%20Alloca... -> I'm also experimenting with a new reduced mutex interface (basically condensing shared_mutex, shared_try_mutex and shared_timed_mutex in a single interprocess_mutex type. Something in the line of the draft "Simplifying And Extending Mutex and Scoped Lock Types For C++ Multi-Threading Library" that you can find here: http://ice.prohosting.com/newfunk/proposals/ I'm also trying to implement a process-shared version of Howard Hinnant's upgradable_mutex class described here: http://home.twcny.rr.com/hinnant/cpp_extensions/rw_perf.html http://home.twcny.rr.com/hinnant/cpp_extensions/threads_move.html I still have some bugs in my upgradable_mutex implementation that I plan to solve soon. I think that this enhazed rw mutex is a winner for Interprocess. -> I'm adding a new mutex type: file_lock. The classic UNIX fcntl / Windows FileLockEx file locking is an interesting addition to Interprocess since both in POSIX and Windows, the lock is automatically unlocked by the OS when the process ends/crashes. This can help problematic Shmem atomic creation+initialization functions if the process crashes executing the atomic code. File locking is also many times easier to use than named mutexes or shared memory + process shared mutexes, specially for simple locking. That's all. Thanks for your patience! Regards, Ion
Ion Gaztan~aga wrote:
Hi to all,
I've just uploaded to Boost Vault a new Shmem bugfix version (0.93) with bugfixes to make Shmem MinGw friendly (props to Jan Stetka).
Quick question -- as I recall interprocess was accepted a few months back -- is there something that's preventing it from going into cvs? Once it is there we get the benefit of regression tests and other developers can easily see it. Jeff
Quick question -- as I recall interprocess was accepted a few months back -- is there something that's preventing it from going into cvs? Once it is there we get the benefit of regression tests and other developers can easily see it.
The problem is that it's very unstable, because when Shmem was accepted, a lot of changes were requested and Shmem's POSIX/Win32 portability problems require reworking a lot of things. I'm trying to add also some new features. Interprocess has also to offer process-shared mutexes, condition variables, etc... I would like to offer the same interface as the proposed C++ standard interface (currently Boost.Threads, but some are proposing simplifying it, reworking scoped locks, etc...). Currently there is no documentation so it wouldn't be very usable (well, there are some tests to know how it works). I could upload it to CVS if any developer wants to start testing it and get suggestions. What do you think, it should go first to boost-sandbox, or directly to the boost main repository? Regards, Ion
Ion Gaztañaga wrote:
Quick question -- as I recall interprocess was accepted a few months back -- is there something that's preventing it from going into cvs? Once it is there we get the benefit of regression tests and other developers can easily see it.
The problem is that it's very unstable, because when Shmem was accepted, a lot of changes were requested and Shmem's POSIX/Win32 portability problems require reworking a lot of things. I'm trying to add also some new features.
I think you should separate the two things. Get the required changes in and make the initial version of the library available, then work on new features.
Interprocess has also to offer process-shared mutexes, condition variables, etc... I would like to offer the same interface as the proposed C++ standard interface (currently Boost.Threads, but some are proposing simplifying it, reworking scoped locks, etc...).
Currently there is no documentation so it wouldn't be very usable (well, there are some tests to know how it works). I could upload it to CVS if any developer wants to start testing it and get suggestions. What do you think, it should go first to boost-sandbox, or directly to the boost main repository?
As soon as you upload and register into the regression system you will get to see how it works on a variety of platforms (well, at least after 1.34 ships you will). Folks interested in a particular compiler/platform may start looking at it an helping you port. And beyond the formal regression tests, others of us regularly update, rebuild, and run regression tests. I don't tend to mess with anything in the sandbox unless I really, really want to use it for something (in fact we really need to clean the old stuff out). Most new Boost developers don't realize that the process of getting in CVS and the regression test system will take a couple months... I just worry about you falling into the circular_buffer trap. A perfectly usable and accepted library that 2 years later because of some issues the author doesn't now have time to address. It's a loss to the community and a shame. I don't have an immediate need for inter-process, but there's alot of concurrency/multi-process code in progress in Boost for SOC and otherwise. Interprocess is a piece of that puzzle -- so I'd like to see more eyes and testing on it sooner rather than later. Getting it into CVS accomplishes the first step. Jeff
I think you should separate the two things. Get the required changes in and make the initial version of the library available, then work on new features.
Right. But I should state that changes in Shmem/Interprocess are quite problematic since binary compatibility is required in all the classes placed in shared memory/memory mapped files (mutex, condition, containers, allocators, rw_locks...). That means that a change in those classes (for example, adding a more efficient version of a synchronization object) will mean that the user must recompile all the processes using Shmem/Interprocess. So I would like to minimize breaking changes, so the first official Interprocess should be pretty stable.
As soon as you upload and register into the regression system you will get to see how it works on a variety of platforms (well, at least after 1.34 ships you will). Folks interested in a particular compiler/platform may start looking at it an helping you port. And beyond the formal regression tests, others of us regularly update, rebuild, and run regression tests. I don't tend to mess with anything in the sandbox unless I really, really want to use it for something (in fact we really need to clean the old stuff out). Most new Boost developers don't realize that the process of getting in CVS and the regression test system will take a couple months...
All right. I will request write access in CVS and I will start learning the regression system.
I just worry about you falling into the circular_buffer trap. A perfectly usable and accepted library that 2 years later because of some issues the author doesn't now have time to address. It's a loss to the community and a shame. I don't have an immediate need for inter-process, but there's alot of concurrency/multi-process code in progress in Boost for SOC and otherwise. Interprocess is a piece of that puzzle -- so I'd like to see more eyes and testing on it sooner rather than later. Getting it into CVS accomplishes the first step.
Agreed. Ion
Hi, Its good that shmem/interprocess is moving on. Concerning MinGW offset_ptr.hpp is still needed before an instance of shmem::map is used. I will give more detail if necessary Thanks
Jan Stetka(e)k dio:
Hi, Its good that shmem/interprocess is moving on. Concerning MinGW offset_ptr.hpp is still needed before an instance of shmem::map is used. I will give more detail if necessary Thanks
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
I will look at it again. These problems are caused by ADL lookup issues, and my big ADL ignorance. In my hands, ADL is more a problem than a solution with just simple swap functions, because I start having include order issues. I've been googling around but I can find any interesting thing related to "proper" ADL use, I only find some simple ADL examples. Do we have any recommendation regarding ADL in Boost? I've seen some threads related to mpl vs. ADL lookup issues, but they are from 2002. Any ADL expert willing to help? Regards, Ion
Ion Gaztañaga wrote:
Jan Stetka(e)k dio:
offset_ptr.hpp is still needed before an instance of shmem::map is used.
I will look at it again. These problems are caused by ADL lookup issues, and my big ADL ignorance. In my hands, ADL is more a problem than a solution with just simple swap functions, because I start having include order issues.
Can you identify what you mean by simple swap functions and where they are used in shmem. I would agree that the differences between compilers is caused by the way they implement ADL but there will be a header ordering that satisfies them all. Thanks
Ion Gaztañaga wrote:
Update on Interprocess:
-> Due to Shmem problems with shared memory/named mutex/name semaphore lifetime between POSIX/Windows. I've decided to change Shmem approach (emulation of Windows lifetime behavior in POSIX with reference counting) to a POSIX lifetime (explicit unlinking shared memory).
This means that I have to rework all main classes to achieve POSIX persistence semantics in Windows even when a process crashes. This will require using memory mapped files to emulate shared memory (with some tricks to avoid performance problems with file-memory synchronization if it's possible). I think that this approach will be much more portable than the Shmem approach.
Well that move makes worries me. The use I currently put shmem/interprocess to is for high-speed IPC communications. Could you explain what the potential performance impact is with this change. -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim - grafikrobot/yahoo
This means that I have to rework all main classes to achieve POSIX persistence semantics in Windows even when a process crashes. This will require using memory mapped files to emulate shared memory (with some tricks to avoid performance problems with file-memory synchronization if it's possible). I think that this approach will be much more portable than the Shmem approach.
Well that move makes worries me. The use I currently put shmem/interprocess to is for high-speed IPC communications. Could you explain what the potential performance impact is with this change.
This means that the OS will synchronize file and memory contents from time to time, since shared memory will be implemented with memory mapped files. However, I will create those files emulating shared memory with the FILE_ATTRIBUTE_TEMPORARY attribute, that according to MSDN: "A file is being used for temporary storage. File systems avoid writing data back to mass storage if sufficient cache memory is available, because an application deletes a temporary file after a handle is closed. In that case, the system can entirely avoid writing the data. Otherwise, the data is written after the handle is closed." I think that with this attribute the overhead will be minimal. I've just got write access to CVS so I will upload Interprocess so that you can test if the performance is enough. Regards, Ion
On Mon, 12 Jun 2006 07:04:26 +0200 Ion Gaztañaga <igaztanaga@gmail.com> wrote:
This means that the OS will synchronize file and memory contents from time to time, since shared memory will be implemented with memory mapped files. However, I will create those files emulating shared memory with the FILE_ATTRIBUTE_TEMPORARY attribute, that according to MSDN:
Why not use the best native support for each platform, falling back onto a more common and portable approach when better native support is not provided?
Jody Hagins(e)k dio:
On Mon, 12 Jun 2006 07:04:26 +0200 Ion Gaztañaga <igaztanaga@gmail.com> wrote:
This means that the OS will synchronize file and memory contents from time to time, since shared memory will be implemented with memory mapped files. However, I will create those files emulating shared memory with the FILE_ATTRIBUTE_TEMPORARY attribute, that according to MSDN:
Why not use the best native support for each platform, falling back onto a more common and portable approach when better native support is not provided?
That would mean that shared memory in POSIX and Win32 would have different lifetimes and your program wouldn't be portable, and you can use the native API for this. My intention was to present a portable solution that can be (maybe) standardized. But maybe I could present additional non-portable classes for Windows if that's useful for Windows programmers. Regards, Ion
participants (5)
-
Ion Gaztañaga -
Jan Stetka -
Jeff Garland -
Jody Hagins -
Rene Rivera