In reply to my review comments, Ion Gaztanaga wrote 2/17/06:
A reference count of create + open - close is maintained for shared memory objects. If this count reaches zero, the shared memory object is unlinked (at least in the posix version). The lack of any documented unlink mechanism might lead one to guess that something like this is going on, and is documented by the last paragraph of this section ("When all processes ... close ..., the shared memory segment is destroyed"). A bit more emphasis might be useful here. On the other hand, I'm actually not convinced this is a good idea. It is certainly fragile, in that a program which crashes (for whatever reason) won't close any shared memory objects it has open, resulting in the reference tracking getting messed up.
You can have problems in POSIX systems, but I couldn't find a better way to implement this. If a program crashes there is no way control anything. The unique solution would be to provide a function that destroys all named objects that I can register with every creation so that you can catch signals and call that functions to free all objects. In windows the OS frees the resources automatically. For standard C++ IPC mechanisms I would request OS help for program crashes, just like heap memory is freed automatically.
For example, one couldn't start up a process which parses some data into an in-memory format that it records in shared memory and then exits, with other programs saving parsing time by just getting the information from shared memory. This doesn't work if those other programs don't get around to opening the shared memory before the parser program exits.
I would like to implement the POSIX-like behaviour in windows, but that would require some permanent store that or a server/ process/service that serves named IPC mechanisms windows. You can use memory mapped files for this behaviour. Take in care that POSIX unlink mechanism is also complex so that if a process unlinks the shared memory, if another process can create a new shared memory with the old name while older processes are still attached to the old shared memory. I need help from POSIX experts.
First let me make sure I understand what is going on here. On Windows, the shmem library is presently using create_file_mapping and open_file_mapping, while on POSIX systems it is using shm_open. An object created with create_file_mapping exists until there are no references, assuming I'm understanding what you've said. (I don't have (easy) access to Windows API documentation, so can't go look up this information. Please correct me if you see any confusion.) A POSIX shared memory object exists from the time it is created until it is unlinked and all references are gone (or the system is rebooted). (It becomes inaccessible to further shm_open calls if unlinked, but remains open to processes that had already opened it.) What the present library implementation is trying to do with this reference count mechanism is to emulate the Windows behavior on POSIX systems. Unfortunately, as has been noted, that emulation really isn't very reliable in the face of ill-behaved (i.e. crashing) clients. And I'm pretty sure there isn't a solution to that problem, at least not with the shm_open &etc API. First question: Why not use the shm_open interface on Windows? One possible answer would be that the Windows POSIX support doesn't include the shm_xxx API. And that might even be the answer, since some web searches have led me to suspect that Windows only supports the SysV shared memory API. Which leads to Second question: Why use different implementations on different platforms? Why not use the (admittedly somewhat clumsy to use directly) SysV shared memory API, which is pretty widely supported? It's a little more painful to use, but that's an implementation detail that won't be exposed to library clients. One issue might be kernel limits on min/max size and number of objects; some of the references I have make mention of such but don't provide much detail. On my stock configured SuSE9.3 machine I see max number of identifiers 4096 and max size 33554432 (32768 * 1024). SysV shared memory also doesn't support resizing, which is available when using the shm_open API (I don't know whether that feature is available with the create_file_mapping API). I don't know if that actually matters for this library. Note that one of the capabilities of SysV shared memory is obtaining the current number of attaches. I don't know if it reliably gets decremented when an attached process exits without explicitly detaching though; I would hope so, but haven't verified it. If so, that could provide a correct reference count implementation. Note also that the size is also recorded, which I think would eliminate the need for the header in the current shared_memory implementation. Hm. If the SysV API provides a correct reference count implementation, and is widely supported, including on Windows, then using that API would permit either lifetime behavior to be implemented (possibly even configurable). Of course, we have yet to have the discussion of which behavior is actually preferable. But it is certainly the case that the present situation, where the behavior is not documented and is buggy on non-Windows platforms, is not ideal.
"Kim Barrett" wrote:
A POSIX shared memory object exists from the time it is created until it is unlinked and all references are gone (or the system is rebooted). (It becomes inaccessible to further shm_open calls if unlinked, but remains open to processes that had already opened it.)
What the present library implementation is trying to do with this reference count mechanism is to emulate the Windows behavior on POSIX systems. Unfortunately, as has been noted, that emulation really isn't very reliable in the face of ill-behaved (i.e. crashing) clients. And I'm pretty sure there isn't a solution to that problem, at least not with the shm_open &etc API.
The API may provide function destroy_old_shmem_and_create_new_one(). With enough of access rights and proper lifetime management (necessary for this kind of applications anyway) this should solve both development phase and crashing clients.
First question: Why not use the shm_open interface on Windows? One possible answer would be that the Windows POSIX support doesn't include the shm_xxx API. And that might even be the answer, since some web searches have led me to suspect that Windows only supports the SysV shared memory API. Which leads to
Second question: Why use different implementations on different platforms? Why not use the (admittedly somewhat clumsy to use directly) SysV shared memory API, which is pretty widely supported?
1. Latest Windows do not support POSIX interface, unless I misread something. 2. Win32 security needs to be taken into account for this library and this requires native API. /Pavel
From: "Pavel Vozenilek" <pavel_vozenilek@hotmail.com> Date: Fri, 10 Mar 2006 01:07:52 +0100
"Kim Barrett" wrote:
What the present library implementation is trying to do with this reference count mechanism is to emulate the Windows behavior on POSIX systems. Unfortunately, as has been noted, that emulation really isn't very reliable in the face of ill-behaved (i.e. crashing) clients. And I'm pretty sure there isn't a solution to that problem, at least not with the shm_open &etc API.
The API may provide function destroy_old_shmem_and_create_new_one().
With enough of access rights and proper lifetime management (necessary for this kind of applications anyway) this should solve both development phase and crashing clients.
I don't think that actually helps at present. For one thing, a Windows developer will never even notice the problem that this proposed function is intended to address, and may well produce a system design which is difficult to port to a POSIX platform because of that. That would defeat some of the purpose of using this library. An example would be a system which used the existing create_or_open variants. If one designed a system around those and developed it on Windows, one would be in for a rude shock when porting to a POSIX system. (It might be argued that the create_or_open variants are just a bad idea because of the problem of distinguishing between a live and a zombie shared object, but eliminating them would prevent access to existing OS facilities.) The problem is that, when correctly functioning, the reference counting mechanism can actually lead one to adopt an approach to lifetime management that simply doesn't work in the face of a broken reference counting implementation. It might be that this can be somewhat ameliorated through documentation, but it would still be a potential portability trap.
First question: Why not use the shm_open interface on Windows? One possible answer would be that the Windows POSIX support doesn't include the shm_xxx API. And that might even be the answer, since some web searches have led me to suspect that Windows only supports the SysV shared memory API. Which leads to
Second question: Why use different implementations on different platforms? Why not use the (admittedly somewhat clumsy to use directly) SysV shared memory API, which is pretty widely supported?
1. Latest Windows do not support POSIX interface, unless I misread something.
Yes, I think that's correct. But I think Windows *does* support the SysV shared memory API.
2. Win32 security needs to be taken into account for this library and this requires native API.
Security needs to be taken into account for any platform, and that's a real weaknesses of the present library interface.
"Kim Barrett" wrote:
1. Latest Windows do not support POSIX interface, unless I misread something.
Yes, I think that's correct. But I think Windows *does* support the SysV shared memory API.
Searching for this I found: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnucmg/html... Windows do provide Posix personality and in addition another company (Interix, those who leaked Windows source code) added even "more comprehensive" Unix layer. However: the Interix stuff needs to be bought and I personally never saw how and where to access the Posix API. (It is not in standard SDK.) I think normal (Windows) programmers would avoid library trying to force them into such exotics. One option may be to use Cygwin (or use similar approach as Cygwin does but this doesn't feel attractive neither. /Pavel
Hi Kim,
First let me make sure I understand what is going on here.
On Windows, the shmem library is presently using create_file_mapping and open_file_mapping, while on POSIX systems it is using shm_open.
Right.
An object created with create_file_mapping exists until there are no references, assuming I'm understanding what you've said. (I don't have (easy) access to Windows API documentation, so can't go look up this information. Please correct me if you see any confusion.)
You can use MSDN (www.msdn.com)
A POSIX shared memory object exists from the time it is created until it is unlinked and all references are gone (or the system is rebooted). (It becomes inaccessible to further shm_open calls if unlinked, but remains open to processes that had already opened it.)
Right. The difference is that in windows, if there's still a process connected to the shared memory, another process can open the same segment. If a UNIX process calls shm_unlink, them the processes that opened the shared memory can still work on it, but if another process can't use that memory, because the name is removed. From OpenGroup site: http://www.opengroup.org/onlinepubs/007908799/xsh/shm_unlink.html "The shm_unlink() function removes the name of the shared memory object named by the string pointed to by name. If one or more references to the shared memory object exist when the object is unlinked, the name is removed before shm_unlink() returns, but the removal of the memory object contents is postponed until all open and map references to the shared memory object have been removed." So if a process calls shm_unlink another process can create new shared memory segment with the same name, but never connect to that old shared memory. This is different from a file unlink behavior, I think.
What the present library implementation is trying to do with this reference count mechanism is to emulate the Windows behavior on POSIX systems.
Right. I find windows behaviour easier to implement with posix (when there is no crash) than emulating posix with windows.
Unfortunately, as has been noted, that emulation really isn't very reliable in the face of ill-behaved (i.e. crashing) clients. And I'm pretty sure there isn't a solution to that problem, at least not with the shm_open &etc API.
I'm sure there is a solution. But every implementation I've checked (apache for example) doesn't solve this. Last time I checked Apache portable runtime tries emulates posix-like behavior but shm_unlink in windows in empty. One option is to use a memory mapped file to emulate posix-like behavior in windows. But it surely will be slower (the OS will dump data to file) and the unlink is not trivial.
First question: Why not use the shm_open interface on Windows? One possible answer would be that the Windows POSIX support doesn't include the shm_xxx API. And that might even be the answer, since some web searches have led me to suspect that Windows only supports the SysV shared memory API. Which leads to
Windows natively does not support posix nor System V. I think that with Unix Services for Windows (previously Interix)you have system V interface. I think Unix Services for Unix is a free download now, but I think you need to use that environment to build a Unix-like application (using gcc compiler). I don't know if those system V functions are available with a normal windows environment.
Of course, we have yet to have the discussion of which behavior is actually preferable. But it is certainly the case that the present situation, where the behavior is not documented and is buggy on non-Windows platforms, is not ideal.
I'm not happy with the behavior of the shared memory. For a not reference-counted implementation you can use memory mapped files. I don't know if this will be slower, though. Ion
On 3/10/06, Ion Gaztañaga <igaztanaga@gmail.com> wrote:
So if a process calls shm_unlink another process can create new shared memory segment with the same name, but never connect to that old shared memory. This is different from a file unlink behavior, I think.
In what way? Isn't this exactly how unlink behaves on files (in Linux)?
Olaf van der Spek wrote:
On 3/10/06, Ion Gaztañaga <igaztanaga@gmail.com> wrote:
So if a process calls shm_unlink another process can create new shared memory segment with the same name, but never connect to that old shared memory. This is different from a file unlink behavior, I think.
In what way? Isn't this exactly how unlink behaves on files (in Linux)?
It is. Most UNIX-a-likes behave this way. -- Bardur Arantsson <bardurREMOVE@THISimada.sdu.dk> <bardurREMOVE@THISscientician.net> - We do try to accomodate our customers, but not being a hotel we find it almost impossible. Model Shop Clerk, 'A Bit of Fry and Laurie'
At 9:25 PM +0100 3/10/06, Ion Gaztañaga wrote:
You can use MSDN (www.msdn.com)
Thanks for the pointer.
A POSIX shared memory object exists from the time it is created until it is unlinked and all references are gone (or the system is rebooted). (It becomes inaccessible to further shm_open calls if unlinked, but remains open to processes that had already opened it.)
Right. The difference is that in windows, if there's still a process connected to the shared memory, another process can open the same segment. If a UNIX process calls shm_unlink, them the processes that opened the shared memory can still work on it, but if another process can't use that memory, because the name is removed.
The other difference is that on Windows, if there are no processes connected to the shared memory, then it goes away. For both POSIX and SysV shared memory, once created, a shared memory exists until unlinked (or system reset). So a process can create and put some stuff in shared memory, and then exit normally, and sometime later another process can come along and still find that same shared memory data. That doesn't work on Windows (or with the current reference counting implementation of the library).
So if a process calls shm_unlink another process can create new shared memory segment with the same name, but never connect to that old shared memory. This is different from a file unlink behavior, I think.
No, this is actually the same a a file unlink behavior. Intentionally so, so that an implementation can basically use files (possibly in a special file system) to implement shared memory. If several processes all have opened a normal file, and that file is then deleted from the file system, the processes still retain their handles on the file and can perform file-io operations on it. Internally the file handles are attached to the inode, which doesn't get reclaimed until there are no references (including from the filesystem).
What the present library implementation is trying to do with this reference count mechanism is to emulate the Windows behavior on POSIX systems.
Right. I find windows behaviour easier to implement with posix (when there is no crash) than emulating posix with windows.
Yes, I can't think of a better way to implement the POSIX behavior on Windows than your suggestions of either memory mapped files or a server program to act as a proxy for the POSIX create/unlink operations. The proxy server is particularly not pretty.
I'm sure there is a solution.
I wish :( There almost could be, by using a server program on POSIX systems. Rough sketch: open connection to server, tell server about opens and closes. When connection is closed, server fixes reference counts for any shared memory opens not yet closed. select() can be used to get notification of connection closes even for crashes. I haven't thought this through carefully, so there may be race conditions in there.
One option is to use a memory mapped file to emulate posix-like behavior in windows. But it surely will be slower (the OS will dump data to file) and the unlink is not trivial.
That does seem like it would be bad if true. Are there any file manipulation options that might be used to control that? I don't know, but I'll ask around and do some reading and see what I can find. [Later] As long as you don't flush the file it would seem like a memory mapped file should be no different from a shared memory in performance, the only difference is where data gets written if it needs to be swapped out. The big downside to using an actual file would be that you need a place in the file system to put it. Then you get into the whole name of the shared memory portability problem again.
Windows natively does not support posix nor System V. I think that with Unix Services for Windows (previously Interix)you have system V interface. I think Unix Services for Unix is a free download now, but I think you need to use that environment to build a Unix-like application (using gcc compiler). I don't know if those system V functions are available with a normal windows environment.
Yeah, I did some more research, and that matches what I found. That path looks like a good way to ensure that Windows programmers don't use the library. Oh well... But wait. We don't need to use SysV shared memory to get reference counting behavior on Windows. So if reference counting behavior is indeed what the library will provide, Windows native API's provide it, and (most or perhaps all) Unix ports could use the SysV API, though as noted in my earlier message, that may have some issues too, regarding kernel limits. For example, I figured out where to find the information for OSX and (by default) you only get 4M maximum size, 32 total, 8 per process. [Note that my own preference is toward not reference counting, but the potential impact on Windows is certainly an issue.]
A different way to address this problem is primarily through documentation. One change to the library interface is also needed, the addition of an explicit deletion/unlink operation. And eliminate the reference counting mechanism from the POSIX port. Here is some proposed text, offered as an option for discussion. Portability Issue for Shared Memory Lifetime The lifetime of a shared memory object varies depending on the platform. On Windows platforms, a shared memory object exists from the time it is created until the last reference to it is removed. For POSIX platforms, a shared memory object exists from the time it is created until it is deleted and the last reference to it is removed; once deleted, no new references can be created. This is a portability issue which unfortunately shows through the library interface. Various methods for eliminating the difference were explored, but none were found to be both satisfactory and robust. As a result of this difference in the lifetime behavior, systems which are intended to be portable across different platforms need to take special care in the design of the lifetime management of shared memory objects. Some recommendations include - For each shared memory object, identify a single process as its owner, and assign it responsibility for creating the shared memory object, deleting any existing object if present, and ensuring that the new object remains accessible for as long as necessary. This may require putting the process into a sleep loop, to ensure that if the system is running on a Windows platform that the reference count will remain positive. Forced deletion by the creator process ensures that a zombie left over from a previous execution of the system will not be reused. - Avoid the use of the create_or_open interfaces provided by this library. This is really a corollary of the previous item. The problem is that on a POSIX system there is no way to use such an operation correctly, because there is no way to distinguish between a properly existing shared memory object that should be opened, and a zombie from a previous execution of the system that should be deleted and re-created. Those interfaces provide access to operating system facilities, so it was deemed desirable to have them present in the library, but their use in portable code is problematic. - Don't assume that explicit deletion of a shared memory object will actually make it inaccessible to future operations. For example, on Windows the explicit deletion operation does nothing, and a shared memory object is only removed when there are no longer any references to it.
A different way to address this problem is primarily through documentation. One change to the library interface is also needed, the addition of an explicit deletion/unlink operation. And eliminate the reference counting mechanism from the POSIX port. Here is some proposed text, offered as an option for discussion.
Although ugly, a function to erase all named objects is an option. You can catch UNIX signals like SIGSEGV and call a function to release all reference counts, after that, abort the program. I would need to register every named object in a global list and the destroy_all_named_before_abort() would destroy every object. After that you can exit. Ion
"Ion Gaztañaga"
Although ugly, a function to erase all named objects is an option. You can catch UNIX signals like SIGSEGV and call a function to release all reference counts, after that, abort the program. I would need to register every named object in a global list and the destroy_all_named_before_abort() would destroy every object. After that you can exit.
Still won't work for SIGKILL and for development phase when you often stop the program via debugger. Up front cleanup, when main application starts (to use shmem), is the most safe option and quite intuitive to use. /Pavel
At 8:46 AM +0100 3/11/06, =?UTF-8?B?SW9uIEdhenRhw7FhZ2E=?= wrote:
Although ugly, a function to erase all named objects is an option. You can catch UNIX signals like SIGSEGV and call a function to release all reference counts, after that, abort the program. I would need to register every named object in a global list and the destroy_all_named_before_abort() would destroy every object. After that you can exit.
Too late! Once SIGSEGV arrives, all bets are off. One might sometimes get lucky and be able to run that cleanup code, but that's highly unreliable. There are situations where it might make sense to *try* to run code after a SIGSEGV (boost.test can do this, with the rationale that some attempt at writing out information is better than just giving up immediately), but that's not good enough for a robust reference count. And as has already been pointed out, this approach doesn't help at all for SIGKILL.
participants (5)
-
Bardur Arantsson -
Ion Gaztañaga -
Kim Barrett -
Olaf van der Spek -
Pavel Vozenilek