[interprocess][iostreams] trac#4234
As part of my work on GIL.IO2 (https://svn.boost.org/svn/boost/sandbox/gil/boost/gil/extension/io2/detail) I've somewhat improved/generalized the original 'proof of claim' code from the post mentioned in the ticket (https://svn.boost.org/trac/boost/ticket/4234)... However, in the same process I've gone through the Interprocess implementation of memory mapped files and AFAICT the various platform specific distinctions seem unnecessary: - AFAIK the POSIX 'anonymous inheritable' shared memory functionality is also available on Win32 - kernel-life time shared memory emulation with memory mapped files on Windows is also unnecessary. It can be either provided directly by bypassing the Win32 APIs using the native NT API (I volunteer to help with the implementation or can do it from scratch by myself) or a new distinction can be made (that does not have anything platform specific in its interface) such that a user can choose between a handle/process life time (which I suppose would be enough most of the time, because what use is there for a client to map a shared memory region after the server has already terminated) or kernel life time shared memory regions (which would then be emulated on Win32)...AFAICT the only thing the Win32 API lacks for this is getting the size of the mapped region which could perhaps be emulated by passing the size at the begining of the mapped region... -- "What Huxley teaches is that in the age of advanced technology, spiritual devastation is more likely to come from an enemy with a smiling face than from one whose countenance exudes suspicion and hate." Neil Postman
On 10/10/2010 08:31 PM, Domagoj Saric wrote:
- AFAIK the POSIX 'anonymous inheritable' shared memory functionality is also available on Win32 - kernel-life time shared memory emulation with memory mapped files on Windows is also unnecessary. It can be either provided directly by bypassing the Win32 APIs using the native NT API (I volunteer to help with the implementation or can do it from scratch by myself)
Be aware that native API calls are not always documented and are subject to change.
AFAICT the only thing the Win32 API lacks for this is getting the size of the mapped region which could perhaps be emulated by passing the size at the begining of the mapped region...
You might be able to do something with VirtualQueryEx, it fills in a MEMORY_BASIC_INFORMATION structure about a specified address. http://msdn.microsoft.com/en-us/library/aa366907.aspx http://msdn.microsoft.com/en-us/library/aa366775.aspx This gives the base address and size of the page range with the same attributes. You can walk through the regions (I wouldn't assume that it's always going to be reported as one contiguous range) and total up the pages with the MEM_MAPPED type ("Indicates that the memory pages within the region are mapped into the view of a section."). Just to ensure another mapped view isn't placed immediately after it, you might want to make a sort of guard page (with VirtualAllocEx) having some non-MEM_MAPPED type right at the end of the mapped view. - Marsh
"Marsh Ray" <marsh@extendedsubset.com> wrote in message news:4CB35BC2.5040302@extendedsubset.com...
On 10/10/2010 08:31 PM, Domagoj Saric wrote:
- AFAIK the POSIX 'anonymous inheritable' shared memory functionality is also available on Win32 - kernel-life time shared memory emulation with memory mapped files on Windows is also unnecessary. It can be either provided directly by bypassing the Win32 APIs using the native NT API (I volunteer to help with the implementation or can do it from scratch by myself)
Be aware that native API calls are not always documented and are subject to change.
I'm aware of that, however considering that: - Interprocess already uses the native NT API - interdependencies between Microsoft's/Windows' own components (and the sheer number of those) make it reasonable to assume breaking changes to the native NT API can in practice ever occur only between major OS releases - in practice breaking changes to system APIs have to be expected between major OS releases even for Win32 and other 'old&public' API as was demonstrated for example by the changes for interactive services between XP and Vista I'd consider this 'danger' practically irrelevant (more like 'academic/by the book paranoia')...especially considering the convenience and efficiency benefits the native NT API gives over Win32...
AFAICT the only thing the Win32 API lacks for this is getting the size of the mapped region which could perhaps be emulated by passing the size at the begining of the mapped region...
You might be able to do something with VirtualQueryEx, it fills in a MEMORY_BASIC_INFORMATION structure about a specified address. http://msdn.microsoft.com/en-us/library/aa366907.aspx http://msdn.microsoft.com/en-us/library/aa366775.aspx
This gives the base address and size of the page range with the same attributes. You can walk through the regions (I wouldn't assume that it's always going to be reported as one contiguous range) and total up the pages with the MEM_MAPPED type ("Indicates that the memory pages within the region are mapped into the view of a section.").
Just to ensure another mapped view isn't placed immediately after it, you might want to make a sort of guard page (with VirtualAllocEx) having some non-MEM_MAPPED type right at the end of the mapped view.
Right! Thanks for the idea ;-) This could be used for a 'backup' (Win32 based) implementation for Windows (that can be used/selected for Windows targets that do not have an underlying native API, if such exist, maybe WinCE...)... All in all, native or otherwise, the proposed changes (independent of the efficiency issues and changes that started all this) seem rather logical as they would make Boost.Interprocess more crossplatform-like...True they would break backward compatibility but I'd consider that well worth it...after all backward compatibility wrappers can always be provided for a few version cycles... Ion, what's your take on this? -- "What Huxley teaches is that in the age of advanced technology, spiritual devastation is more likely to come from an enemy with a smiling face than from one whose countenance exudes suspicion and hate." Neil Postman
On 10/19/2010 10:28 AM, Domagoj Saric wrote:
"Marsh Ray" <marsh@extendedsubset.com> wrote in message news:4CB35BC2.5040302@extendedsubset.com...
Be aware that native API calls are not always documented and are subject to change.
I'm aware of that, however considering that: - Interprocess already uses the native NT API
The documented parts or undocumented parts?
- interdependencies between Microsoft's/Windows' own components (and the sheer number of those) make it reasonable to assume breaking changes to the native NT API can in practice ever occur only between major OS releases
Or service packs, or whenever. http://msdn.microsoft.com/en-us/library/bb432200.aspx :
The functions and structures in Winternl.h are internal to the operating system and subject to change from one release of Windows to the next, and possibly even between service packs for each release. To maintain the compatibility of your application, you should use the equivalent public functions instead. Further information is available in the header file, Winternl.h, and the documentation for each function.
I.e., "fair notice is given: ignore it at your peril".
- in practice breaking changes to system APIs have to be expected between major OS releases even for Win32 and other 'old&public' API as was demonstrated for example by the changes for interactive services between XP and Vista
That was a rare architectural change to Windows (explicit kernel support for multiple terminal services sessions) that broke the previous single-user assumption. App developers had been told that change was coming since post-NT4 days, IIRC. It doesn't discredit the general claim of compatibility across Win32. To the contrary, look at all the processes on your Windows system. How many are created as acutal native NT processes? Maybe two or three (winlogon, csrss), and last I checked the first thing they did was to load kernel32 like everybody else. Step through MS's own internally developed code and you'll see that even they use the documented Win32 APIs as much as possible.
I'd consider this 'danger' practically irrelevant (more like 'academic/by the book paranoia')
My impression from reading the list is that a lot of Boost developers appreciate the "by the book" observance of documented APIs. It's a design principle, not just a paranoid fear based on a perceived probability of breakage. Interfaces are defined not just for the purposes of assigning "blame", but so that we can discuss program correctness rather than just making empirical statements about observed behavior. Next time you read industry reports like "IT departments are delaying deployment of the update (containing 43 critical security fixes) for a few months until they better assess app compatibility" you might take that as evidence that app developers as a whole are not excessively "paranoid".
...especially considering the convenience and efficiency benefits the native NT API gives over Win32...
OK, at work we ship some commercial code that runs on Windows. We use Boost libraries extensively. Sometimes, we have to do some fairly "heroic" and undocumented things to implement the functionality of our product. DLL injection, API hooking, the type of stuff antivirus programs need to do. Right now I'm working on the Vista implementation of a feature that works three different ways on three different versions of Windows. So we are willing to go there when necessary and, most importantly, when there is no other option. But we do it with our eyes open, knowing the maintenance effort it may bring. Occasionally we run into situations where another 3rd party vendor has some creative ideas of their own for the same system customization points and we end up in undefined territory that way. Earlier Boost functionality had been implemented with just standard C++ features, but there was lots of compiler-specific stuff too, mainly to deal with divergence from the standard. Once Boost began venturing into system-level territory, it started acquiring code that needed system headers and specific operating system features. (Isn't it easy to cross these lines silently?) But now we are talking about dependencies on stuff that the platform vendor explicitly disclaims stability across OS versions and even interim updates. Again, if it's necessary to implement useful functionality, that's fine. Yeah I agree that the Nt* APIs aren't going to change all that often. BUT PLEASE STATE THIS DEPENDENCY BOLDLY IN THE LIBRARY DOCUMENTATION along with which OS versions and service packs (not just compiler versions) the library has been tested to work with. Pretty please? :-) It wouldn't rule out the use of such a library in our code, but we would at least know where to focus the extra testing with specific OS versions and where to start looking when customers report unexpected incompatibilities. - Marsh
"Marsh Ray" <marsh@extendedsubset.com> wrote in message news:4CBDEF97.6000101@extendedsubset.com...
On 10/19/2010 10:28 AM, Domagoj Saric wrote:
I'm aware of that, however considering that: - Interprocess already uses the native NT API
The documented parts or undocumented parts?
Search for "ntdll.dll" in details/win32_api.hpp ;)
- interdependencies between Microsoft's/Windows' own components (and the sheer number of those) make it reasonable to assume breaking changes to the native NT API can in practice ever occur only between major OS releases
Or service packs, or whenever.
http://msdn.microsoft.com/en-us/library/bb432200.aspx :
The functions and structures in Winternl.h are internal to the operating system and subject to change from one release of Windows to the next, and possibly even between service packs for each release. To maintain the compatibility of your application, you should use the equivalent public functions instead. Further information is available in the header file, Winternl.h, and the documentation for each function.
I.e., "fair notice is given: ignore it at your peril".
Right, I know this, that is why I wrote >in practice<... And notice that the wording is between releases and possibly between service packs...without the 'whenever' part...
- in practice breaking changes to system APIs have to be expected between major OS releases even for Win32 and other 'old&public' API as was demonstrated for example by the changes for interactive services between XP and Vista
That was a rare architectural change to Windows (explicit kernel support for multiple terminal services sessions) that broke the previous single-user assumption. App developers had been told that change was coming since post-NT4 days, IIRC. It doesn't discredit the general claim of compatibility across Win32.
However you look at it, it still did happen...and it is not the only such change...it happens even on much more higher levels...for example a single hotfix broke backward compatibility for the cryptographic API in Vista... Also please note my point correctly, it is not to 'discredit' the Win32 API but the (in practice) unjustified over-emphasis on the discrepancy between the stability of Win32 API and the native NT API (and the arguments drawn from that 'unjustified over-emphasis')...
To the contrary, look at all the processes on your Windows system. How many are created as acutal native NT processes? Maybe two or three (winlogon, csrss), and last I checked the first thing they did was to load kernel32 like everybody else. Step through MS's own internally developed code and you'll see that even they use the documented Win32 APIs as much as possible.
Loading the kernel32.dll first is no proof of not using the native NT API...e.g. Boost.Interprocess will also cause kernel32.dll to load first (because of static linking) while at the same time using the native NT API (through dynamic runtime loading of ntdll.dll, which is by then already loaded by kernel32.dll anyway...)... Similarly for 'as much as possible', while I have nothing against the principle per se, it obviously does not imply 'always'/'only'/'solely'...
I'd consider this 'danger' practically irrelevant (more like 'academic/by the book paranoia')
My impression from reading the list is that a lot of Boost developers appreciate the "by the book" observance of documented APIs. It's a design principle, not just a paranoid fear based on a perceived probability of breakage. Interfaces are defined not just for the purposes of assigning "blame", but so that we can discuss program correctness rather than just making empirical statements about observed behavior.
Right, however, the whole point of my unholy POV on 'almost dogmatic by-the-bookness' boils down to the fact that "in theory there is no difference between practice and theory, in practice there is"... IOW in practice the choice is not black and white, there is only the more treaded path and the less treaded path(s)...If there are no objections to the more treaded path the choice is obvious but if even the more treaded path gives you thorns in your feet the possibility of exploring the less treaded path(s) may become a viable cost-benefit analysis outcome (which is where I think we basically agree)... But, this is again 'academic' side tracking :) To return to the subject, IMNHO, from the practical/empirical POV, the issue/goal/ideal of 'correctness' must also include some measure of efficiency... Somehow, when using today's software on today's trilions-of-bytes-of-memory-and-trilions-of-operations-per-second in Joe-Sixpack-hardware that still performs far below one's expectations, this seems self-evident... The fact that a browser plugin is able to 'correctly' bring up a functional dialog means little (to me) if it, for that single dialog, also brings in a 15 MB Qt-like monster of an 'auxiliary' library...and when you multiply that by the number of plugins one can have......
Next time you read industry reports like "IT departments are delaying deployment of the update (containing 43 critical security fixes) for a few months until they better assess app compatibility" you might take that as evidence that app developers as a whole are not excessively "paranoid".
I'm not sure if I understand this point...If it is something along the lines that 'you cannot call A large if there exists B that is larger' such arguments are a priori false... IOW I am aware that when it comes to paranoia (as well as irrational/vanity based policies) few things compare to the corporate world :D but this does not necessarily excuse app developers ;)
Sometimes, we have to do some fairly "heroic" and undocumented things to implement the functionality of our product. DLL injection, API hooking, the type of stuff antivirus programs need to do. Right now I'm working on the Vista implementation of a feature that works three different ways on three different versions of Windows. So we are willing to go there when necessary and, most importantly, when there is no other option. But we do it with our eyes open, knowing the maintenance effort it may bring.
And the practical question is, is there a significant discrepancy between the number of these inter-version specifics/incompatibilities caused by public/Win32 API incompatiblities and those caused by native NT API changes?
Again, if it's necessary to implement useful functionality, that's fine. Yeah I agree that the Nt* APIs aren't going to change all that often.
And this is the core point, where we seem to agree fully ;)
BUT PLEASE STATE THIS DEPENDENCY BOLDLY IN THE LIBRARY DOCUMENTATION
Of course, I would also hold this requirement as self-evident...Considering that the relative number of API functions used would probably not be large I'd go for both implementations (if the Win32 API allows it) that one can choose from with a configuration macro... Finally, after all this hard talk, if we consider that the undocumented part of the native NT API (of which, for the use in question, only a few functions would have to be used, probably NTQuerySection and NTExtendSection) is actually documented by the SysInternals guys that now work for Microsoft I again must conclude that the whole issue much more 'theoretical' than 'practical'... ps. As you have correctly deduced by splitting this into a different thread, the issue/idea of using the native NT API is only a side issue of the original ideas which were to make the Interprocess shared/mapped memory interface more cross-platform like and the implementation of shared/mapped memory for both Interprocess and IOStreams more efficient/less bloated... -- "What Huxley teaches is that in the age of advanced technology, spiritual devastation is more likely to come from an enemy with a smiling face than from one whose countenance exudes suspicion and hate." Neil Postman
participants (3)
-
Domagoj Saric -
Domagoj Saric -
Marsh Ray