I recommend that Shmem be accepted into Boost. Most of my comments below have already been raised by other posters, so I'll just mention which I agree with. I would use Shmem today if it were available in its final form in Boost. - What is your evaluation of the design? I was impressed with the breadth and depth of the design. It thoroughly covers this domain. My only suggestion for the design would be to separate the concept of mmapped_file from that of the mapped region. The mmapped_file class should just be concerned with managing the "backing store", whether that is an ordinary file, /dev/zero, or the Windows swap file. The mmapped_region is responsible for mapping a portion (or all) of the mmapped_file into the process's memory space, and you could have multiple mmapped_regions per mmapped_file. The current implementation could be covered with a convenience class/factory that creates a mmapped_region that maps an entire mmapped_file. - What is your evaluation of the implementation? Overall, quite impressed. I am in strong agreement with the "RAII crowd" in that I do not want to see two phase initialization. To summarize which of the sub-arguments I agree with, or find to be important: 1. I would like to see constructors that throw when they cannot construct the object. 2. I see no compelling need for fstream() equivalent "empty" constructors 3. I do agree that "reopen()" is a useful concept, and that it could also throw. 4. For embedded environments that cannot use exceptions, I suggest using boost::config to disable the throwing() behavior, and provide an "isValid/isOpen" accessor function which would allow embedded developers to interrogate the state of the object. When exceptions are turned on, this function would always return true. Re: Boost::Threads and synchronization primitives I agree that Shmem provides (yet another) compelling case for factoring Boost::Threads into synchronization primitives and higher level thread management objects (threads, pools, thread specific storage, etc.). The synchronization primitives are too useful as separate objects. - What is your evaluation of the documentation? I actually thought it was adequate, if a little rough around the edges. I'm familiar with the problem domain, though. - What is your evaluation of the potential usefulness of the library? Having a comprehensive, cross-platform library for shared memory is hugely useful. - Did you try to use the library? With what compiler? Did you have any problems? I evaluated this library using VC 7.1 and VC 8.0 without problems. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I put a moderate amount of effort into this evaluation. I read the documentation and looked in depth at the file mapping and allocator objects. - Are you knowledgeable about the problem domain? Yes, I've used shared memory objects and synchronization extensively in financial and research application development. Regards, Dave Moore
Hi Dave,
My only suggestion for the design would be to separate the concept of mmapped_file from that of the mapped region. The mmapped_file class should just be concerned with managing the "backing store", whether that is an ordinary file, /dev/zero, or the Windows swap file. The mmapped_region is responsible for mapping a portion (or all) of the mmapped_file into the process's memory space, and you could have multiple mmapped_regions per mmapped_file.
Yes. The already suggested mmaped_file + mmapped_region is a good idea to optimize descriptor usage and to implement complex cache mechanisms. I will try to implement it for the final version.
I am in strong agreement with the "RAII crowd" in that I do not want to see two phase initialization. To summarize which of the sub-arguments I agree with, or find to be important:
4. For embedded environments that cannot use exceptions, I suggest using boost::config to disable the throwing() behavior, and provide an "isValid/isOpen" accessor function which would allow embedded developers to interrogate the state of the object. When exceptions are turned on, this function would always return true.
The only problem I see to this behavior in my experience is that many times you develop a class for an embedded environment without exceptions (because of limitations) and you want to also test it in PC platforms or reuse use that code in another project and environment that do use exceptions. Since the code has not exception support (you don't program with exceptions AND error returns) you want the code to continue working in the new environment when the exceptions are thrown. That's why I would prefer overloading the constructors with an additional std::nothrow parameter for no-exception environments to disable exception and use is_valid() approach instead of defining a macro. This code would still work in new environments when the code is used somewhere else where exceptions are enabled. After the long debate about RAII I don't know if this solution would be accepted by boost members or only compile-time configuration is accepted.
Re: Boost::Threads and synchronization primitives I agree that Shmem provides (yet another) compelling case for factoring Boost::Threads into synchronization primitives and higher level thread management objects (threads, pools, thread specific storage, etc.). The synchronization primitives are too useful as separate objects.
I see a need to define the official Boost time management functions for Thread/Processes. boost::xtime was supposed to be a temporary solution. Is Boost.DateTime the way to go? I think that it would be useful to decide it before Shmem is released. Thanks for the review, Ion
I see a need to define the official Boost time management functions for Thread/Processes. boost::xtime was supposed to be a temporary solution. Is Boost.DateTime the way to go? I think that it would be useful to decide it before Shmem is released.
I would vote for using Boost.DateTime. It would be unfortunate to add a dependency between Shmem and DateTime, but I believe that Boost.Threads is supposed to be a temporary solution, and increasing the dependence on Threads may be worse than creating the dependence on DateTime. Having said that, I suspect that it wouldn't matter all that much to a user until Boost.Threads is changed. -Fred
It seems likely that Shmem can use the same technique as asio, where it uses the headers only for DateTime, which at least eliminates the DLL/so/lib/a dependency. Dave On 2/15/06, Fred Bertsch <fred.bertsch@gmail.com> wrote:
I see a need to define the official Boost time management functions for Thread/Processes. boost::xtime was supposed to be a temporary solution. Is Boost.DateTime the way to go? I think that it would be useful to decide it before Shmem is released.
I would vote for using Boost.DateTime. It would be unfortunate to add a dependency between Shmem and DateTime, but I believe that Boost.Threads is supposed to be a temporary solution, and increasing the dependence on Threads may be worse than creating the dependence on DateTime.
Having said that, I suspect that it wouldn't matter all that much to a user until Boost.Threads is changed.
-Fred
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
4. For embedded environments that cannot use exceptions, I suggest using boost::config to disable the throwing() behavior, and provide an "isValid/isOpen" accessor function which would allow embedded developers to interrogate the state of the object. When exceptions are turned on, this function would always return true.
The only problem I see to this behavior in my experience is that many times you develop a class for an embedded environment without exceptions (because of limitations) and you want to also test it in PC platforms or reuse use that code in another project and environment that do use exceptions. Since the code has not exception support (you don't program with exceptions AND error returns) you want the code to continue working in the new environment when the exceptions are thrown.
That's why I would prefer overloading the constructors with an additional std::nothrow parameter for no-exception environments to disable exception and use is_valid() approach instead of defining a macro. This code would still work in new environments when the code is used somewhere else where exceptions are enabled.
After the long debate about RAII I don't know if this solution would be accepted by boost members or only compile-time configuration is accepted.
This issue seems to be the only controversial point so far in this review. :) I'd like to elicit some feedback on this so that I can close up the review as quickly as possible after it ends tomorrow. Ion is proposing a solution that solves many of the problems with a two stage construction process. It is still possible to use the constructors with std::no_throw and get an invalid state. I shouldn't comment on the no-exceptions, embedded world that Ion is hoping to placate with his strategy, but in the world where exceptions are possible, it should be easy to tell that you're tripping over something nasty when you pass std::no_throw into a constructor. It seems to me that this should be good enough even if it's not ideal. Because the topic generated some controversy, I'm hoping to get a few people who complained about the two stage construction to comment on the proposed solution before the review ends. -Fred
On 2/15/06, Fred Bertsch <fred.bertsch@gmail.com> wrote:
4. For embedded environments that cannot use exceptions, I suggest using boost::config to disable the throwing() behavior, and provide an "isValid/isOpen" accessor function which would allow embedded developers to interrogate the state of the object. When exceptions are turned on, this function would always return true.
The only problem I see to this behavior in my experience is that many times you develop a class for an embedded environment without exceptions (because of limitations) and you want to also test it in PC platforms or reuse use that code in another project and environment that do use exceptions. Since the code has not exception support (you don't program with exceptions AND error returns) you want the code to continue working in the new environment when the exceptions are thrown.
That's why I would prefer overloading the constructors with an additional std::nothrow parameter for no-exception environments to disable exception and use is_valid() approach instead of defining a macro. This code would still work in new environments when the code is used somewhere else where exceptions are enabled.
After the long debate about RAII I don't know if this solution would be accepted by boost members or only compile-time configuration is accepted.
This would satisfy my concerns, as it eliminates the 2-phase construction possibility, and we would simply avoid the nothrow() constructors to guard against creating 'zombies' Thanks, Dave Moore
Fred Bertsch <fred.bertsch@gmail.com> writes:
Ion is proposing a solution that solves many of the problems with a two stage construction process. It is still possible to use the constructors with std::no_throw and get an invalid state. I shouldn't comment on the no-exceptions, embedded world that Ion is hoping to placate with his strategy, but in the world where exceptions are possible, it should be easy to tell that you're tripping over something nasty when you pass std::no_throw into a constructor.
But it's not possible to tell is that someone has passed you something nasty when somebody else uses no_throw and you're just getting a reference to the object. That's my whole point. Strong invariants make it possible to develop code separately without adding lots of nasty preconditions everywhere to "ensure" that nobody has given you something nasty.
It seems to me that this should be good enough even if it's not ideal.
It's not really much better than having a default ctor that doesn't initialize the object. The problem doesn't occur at the point of construction; it occurs everywhere else in the program. If you make it easy for the guy doing construction to use a 2-phase approach, you make reasoning about code much more difficult for everyone else, and you either complicate their preconditions or you force them into checking for initialization and throwing exceptions. It's a mess.
Because the topic generated some controversy, I'm hoping to get a few people who complained about the two stage construction to comment on the proposed solution before the review ends.
Well, I hope that helps. -- Dave Abrahams Boost Consulting www.boost-consulting.com
Hi David,
But it's not possible to tell is that someone has passed you something nasty when somebody else uses no_throw and you're just getting a reference to the object. That's my whole point. Strong invariants make it possible to develop code separately without adding lots of nasty preconditions everywhere to "ensure" that nobody has given you something nasty.
Well, if the programmer never checks the error of the construction function when he is explicitly requesting a no-throw, the programmer is the problem. So if the programmer requests a new[](std::nothrow) and does not check if the returned pointer is not null, and you receive a null pointer, you have the same problem.
It seems to me that this should be good enough even if it's not ideal.
It's not really much better than having a default ctor that doesn't initialize the object. The problem doesn't occur at the point of construction; it occurs everywhere else in the program. If you make it easy for the guy doing construction to use a 2-phase approach, you make reasoning about code much more difficult for everyone else, and you either complicate their preconditions or you force them into checking for initialization and throwing exceptions. It's a mess.
But there is no 2-phase construction: You can only initialize the object in the constructor and there is no open/create function. It's one-phase. But if the programmer does not check the error when he passes the std::nothrow extra argument (so we suppose the programmer is requesting a manual error check), then it's obvious that that object will be ill-formed. There is always a way to crash a program. No doubt. My only concern is that code reuse from embedded systems that don't use exceptions (mainly because of the space overhead and the lack of predictability when the exception is thrown) won't be possible. And this happens many times. Maybe in the Shmem case it won't be so usual (although there are multi-process real-time Boost supported operating systems like QNX). But in multi-threaded embedded applications where almost always exceptions and RTTI are disabled to gain space you surely need a threading library with an alternative with no exceptions where code reuse should be possible. And C++ is becoming more and more important in those systems, replacing C programming. Just my 2 cents. But I accept the boosters verdict: pure RAII only. Regards, Ion
Ion Gaztañaga <igaztanaga@gmail.com> writes:
Hi David,
But it's not possible to tell is that someone has passed you something nasty when somebody else uses no_throw and you're just getting a reference to the object. That's my whole point. Strong invariants make it possible to develop code separately without adding lots of nasty preconditions everywhere to "ensure" that nobody has given you something nasty.
Well, if the programmer never checks the error of the construction function when he is explicitly requesting a no-throw, the programmer is the problem. So if the programmer requests a new[](std::nothrow) and does not check if the returned pointer is not null, and you receive a null pointer, you have the same problem.
Which is why we assume pointers may be NULL. Not so with references. Everything you've just said misses the point by such a wide margin that I don't know where to begin, and I don't have time to finish right now, so I'll just say this: When a precondition is violated, the programmer that violates it "is the problem." I advocate a system that requires fewer preconditions, and thus offers fewer opportunities for anyone to "be the problem." -- Dave Abrahams Boost Consulting www.boost-consulting.com
Which is why we assume pointers may be NULL. Not so with references. Everything you've just said misses the point by such a wide margin that I don't know where to begin, and I don't have time to finish right now, so I'll just say this:
When a precondition is violated, the programmer that violates it "is the problem." I advocate a system that requires fewer preconditions, and thus offers fewer opportunities for anyone to "be the problem." It looks to me that you are talking about different things. I think Ion agrees about invariants but advocating an interface that can be used without exceptions, to make the library accessible to the
On 2/16/06, David Abrahams <dave@boost-consulting.com> wrote: platforms with no exception available. And without exceptions, you can't ensure that no object with invalid state is constructed. So there is two choices - or to provide an exception-only interface and strong invariants, or to allow the library to be used in a restricting embedded environment. Please correct me if I'm wrong. -- Best regards, Zigmar
It looks to me that you are talking about different things. I think Ion agrees about invariants but advocating an interface that can be used without exceptions, to make the library accessible to the platforms with no exception available. And without exceptions, you can't ensure that no object with invalid state is constructed. So there is two choices - or to provide an exception-only interface and strong invariants, or to allow the library to be used in a restricting embedded environment.
Someone please correct me if I'm wrong, but doesn't the GCC C++ std library (and maybe others) support a "no exception" compile time flag, by replacing thrown exceptions with an assert? Within the library, every place an exception would be thrown goes through a function that has a compile time conditional on either throwing the appropriate std lib exception or calling assert. I think the goal of providing alternate interfaces for embedded environments where exceptions are not supported is misguided. Note that support for replacing the thrown exception with assert is NOT supporting alternate interfaces to the library - it is supporting alternate error recovery mechanisms, which is a different thing. I wonder how environments without exception support handle the C++ std library, since exceptions are part of the standard (and I haven't looked at the EC++ effort, which I think was promoted by Dinkumware and maybe others). I'm a very big believer in preconditions, invariants, etc. I can provide real-world experience (multiple large-scale projects) about the problems caused by two-phase construction (or more specifically, the easy misuse and hidden costly bugs introduced as part of a two-phase construction design). If we were still in the early 90's, where C++ exceptions were not the norm (or not well implemented), it would be a different story (e.g. the ACE library is full of two-phase construction designs due to it's vintage, but if the ACE people were starting from scratch today, it would be a far different story). I guess I just don't see where Shmem is fundamentally different from other Boost libraries, where it must support alternate interfaces for exception-less environments. I haven't recently looked at Shmem, so I'm not criticizing it's (most likely excellent) quality. I'm just chiming in with Dave A and others about class design principles, of which I hold strong opinions. Cliff
"Fred Bertsch" wrote
4. For embedded environments that cannot use exceptions, I suggest using boost::config to disable the throwing() behavior, and provide an "isValid/isOpen" accessor function which would allow embedded developers to interrogate the state of the object. When exceptions are turned on, this function would always return true.
The only problem I see to this behavior in my experience is that many times you develop a class for an embedded environment without exceptions (because of limitations) and you want to also test it in PC platforms or reuse use that code in another project and environment that do use exceptions.
It might be useful to specify some use cases where you are running in an embedded environment with separate processes that need shmem and you have no exceptions, and if no-exceptions is policy or forced by compiler. Its just that that seems to be quite a narrow set. Some use cases would help. Since the code has not exception support (you don't program
with exceptions AND error returns) you want the code to continue working in the new environment when the exceptions are thrown.
That's why I would prefer overloading the constructors with an additional std::nothrow parameter for no-exception environments to disable exception and use is_valid() approach instead of defining a macro.
Sorry for being dim. Whats a std::nothrow parameter?
This code would still work in new environments when the code is used somewhere else where exceptions are enabled.
After the long debate about RAII I don't know if this solution would be accepted by boost members or only compile-time configuration is accepted.
This issue seems to be the only controversial point so far in this review. :) I'd like to elicit some feedback on this so that I can close up the review as quickly as possible after it ends tomorrow.
Ion is proposing a solution that solves many of the problems with a two stage construction process. It is still possible to use the constructors with std::no_throw and get an invalid state. I shouldn't comment on the no-exceptions, embedded world that Ion is hoping to placate with his strategy, but in the world where exceptions are possible, it should be easy to tell that you're tripping over something nasty when you pass std::no_throw into a constructor. It seems to me that this should be good enough even if it's not ideal.
Because the topic generated some controversy, I'm hoping to get a few people who complained about the two stage construction to comment on the proposed solution before the review ends.
First I agree with Dave Abrahams most recent comments. IMO The only way that the choice would be acceptable is by having firstly RAII behaviour and then optionally having no-throw behaviour but no exception throwing behaviour (which seems logical as you would only use this if you dont have exceptions anyway) enabled by a macro. This does seem to cater for the emulator case. You just emulate the nothrow behaviour by defining said macro for testing. regards Andy Little
Andy Little(e)k dio:
"Fred Bertsch" wrote
4. For embedded environments that cannot use exceptions, I suggest using boost::config to disable the throwing() behavior, and provide an "isValid/isOpen" accessor function which would allow embedded developers to interrogate the state of the object. When exceptions are turned on, this function would always return true. The only problem I see to this behavior in my experience is that many times you develop a class for an embedded environment without exceptions (because of limitations) and you want to also test it in PC platforms or reuse use that code in another project and environment that do use exceptions.
It might be useful to specify some use cases where you are running in an embedded environment with separate processes that need shmem and you have no exceptions, and if no-exceptions is policy or forced by compiler. Its just that that seems to be quite a narrow set. Some use cases would help.
I can only talk about my experience in railway embedded systems, where we use QNX multi-process operating system. Exceptions are disabled to save space and increase error-reporting speed to prevent as quickly as possible important failures in control systems. The systems has multiple processes, (one alarm detector, one log server, one communication proxy, one monitorization process...) that share a common shared memory data-base. The most important thing is that multi-processing allows memory-protection, so that if the monitorization process crashes, it can't make crash other processes with some memory overflow issue and it can be restarted and it will start working again. It could be possible to make all processes threads of just one big process but this way, you can't protect important processes from crashes of other less critical processes. Separate processes also allow easier development, in my opinion.
Since the code has not exception support (you don't program
with exceptions AND error returns) you want the code to continue working in the new environment when the exceptions are thrown.
That's why I would prefer overloading the constructors with an additional std::nothrow parameter for no-exception environments to disable exception and use is_valid() approach instead of defining a macro.
Sorry for being dim. Whats a std::nothrow parameter?
See http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vcstdlib/ht...
IMO The only way that the choice would be acceptable is by having firstly RAII behaviour and then optionally having no-throw behaviour but no exception throwing behaviour (which seems logical as you would only use this if you dont have exceptions anyway) enabled by a macro. This does seem to cater for the emulator case. You just emulate the nothrow behaviour by defining said macro for testing.
The problem is not emulation but reusing that code in another environment that uses exceptions, since the reused code won't work anymore. Anyway, I will repeat it here again: I accept RAII only for Shmem. Regards, Ion
"Ion Gaztañaga" wrote
Andy Little(e)k dio:
[...]
It might be useful to specify some use cases where you are running in an embedded environment with separate processes that need shmem and you have no exceptions, and if no-exceptions is policy or forced by compiler. Its just that that seems to be quite a narrow set. Some use cases would help.
I can only talk about my experience in railway embedded systems, where we use QNX multi-process operating system. Exceptions are disabled to save space and increase error-reporting speed to prevent as quickly as possible important failures in control systems. The systems has multiple processes, (one alarm detector, one log server, one communication proxy, one monitorization process...) that share a common shared memory data-base. The most important thing is that multi-processing allows memory-protection, so that if the monitorization process crashes, it can't make crash other processes with some memory overflow issue and it can be restarted and it will start working again. It could be possible to make all processes threads of just one big process but this way, you can't protect important processes from crashes of other less critical processes. Separate processes also allow easier development, in my opinion.
Ok ...thanks for the example. Though it raise a lot of questions which perhaps cant be anwered. As I understand it the no-exceptions requirement in this example is a policy based on the fact that exceptions for error recovery have been found to be too slow to respond? I'm surprised at the extra memory use as there surely must be some extra error checking in the non-exception case. OTOH Is exception use increasing the size of stack required?. Hmm... being no expert , I guess I should avoid prying too far into your applications ;-) It also helps to understand why shmem is set up currently as it is. [...]
Sorry for being dim. Whats a std::nothrow parameter?
See http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vcstdlib/ht...
Thanks for the link!
The problem is not emulation but reusing that code in another environment that uses exceptions, since the reused code won't work anymore.
If anyone wished to do that I guess you would need to revisit the reasoning to them in this review.
Anyway, I will repeat it here again: I accept RAII only for Shmem.
I am now wondering though if the change over to RAII is going to have other unwelcome side-effects on a design that has not been designed around it? This is pure speculation. I havent looked deeply into the matter. Also are you reasonably happy with that decision ( RAII only for Shmem)? I guess it is not too helpful for the application above for example. I wonder if there are other situations where the decision has been taken not to use exceptions, because if there were enough of them it might be worth re-opening the argument for the defense ;-) regards Andy Little
Hi Andy,
Ok ...thanks for the example. Though it raise a lot of questions which perhaps cant be anwered. As I understand it the no-exceptions requirement in this example is a policy based on the fact that exceptions for error recovery have been found to be too slow to respond?
As Dave mentioned exception handling can be slow when there is an error. As a general recommendation, many embedded system enterprises don't use exceptions (yes, including EC++ group). Technical Report on C++ (N1668) Performance confirms that a "code" exception implementation has run-time overhead in normal code but a "table" exception has no runtime overhead in normal code. However there is a space overhead. N1668 reports 15% overhead in size from one vendor. But the problem is predictability. There also international norms that regulate what languages can be used depending on security/reliability levels in some areas. Those norms state that only a subset of C++ can be used to program some security/reliability equipment. That subset does not include exceptions, so many times you are forced to avoid exceptions. Many enterprises take the decision to program without exceptions for all embedded devices because they don't know if in the future a developed code can end in one of those devices. If you need more information about exceptions N1668 has some good explanations.
I am now wondering though if the change over to RAII is going to have other unwelcome side-effects on a design that has not been designed around it? This is pure speculation. I havent looked deeply into the matter.
I don't think so.
Also are you reasonably happy with that decision (RAII only for Shmem)? I guess it is not too helpful for the application above for example.
Well, not *very* happy, but the point is that I should implement what boosters want. If in the future there are requests for a non-throwing alternative interface, that can be discussed and developed in the future. But this will be more needed in threads, since there are hard-realtime embedded systems without exceptions but using threads. You always have the option to use the C API. This RAII issue is closed to me. Now I have a lot for work to do for the final Shmem version. Regards, Ion
Hi Ion, "Ion Gaztañaga" wrote [...]
As Dave mentioned exception handling can be slow when there is an error. As a general recommendation, many embedded system enterprises don't use exceptions (yes, including EC++ group). Technical Report on C++ (N1668)
BTW I think N1668 is the wrong reference. Probably should be n1666 "Technical Report on C++ Performance" 28 April 2003. ( Section 3 in n1666 is a very good example of a Terms and Definitions section, comprehensive and early in the documentation FWIW). Congratulations on getting shmem accepted to boost! regards Andy Little
Fred Bertsch wrote: ...
After the long debate about RAII I don't know if this solution would be accepted by boost members or only compile-time configuration is accepted.
...
Ion is proposing a solution that solves many of the problems with a two stage construction process. It is still possible to use the constructors with std::no_throw and get an invalid state. I shouldn't
My vote would be to only have the RAII type. My second would be that they be separate types in separate headers so that users of the properly designed RAII type not be concerned with the sins of the two-phase type. :) And of course the two-phase type should come with the appropriate warning labels, and be more difficult to find.
comment on the no-exceptions, embedded world that Ion is hoping to placate with his strategy, but in the world where exceptions are possible, it should be easy to tell that you're tripping over something nasty when you pass std::no_throw into a constructor. It seems to me that this should be good enough even if it's not ideal.
I concur with Dave Abrahams' reasoning here. I work in an evironment of primarily C programmers who venture into C++ now and then. I'd prefer to have them come to me and ask how to add an RAII shmem data member and initialize it. Rather than wade through their code trying to fix some totally unrelated problem, just to see they have fallen prey to two-phase initialization issues. Jeff Flinn
participants (8)
-
Andy Little -
Cliff Green -
Dave Moore -
David Abrahams -
Fred Bertsch -
Ion Gaztañaga -
Jeff Flinn -
Pavel Antokolsky aka Zigmar