I would like to store regexes in shared memory, but it seems like Boost regex falls just barely short of supporting this. In the basic_regex documentation, it says "The definition of basic_regex... is based very closely on class basic_string and fulfils the requirements for a constant-container of charT." This may be technically correct, however basic_string and other containers take an allocator template argument, while basic_regex does not. If basic_regex had an allocator it would reap all the same benefits that other string and container types do. In particular, it would be possible to use boost::interprocess to place them in shared memory. Since the allocator would be defaulted to allocator<charT>, I think it would IMO be compatible with the vast majority of existing code (not quite all because template<class charT, class traits> void f(basic_regex<charT, traits>) wouldn't work). Does this make sense? Do others agree that it would make regexes more like other strings and containers as advertised? Thanks, Mike
Hi Mike, Mike Spertus wrote:
basic_string and other containers take an allocator template argument, while basic_regex does not.
If basic_regex had an allocator it would reap all the same benefits that other string and container types do. In particular, it would be possible to use boost::interprocess to place them in shared memory.
IIUC, while the custom allocators let you put std::containers in shared memory they are still not very useful since they can only be used by the process that put them there - other processes will (potentially) see the shared memory at a different address and all the pointers will be wrong. Is this level of functionality useful to you? Phil.
basic_string and other containers take an allocator template argument, while basic_regex does not.
If basic_regex had an allocator it would reap all the same benefits that other string and container types do. In particular, it would be possible to use boost::interprocess to place them in shared memory.
IIUC, while the custom allocators let you put std::containers in shared memory they are still not very useful since they can only be used by the process that put them there - other processes will (potentially) see the shared memory at a different address and all the pointers will be wrong. Is this level of functionality useful to you?
That would certainly be a killer blow as far as using this with regex is concerned... but don't you have the option of at least trying to load shared memory at a specific address? John.
John Maddock" wrote:
basic_string and other containers take an allocator template argument, while basic_regex does not.
If basic_regex had an allocator it would reap all the same benefits that other string and container types do. In particular, it would be possible to use boost::interprocess to place them in shared memory.
IIUC, while the custom allocators let you put std::containers in shared memory they are still not very useful since they can only be used by the process that put them there - other processes will (potentially) see the shared memory at a different address and all the pointers will be wrong. Is this level of functionality useful to you?
That would certainly be a killer blow as far as using this with regex is concerned... but don't you have the option of at least trying to load shared memory at a specific address?
Yes you can ask to map it at a particular address, but that can fail. Actually there is another case that I forgot about before: if you create the shared memory before forking, the new process will see it at the same address as its parent. So this could be useful in that situation. Sorry, it's a while since I did anything with shared memory... Phil.
On Dec 5, 2009, at 10:14 PM, Phil Endecott wrote:
while the custom allocators let you put std::containers in shared memory they are still not very useful since they can only be used by the process that put them there - other processes will (potentially) see the shared memory at a different address and all the pointers will be wrong
Isn't this *exactly* what the allocator's nested ::pointer type is for? -- David Abrahams BoostPro Computing http://boostpro.com
while the custom allocators let you put std::containers in shared memory they are still not very useful since they can only be used by the process that put them there - other processes will (potentially) see the shared memory at a different address and all the pointers will be wrong
Isn't this *exactly* what the allocator's nested ::pointer type is for?
Maybe, but it's use would be pretty limited. The std lib is entitled to assume that: "The typedef members pointer, const_pointer, size_type, and difference_type are required to be T*,T const*, size_t, and ptrdiff_t, respectively" So I wouldn't be able to use any std lib containers within regex if we were to allow the ::pointer type to be useful :-( And boost::shared_ptr, while allocating it's memory via an allocator is completely agnostic about it's type, so I couldn't use any smart pointer classes either :-( So... all I would need to do is reimplement the std lib, and shared_ptr etc... or maybe not! Which is not to say that having an allocator for other purposes isn't worthwhile, Cheers, John.
John Maddock wrote:
Maybe, but it's use would be pretty limited. The std lib is entitled to assume that:
"The typedef members pointer, const_pointer, size_type, and difference_type are required to be T*,T const*, size_t, and ptrdiff_t, respectively"
So why are ::pointer et al provided at all? --Jeffrey Bosboom
Jeffrey Bosboom wrote:
So why are ::pointer et al provided at all?
Perhaps because it would be a part of a reasonable allocator abstraction? So the right interface is there, but it is not fully supported by the std lib (but you would be able to provide an implementation of the std lib that would fully support it).
Thomas Klimpel wrote:
Jeffrey Bosboom wrote:
So why are ::pointer et al provided at all?
Perhaps because it would be a part of a reasonable allocator abstraction? So the right interface is there, but it is not fully supported by the std lib (but you would be able to provide an implementation of the std lib that would fully support it).
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Then I guess a better question would be, "Why don't standard library implementations properly use ::pointer?" (My guess was that doing so inhibits optimizations or something similar, since it can't be that much harder to use pointer than just using T* everywhere.) --Jeffrey Bosboom
Jeffrey Bosboom wrote:
Then I guess a better question would be, "Why don't standard library implementations properly use ::pointer?"
In 2005 I asked the GNU libstdc++ people about this, and Matt Austern replied: "The sticking point, and the reason why nobody has done this yet for libstdc++, is that nobody has spelled out precisely what requirements my_allocator<T>::pointer must satisfy. Is my_allocator::pointer allowed to have a nontrivial copy constructor? Does it have to be a POD? Can you use const_cast to convert my_allocator::const_pointer to my_allocator::pointer? Can you use static_cast to convert my_allocator<void>::pointer to my_allocator<T>::pointer? If there's an implicit conversion from D* to B*, does it necessarily follow that there's an implicit conversion from my_allocator<D>::pointer to my_allocator<B>::pointer? Martin Sobor (of Dinkumware) said: "With user-defined pointers you can't make many of the assumptions and optimizations that you can otherwise. For example, you can't assume that such pointer types are convertible to true pointers (such as void*), you might need to be prepared to deal with exceptions where they normally cannot occur, you can't use memcpy to copy objects of user-defined pointers, etc. The thread starts here: http://gcc.gnu.org/ml/libstdc++/2005-05/msg00294.html I'm pleased to read in Mike's post that this is resolved for C++0x. I don't believe this is yet implemented in GNU libstdc++ (the bug is still open). Regards, Phil.
Phil Endecott <spam_from_boost_dev <at> chezphil.org> writes:
Jeffrey Bosboom wrote:
Then I guess a better question would be, "Why don't standard library implementations properly use ::pointer?"
In 2005 I asked the GNU libstdc++ people about this, and Matt Austern replied:
"The sticking point, and the reason why nobody has done this yet for libstdc++, is that nobody has spelled out precisely what requirements my_allocator<T>::pointer must satisfy. Is my_allocator::pointer allowed to have a nontrivial copy constructor? Does it have to be a POD? Can you use const_cast to convert my_allocator::const_pointer to my_allocator::pointer? Can you use static_cast to convert my_allocator<void>::pointer to my_allocator<T>::pointer? If there's an implicit conversion from D* to B*, does it necessarily follow that there's an implicit conversion from my_allocator<D>::pointer to my_allocator<B>::pointer?
Martin Sobor (of Dinkumware) said:
"With user-defined pointers you can't make many of the assumptions and optimizations that you can otherwise. For example, you can't assume that such pointer types are convertible to true pointers (such as void*), you might need to be prepared to deal with exceptions where they normally cannot occur, you can't use memcpy to copy objects of user-defined pointers, etc.
The thread starts here: http://gcc.gnu.org/ml/libstdc++/2005-05/msg00294.html
I'm pleased to read in Mike's post that this is resolved for C++0x. I don't believe this is yet implemented in GNU libstdc++ (the bug is still open).
Shared memory containers are fully usable now. We just use the container implementations provided for that purpose in Boost::interprocess, which work just great. Our experience is that this is a mature production-quality implementation comparable to other Boost libraries. In particular, the relative pointer type (offset_pointer) provided by the boost::interprocess shared memory allocators obey enough of the requirements one would expect from a pointer replacement that the boost::interproces STL containers can use them. I will certainly be pleased when I can use a std::map this way (actually, I think you can with the current Dinkumware) but until then, there is absolutely no problem with using the boost::interprocess::map reimplementation instead. Mike
Hi John, John Maddock <john <at> johnmaddock.co.uk> writes:
while the custom allocators let you put std::containers in shared memory they are still not very useful since they can only be used by the process that put them there - other processes will (potentially) see the shared memory at a different address and all the pointers will be wrong
Isn't this *exactly* what the allocator's nested ::pointer type is for?
Maybe, but it's use would be pretty limited. The std lib is entitled to assume that:
"The typedef members pointer, const_pointer, size_type, and difference_type are required to be T*,T const*, size_t, and ptrdiff_t, respectively"
So I wouldn't be able to use any std lib containers within regex if we were to allow the ::pointer type to be useful
And boost::shared_ptr, while allocating it's memory via an allocator is completely agnostic about it's type, so I couldn't use any smart pointer classes either
So... all I would need to do is reimplement the std lib, and shared_ptr etc... or maybe not!
Fortunately, (1) boost::interprocess has done it for you already a complete set of STL- compliant containers that are guaranteed to respect the pointer typedef. (2) The Microsoft/Dinkumware standard libraries included with Visual C++ respect the pointer typedef. (3) In fact, other boost containers such as Boost::unordered promise to respect the pointer typedef. See http://www.boost.org/doc/libs/1_41_0/doc/html/interprocess/allocators_contai... s.html#interprocess.allocators_containers.additional_containers (4) Finally, in C++0x, all container will be required to respect the pointer typedef (It removes the words you quote above). Have you ever even used Boost::interprocess? Just to be clear, we use STL containers in position-independent shared memory segments every day. For the reasons given above, there are perfectly safe ways to ensure your standard containers and strings (except for regex) respect the pointer typedef. You can see loads of examples of perfectly safe and unrestricted storage of standard containers in position-independent shared-memory segments at http://www.boost.org/doc/libs/1_41_0/doc/html/interprocess/quick_guide.html#... erprocess.quick_guide.qg_interprocess_container http://www.boost.org/doc/libs/1_41_0/doc/html/interprocess/quick_guide.html#... erprocess.quick_guide.qg_interprocess_map http://www.boost.org/doc/libs/1_41_0/doc/html/interprocess/allocators_contai... s.html#interprocess.allocators_containers.containers_explained.containers_of_co ntainers http://www.boost.org/doc/libs/1_41_0/doc/html/interprocess/allocators_contai... s.html#interprocess.allocators_containers.containers_explained http://www.boost.org/doc/libs/1_41_0/doc/html/interprocess/allocators_contai... s.html#interprocess.allocators_containers.additional_containers.unordered http://www.boost.org/doc/libs/1_41_0/doc/html/interprocess/allocators_contai... s.html#interprocess.allocators_containers.additional_containers.multi_index Really, the points you raise are complete non-issues in practice with great support. By contrast, the lack of an allocator in regex is a big problem with no support either in Boost or the standards.
Which is not to say that having an allocator for other purposes isn't worthwhile,
Cheers, John.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
David Abrahams wrote:
On Dec 5, 2009, at 10:14 PM, Phil Endecott wrote:
while the custom allocators let you put std::containers in shared memory they are still not very useful since they can only be used by the process that put them there - other processes will (potentially) see the shared memory at a different address and all the pointers will be wrong
Isn't this *exactly* what the allocator's nested ::pointer type is for?
It would be, except that it's broken. Maybe someone else can recall the details, but it's something like allocators are required to define pointer as T* or containers are allowed to assume that they have done so. Phil.
Hi Phil, Phil Endecott <spam_from_boost_dev <at> chezphil.org> writes:
Hi Mike,
Mike Spertus wrote:
basic_string and other containers take an allocator template argument, while basic_regex does not.
If basic_regex had an allocator it would reap all the same benefits that other string and container types do. In particular, it would be possible to use boost::interprocess to place them in shared memory.
IIUC, while the custom allocators let you put std::containers in shared memory they are still not very useful since they can only be used by the process that put them there - other processes will (potentially) see the shared memory at a different address and all the pointers will be wrong. Is this level of functionality useful to you?
Custom allocators do let multiple processes use the same containers in shared memory, even if they are mapped to different addresses because allocators define a pointer type for the container to use internally. For example, Boost::Interprocess provides an interprocess allocator that uses relative pointers to handle the exact issue you describe. See http://www.boost.org/doc/libs/1_41_0/doc/html/interprocess/allocators_contai... for details. So shared memory STL containers are actually very useful, and we use them extensively in our project to share fully STL-compatible data structures between many processes. The failure to have an allocator in regexes is a big problem for us because configurations (which is what we like to share) often include regular expressions. Mike
Phil.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Phil Endecott wrote:
Hi Mike,
Mike Spertus wrote:
basic_string and other containers take an allocator template argument, while basic_regex does not.
If basic_regex had an allocator it would reap all the same benefits that other string and container types do. In particular, it would be possible to use boost::interprocess to place them in shared memory.
IIUC, while the custom allocators let you put std::containers in shared memory they are still not very useful since they can only be used by the process that put them there - other processes will (potentially) see the shared memory at a different address and all the pointers will be wrong. Is this level of functionality useful to you?
I suppose there wouldn't be any problem if the library author used Allocator::pointer instead of T*.
participants (7)
-
David Abrahams -
Jeffrey Bosboom -
John Maddock -
Mathias Gaunard -
Mike Spertus -
Phil Endecott -
Thomas Klimpel