interprocess with unordered_map
Folks, I've built a unordered_map in a memory mapped file. The map goes from: typedef boost::interprocess::allocator< char, boost::interprocess::managed_mapped_file::segment_manager
char_allocator_t;
typedef boost::interprocess::basic_string< char, std::char_traits<char>, char_allocator_t
typedef std::pair< key_type, pay_type> val_type; typedef boost::interprocess::allocator< val_type, boost::interprocess::managed_mapped_file::segment_manager> allocator_t; typedef boost::unordered_map< key_type, pay_type, boost::hash<key_type>, std::equal_to<key_type>, allocator_t> map_type; Everything works fine on creation. But when I later go to read from the map in a different program where the mapped file is opened read only, I have to do gymcrackery like: // A temporary string that allocates from the mapped file struct shm_clean { // cleanup shared memory on stack-unwind shm_clean() { boost::interprocess::shared_memory_object::remove("StupidSharedMemory"); } ~shm_clean() { boost::interprocess::shared_memory_object::remove("StupidSharedMemory"); } } cleaner; boost::interprocess::managed_shared_memory stupid(boost::interprocess::create_only ,"StupidSharedMemory" ,500); key_type key_val(stupid.get_segment_manager()); to create a temporary to allow me to search since the maps find() method will only take a key_type and interprocess::basic_string WILL NOT implicitly construct from a std::string, nor will it implicitly construct from a char* to itself. Can the interprocess::basic_string be modified to allow these conversions? I hate having to do: key_val = "fruitcake"; db.find(key_val); Suggestions? Thanks! Joel
El 15/03/2012 3:14, Joel Young escribió:
Can the interprocess::basic_string be modified to allow these conversions?
I hate having to do:
key_val = "fruitcake"; db.find(key_val);
interprocess::basic_string is now boost::container::basic_string, which follows std::basic_string interface. And you always need an allocator to place the string in shared memory. I don' think it could be implicitly constructible from char* or std::string. Ion
2012/3/15 Ion Gaztañaga <igaztanaga@gmail.com>:
El 15/03/2012 3:14, Joel Young escribió:
Can the interprocess::basic_string be modified to allow these conversions?
I hate having to do:
key_val = "fruitcake"; db.find(key_val);
interprocess::basic_string is now boost::container::basic_string, which follows std::basic_string interface. And you always need an allocator to place the string in shared memory. I don' think it could be implicitly constructible from char* or std::string.
But there's no (good) reason for the input to be required to be in shared memory. -- Olaf
El 15/03/2012 23:18, Olaf van der Spek escribió:
2012/3/15 Ion Gaztañaga<igaztanaga@gmail.com>:
El 15/03/2012 3:14, Joel Young escribió:
Can the interprocess::basic_string be modified to allow these conversions?
I hate having to do:
key_val = "fruitcake"; db.find(key_val);
interprocess::basic_string is now boost::container::basic_string, which follows std::basic_string interface. And you always need an allocator to place the string in shared memory. I don' think it could be implicitly constructible from char* or std::string.
But there's no (good) reason for the input to be required to be in shared memory.
A shared memory allocator is designed to allocate from a managed shared memory segment. And this requires a base address of that shared memory segment . And even with std::string you need to build a temporary string to compare it with the ones stored in the container. Do you have any proposal or alternative idea? Ion
2012/3/15 Ion Gaztañaga <igaztanaga@gmail.com>:
El 15/03/2012 23:18, Olaf van der Spek escribió:
2012/3/15 Ion Gaztañaga<igaztanaga@gmail.com>:
El 15/03/2012 3:14, Joel Young escribió:
Can the interprocess::basic_string be modified to allow these conversions?
I hate having to do:
key_val = "fruitcake"; db.find(key_val);
interprocess::basic_string is now boost::container::basic_string, which follows std::basic_string interface. And you always need an allocator to place the string in shared memory. I don' think it could be implicitly constructible from char* or std::string.
But there's no (good) reason for the input to be required to be in shared memory.
A shared memory allocator is designed to allocate from a managed shared memory segment. And this requires a base address of that shared memory segment . And even with std::string you need to build a temporary string to compare it with the ones stored in the container.
Do you have any proposal or alternative idea?
Yes, do a direct compare between your key type and const char* -- Olaf
Ion Gaztañaga <igaztanaga <at> gmail.com> writes:
El 15/03/2012 23:18, Olaf van der Spek escribió:
2012/3/15 Ion Gaztañaga<igaztanaga <at> gmail.com>:
El 15/03/2012 3:14, Joel Young escribió:
Can the interprocess::basic_string be modified to allow these conversions?
I hate having to do:
key_val = "fruitcake"; db.find(key_val);
place the string in shared memory. I don' think it could be implicitly constructible from char* or std::string.
My code above used the assignment operator to convert it. The constructor can do it also. If not, then at least provide a convenience function to do it!
But there's no (good) reason for the input to be required to be in shared memory.
Exactly. I've got my container in the mapped file or shared memory. Why should I want to clutter up the shared memory with a temporary? Especially if I'm just doing a find in a read-only chunk of memory!
A shared memory allocator is designed to allocate from a managed shared memory segment. And this requires a base address of that shared memory segment . And even with std::string you need to build a temporary string to compare it with the ones stored in the container.
But the temporary string isn't used for anything! I create a temporary string, allocated into a DIFFERENT memory pool and it works just fine. If I can create an arbitrary basic_string with an allocator to a random place to serve as a temporary, the library can darn well do it too! Joel
El 17/03/2012 5:53, Joel escribió:
Ion Gaztañaga<igaztanaga<at> gmail.com> writes:
El 15/03/2012 23:18, Olaf van der Spek escribió:
2012/3/15 Ion Gaztañaga<igaztanaga<at> gmail.com>:
El 15/03/2012 3:14, Joel Young escribió:
Can the interprocess::basic_string be modified to allow these conversions?
I hate having to do:
key_val = "fruitcake"; db.find(key_val);
place the string in shared memory. I don' think it could be implicitly constructible from char* or std::string.
My code above used the assignment operator to convert it. The constructor can do it also. If not, then at least provide a convenience function to do it!
But there's no (good) reason for the input to be required to be in shared memory.
Exactly. I've got my container in the mapped file or shared memory. Why should I want to clutter up the shared memory with a temporary? Especially if I'm just doing a find in a read-only chunk of memory!
A shared memory allocator is designed to allocate from a managed shared memory segment. And this requires a base address of that shared memory segment . And even with std::string you need to build a temporary string to compare it with the ones stored in the container.
But the temporary string isn't used for anything! I create a temporary string, allocated into a DIFFERENT memory pool and it works just fine. If I can create an arbitrary basic_string with an allocator to a random place to serve as a temporary, the library can darn well do it too!
Ok, but this is a different issue. We still need to create a temporary string (and allocate dynamic memory, in shared memory, heap, etc... ) so you don't get any free comparison. Your original statement was: "I have to do gymcrackery like: key_type key_val(stupid.get_segment_manager()); to create a temporary to allow me to search since the maps find() method will only take a key_type and interprocess::basic_string WILL NOT implicitly construct from a std::string, nor will it implicitly construct from a char* to itself. Can the interprocess::basic_string be modified to allow these conversions?" No basic_string change will change the fact that you need to create a temporary (explicit or implicit) to find a value, as unordered requires it. Changing basic_string with implicit conversions to save some keystrokes will do much more harm than good. And you can reuse that explicit temporary if you search several keys in a loop obtaining much better performance. If we want to achieve direct comparison, Boost.Unordered needs to go Boost.Intrusive or Boost.Multiindex route, and offer additional search overloads that can find "equivalent keys": http://www.boost.org/doc/libs/1_49_0/libs/multi_index/doc/reference/ord_indi... http://www.boost.org/doc/libs/1_49_0/doc/html/intrusive/advanced_lookups_ins... I think the annoyance of writing a temporary string is much better than changing boost::container::basic_string (which knows nothing about shared memory as it's an std::string-like class) with conversions that will bring a lot of overload problems. The only point that I consider negative is that we need to allocate memory in a shared memory segment to do a comparison. Maybe we have no write access to shared memory (open_read_only flag) and having to allocate memory will ruin this advantage. A solution for this is to find a way to explicitly build an special temporary of type basic_string< T, boost::interprocess::allocator<> > that will hold the string in heap. I won't make this implicit (say, a default constructed boost::interprocess::allocator) as otherwise tons of new users will start having problems thinking they've placed a string in shm when they are actually building it in the heap). Ion
2012/3/17 Ion Gaztañaga <igaztanaga@gmail.com>:
El 17/03/2012 5:53, Joel escribió: If we want to achieve direct comparison, Boost.Unordered needs to go Boost.Intrusive or Boost.Multiindex route, and offer additional search overloads that can find "equivalent keys":
http://www.boost.org/doc/libs/1_49_0/libs/multi_index/doc/reference/ord_indi...
http://www.boost.org/doc/libs/1_49_0/doc/html/intrusive/advanced_lookups_ins...
That route looks like a good one. Avoiding the need for expensive temps is great. -- Olaf
2012/3/17 Ion Gaztañaga <igaztanaga <at> gmail.com>:
If we want to achieve direct comparison, Boost.Unordered needs to go Boost.Intrusive or Boost.Multiindex route, and offer additional search overloads that can find "equivalent keys":
Olaf van der Spek <ml <at> vdspek.org> writes:
That route looks like a good one. Avoiding the need for expensive temps is great.
Perhaps map and flat_map need this as well? Should bugs be filed? Thanks! Joel
On Sun, Mar 18, 2012 at 9:07 PM, Joel <jdy@cryregarder.com> wrote:
2012/3/17 Ion Gaztañaga <igaztanaga <at> gmail.com>:
If we want to achieve direct comparison, Boost.Unordered needs to go Boost.Intrusive or Boost.Multiindex route, and offer additional search overloads that can find "equivalent keys":
Olaf van der Spek <ml <at> vdspek.org> writes:
That route looks like a good one. Avoiding the need for expensive temps is great.
Perhaps map and flat_map need this as well? Should bugs be filed?
All find() might be affected, but it may be a conscious design decision. You could file bugs and find out. -- Olaf
El 18/03/2012 21:07, Joel escribió:
2012/3/17 Ion Gaztañaga<igaztanaga<at> gmail.com>:
If we want to achieve direct comparison, Boost.Unordered needs to go Boost.Intrusive or Boost.Multiindex route, and offer additional search overloads that can find "equivalent keys":
Olaf van der Spek<ml<at> vdspek.org> writes:
That route looks like a good one. Avoiding the need for expensive temps is great.
Perhaps map and flat_map need this as well? Should bugs be filed?
Yes, please. At least in Boost.Container, as those containers are based on Boost.Intrusive, implementation should be pretty easy. Best, Ion
El 18/03/2012 21:07, Joel escribió:
Perhaps map and flat_map need this as well? Should bugs be filed?
Ion Gaztañaga <igaztanaga <at> gmail.com> writes:
Yes, please. At least in Boost.Container, as those containers are based on Boost.Intrusive, implementation should be pretty easy.
I went ahead and filed. Containers: https://svn.boost.org/trac/boost/ticket/6710 Interprocess: https://svn.boost.org/trac/boost/ticket/6711 For the interprocess one, I messed up and didn't specify the right component. I'm not seeing how to changed it as I don't have an account. Joel
El 20/03/2012 1:31, Joel escribió:
El 18/03/2012 21:07, Joel escribió:
Perhaps map and flat_map need this as well? Should bugs be filed?
Ion Gaztañaga<igaztanaga<at> gmail.com> writes:
Yes, please. At least in Boost.Container, as those containers are based on Boost.Intrusive, implementation should be pretty easy.
I went ahead and filed.
Containers: https://svn.boost.org/trac/boost/ticket/6710
Interprocess: https://svn.boost.org/trac/boost/ticket/6711
For the interprocess one, I messed up and didn't specify the right component. I'm not seeing how to changed it as I don't have an account.
Ok, thanks. Ion
Joel <jdy <at> cryregarder.com> writes:
Ion Gaztañaga <igaztanaga <at> gmail.com> writes:
Yes, please. At least in Boost.Container, as those containers are based on Boost.Intrusive, implementation should be pretty easy.
I went ahead and filed.
Containers: https://svn.boost.org/trac/boost/ticket/6710
Interprocess: https://svn.boost.org/trac/boost/ticket/6711
Ticket 6711 was closed "won't fix" https://svn.boost.org/trac/boost/ticket/6711 :
Closing as "won't fix", since boot unordered is an implementation of the standard unordered containers, and this would be a break from the standard. It would also be worse when using std::equal_to or boost::hash since it would have to cast for every call to those function objects rather than just once for the call to operator[].
Perhaps the interprocess docs can be updated to show how to create a simple allocator from the heap for easy creation of these temporary copiers and how to cleanly provide the thunking between "normal" equivalent types (e.g. strings) and those used in memory regions (basic_string). Joel
El 21/03/2012 22:56, Joel escribió:
Perhaps the interprocess docs can be updated to show how to create a simple allocator from the heap for easy creation of these temporary copiers and how to cleanly provide the thunking between "normal" equivalent types (e.g. strings) and those used in memory regions (basic_string).
Interprocess allocators would require changes to support that and I need to think about it, but I definitely think it should ease this task, as it is demanded by users. Thanks for your reports and comments, Ion
2012/3/23 Ion Gaztañaga <igaztanaga@gmail.com>:
El 21/03/2012 22:56, Joel escribió:
Perhaps the interprocess docs can be updated to show how to create a simple allocator from the heap for easy creation of these temporary copiers and how to cleanly provide the thunking between "normal" equivalent types (e.g. strings) and those used in memory regions (basic_string).
Interprocess allocators would require changes to support that and I need to think about it, but I definitely think it should ease this task, as it is demanded by users.
There's still no technical reason to require this copy, is there? Can't we come up with a solution that avoids this copy? -- Olaf
El 23/03/2012 12:57, Olaf van der Spek escribió:
2012/3/23 Ion Gaztañaga<igaztanaga@gmail.com>:
El 21/03/2012 22:56, Joel escribió:
Perhaps the interprocess docs can be updated to show how to create a simple allocator from the heap for easy creation of these temporary copiers and how to cleanly provide the thunking between "normal" equivalent types (e.g. strings) and those used in memory regions (basic_string).
Interprocess allocators would require changes to support that and I need to think about it, but I definitely think it should ease this task, as it is demanded by users.
There's still no technical reason to require this copy, is there? Can't we come up with a solution that avoids this copy?
Without changing the container the copy (either implicit or explicit) is unavoidable. In those cases, maybe we can avoid a shared memory allocation (which will be always slower than a heap allocation) for that copy. Ion
2012/3/23 Ion Gaztañaga <igaztanaga@gmail.com>:
El 23/03/2012 12:57, Olaf van der Spek escribió:
2012/3/23 Ion Gaztañaga<igaztanaga@gmail.com>:
El 21/03/2012 22:56, Joel escribió:
Perhaps the interprocess docs can be updated to show how to create a simple allocator from the heap for easy creation of these temporary copiers and how to cleanly provide the thunking between "normal" equivalent types (e.g. strings) and those used in memory regions (basic_string).
Interprocess allocators would require changes to support that and I need to think about it, but I definitely think it should ease this task, as it is demanded by users.
There's still no technical reason to require this copy, is there? Can't we come up with a solution that avoids this copy?
Without changing the container the copy (either implicit or explicit) is unavoidable. In those cases, maybe we can avoid a shared memory allocation (which will be always slower than a heap allocation) for that copy.
What's stopping us from changing the container? -- Olaf
participants (4)
-
Ion Gaztañaga -
Joel -
Joel Young
-
Olaf van der Spek