Shared memory between Visual Studio build and MinGW build
Hi folks, got a strange one here. I am trying to get two programs to communicate with each other on Windows 10 via shared memory. Both of them are using Boost 1.72, compiled from the same set of headers. I am only using the shared memory portion of boost. However, one is compiled with Visual studio 2019 community, and the other one is compiled with MinGW 64, g++ version 9.2.0. I have to use MinGW, because I am writing a plugin for another program, which only takes dlls compiled with MinGW. The program compiled with MinGW creates the shared memory and allocates several objects. Then I started the second program that was compiled with Visual Studio 2019. It was able to find and connect to the managed segment. When I am viewing the segment info from Visual Studio's debugger, everything looks ok. However, when I call get_free_memory(). It spits back 18446744073709551104, which is just bizarre. Then, when I tried to find_or_construct on the managed segment. It just hangs, apparently stuck in winapi::sleep_tick() with the following tracedll!boost::interprocess::winapi::sleep_tick() Line 1288 C++dll!boost::interprocess::ipcdetail::thread_sleep_tick() Line 187 C++dll!boost::interprocess::spin_wait::yield() Line 128 C++dll!boost::interprocess::ipcdetail::try_based_lock<boost::interprocess::ipcdetail::spin_mutex>(boost::interprocess::ipcdetail::spin_mutex & m) Line 71 C++dll!boost::interprocess::ipcdetail::spin_mutex::lock() Line 81 C++dll!boost::interprocess::ipcdetail::spin_recursive_mutex::lock() Line 95 C++dll!boost::interprocess::interprocess_recursive_mutex::lock() Line 165 C++dll!boost::interprocess::scoped_lock<boost::interprocess::interprocess_recursive_mutex>::scoped_lock<boost::interprocess::interprocess_recursive_mutex>(boost::interprocess::interprocess_recursive_mutex & m) Line 81 C++dll!boost::interprocess::segment_manager<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>,boost::interprocess::iset_index>::priv_generic_named_construct<char>(unsigned char type, const char * name, unsigned __int64 num, bool try2find, bool dothrow, boost::interprocess::ipcdetail::in_place_interface & table, boost::interprocess::iset_index<boost::interprocess::ipcdetail::index_config<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>>> & index, boost::interprocess::ipcdetail::bool_<1> is_intrusive) Line 1076 C++dll!boost::interprocess::segment_manager<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>,boost::interprocess::iset_index>::priv_generic_construct(const char * name, unsigned __int64 num, bool try2find, bool dothrow, boost::interprocess::ipcdetail::in_place_interface & table) Line 760 C++dll!boost::interprocess::segment_manager<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>,boost::interprocess::iset_index>::generic_construct<SharedMem_Info>(const char * name, unsigned __int64 num, bool try2find, bool dothrow, boost::interprocess::ipcdetail::in_place_interface & table) Line 705 C++dll!boost::interprocess::ipcdetail::named_proxy<boost::interprocess::segment_manager<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>,boost::interprocess::iset_index>,SharedMem_Info,0>::operator()<char const (&)[17],int,int,int,int,bool,boost::interprocess::allocator<void,boost::interprocess::segment_manager<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>,boost::interprocess::iset_index>> &>(const char[17] & <args_0>, int && <args_1>, int && <args_2>, int && <args_3>, int && <args_4>, bool && <args_5>, boost::interprocess::allocator<void,boost::interprocess::segment_manager<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>,boost::interprocess::iset_index>> & <args_6>) Line 131 C++ It looks like it the segment manager was unable to get a mutex lock on the memory? But I don't believe I put a mutex lock on the segment manager from the other end. Everything works just fine if I am doing shared memory between two programs compiled with visual studio. So, it feels like somewhere some how memory alignment is off? Anybody ran into a similar issue before? Any help would be greatly appreciated! Thank you!!
It seems that Boost has different binary layouts of its data structures when compiled with MSVC and MINGW, i.e., they’re not compatible between the two compilers. The result is therefore not unexpected. MAYBE it’s possible to make it work (e.g., compile MSVC code in Release mode), but 1) I wouldn’t bet on it, 2) even if you get it to work it’ll be extremely fragile. * Stian From: Boost-users <boost-users-bounces@lists.boost.org> On Behalf Of Da Xu via Boost-users Sent: Sunday, March 1, 2020 00:53 To: Boost-users@lists.boost.org Cc: Da Xu <djsignal7@yahoo.com> Subject: [Boost-users] Shared memory between Visual Studio build and MinGW build Hi folks, got a strange one here. I am trying to get two programs to communicate with each other on Windows 10 via shared memory. Both of them are using Boost 1.72, compiled from the same set of headers. I am only using the shared memory portion of boost. However, one is compiled with Visual studio 2019 community, and the other one is compiled with MinGW 64, g++ version 9.2.0. I have to use MinGW, because I am writing a plugin for another program, which only takes dlls compiled with MinGW. The program compiled with MinGW creates the shared memory and allocates several objects. Then I started the second program that was compiled with Visual Studio 2019. It was able to find and connect to the managed segment. When I am viewing the segment info from Visual Studio's debugger, everything looks ok. However, when I call get_free_memory(). It spits back 18446744073709551104, which is just bizarre. Then, when I tried to find_or_construct on the managed segment. It just hangs, apparently stuck in winapi::sleep_tick() with the following trace dll!boost::interprocess::winapi::sleep_tick() Line 1288 C++ dll!boost::interprocess::ipcdetail::thread_sleep_tick() Line 187 C++ dll!boost::interprocess::spin_wait::yield() Line 128 C++ dll!boost::interprocess::ipcdetail::try_based_lock<boost::interprocess::ipcdetail::spin_mutex>(boost::interprocess::ipcdetail::spin_mutex & m) Line 71 C++ dll!boost::interprocess::ipcdetail::spin_mutex::lock() Line 81 C++ dll!boost::interprocess::ipcdetail::spin_recursive_mutex::lock() Line 95 C++ dll!boost::interprocess::interprocess_recursive_mutex::lock() Line 165 C++ dll!boost::interprocess::scoped_lock<boost::interprocess::interprocess_recursive_mutex>::scoped_lock<boost::interprocess::interprocess_recursive_mutex>(boost::interprocess::interprocess_recursive_mutex & m) Line 81 C++ dll!boost::interprocess::segment_manager<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>,boost::interprocess::iset_index>::priv_generic_named_construct<char>(unsigned char type, const char * name, unsigned __int64 num, bool try2find, bool dothrow, boost::interprocess::ipcdetail::in_place_interface & table, boost::interprocess::iset_index<boost::interprocess::ipcdetail::index_config<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>>> & index, boost::interprocess::ipcdetail::bool_<1> is_intrusive) Line 1076 C++ dll!boost::interprocess::segment_manager<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>,boost::interprocess::iset_index>::priv_generic_construct(const char * name, unsigned __int64 num, bool try2find, bool dothrow, boost::interprocess::ipcdetail::in_place_interface & table) Line 760 C++ dll!boost::interprocess::segment_manager<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>,boost::interprocess::iset_index>::generic_construct<SharedMem_Info>(const char * name, unsigned __int64 num, bool try2find, bool dothrow, boost::interprocess::ipcdetail::in_place_interface & table) Line 705 C++ dll!boost::interprocess::ipcdetail::named_proxy<boost::interprocess::segment_manager<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>,boost::interprocess::iset_index>,SharedMem_Info,0>::operator()<char const (&)[17],int,int,int,int,bool,boost::interprocess::allocator<void,boost::interprocess::segment_manager<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>,boost::interprocess::iset_index>> &>(const char[17] & <args_0>, int && <args_1>, int && <args_2>, int && <args_3>, int && <args_4>, bool && <args_5>, boost::interprocess::allocator<void,boost::interprocess::segment_manager<char,boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family,boost::interprocess::offset_ptr<void,__int64,unsigned __int64,0>,0>,boost::interprocess::iset_index>> & <args_6>) Line 131 C++ It looks like it the segment manager was unable to get a mutex lock on the memory? But I don't believe I put a mutex lock on the segment manager from the other end. Everything works just fine if I am doing shared memory between two programs compiled with visual studio. So, it feels like somewhere some how memory alignment is off? Anybody ran into a similar issue before? Any help would be greatly appreciated! Thank you!!
On Mon, 2 Mar 2020 at 02:11, Stian Zeljko Vrba via Boost-users < boost-users@lists.boost.org> wrote:
It seems that Boost has different binary layouts of its data structures when compiled with MSVC and MINGW, i.e., they’re not compatible between the two compilers.
Erm, gcc and vc will never output the same binary code, I haven't done or seen this, but it must be possible to compile clang (with mingw-gcc) for use with mingw, then with that clang-cl you might stand a chance. Sticking to static libraries, helps already (with exceptions across library boundaries), using C as glue could fix the issue forever. Another, much simpler solution is to ditch mingw and use clang-cl as your main (or support to VC) compiler. Clang can 'theoretically' compile 'anything' (bar bugs in Clang), as long as it's coded in a std that's supported by that version of clang and the code adheres to that std, including but not limited to gcc-extensions and microsoft-extensions (in addition to clang-extensions). Contrary to what Clang claims, Clang is not fast (!!!), but it generates good code and excellent (short) error messages, sometimes better (faster binary) than VC, sometimes worse than VC, it's a crap-shoot, for speed-critical code, compiling bits with VC and bits with Clang after having identified which bits work best which compiler after bench-marking, gives to absolute best result, at least at par with gcc if not better. -- @systemdeg "We value your privacy, click here!" Sod off! - degski "Anyone who believes that exponential growth can go on forever in a finite world is either a madman or an economist" - Kenneth E. Boulding "Growth for the sake of growth is the ideology of the cancer cell" - Edward P. Abbey
Thank you guys. Yah, the problem is that the pieces I am writing are all DLLs. And of course, gcc DLL and Msvc DLLs aren't compatible. Therefore, I HAVE to use two compilers. So, clang might produce data structure layout that might be compatible with gcc? One idea I thought about was to forgo data structures entirely. Allocate multiple pieces of named shared memory to hold basic data types. It would would be a book keeping nightmare though. And I am not entirely sure how to deal with mutex locking etc in that case. I don't think standard c++ mutex would work for shared memory... What do u guys think? Thank you! On Monday, March 2, 2020, 09:01:59 AM EST, degski via Boost-users <boost-users@lists.boost.org> wrote: On Mon, 2 Mar 2020 at 02:11, Stian Zeljko Vrba via Boost-users <boost-users@lists.boost.org> wrote: It seems that Boost has different binary layouts of its data structures when compiled with MSVC and MINGW, i.e., they’re not compatible between the two compilers. Erm, gcc and vc will never output the same binary code, I haven't done or seen this, but it must be possible to compile clang (with mingw-gcc) for use with mingw, then with that clang-cl you might stand a chance. Sticking to static libraries, helps already (with exceptions across library boundaries), using C as glue could fix the issue forever. Another, much simpler solution is to ditch mingw and use clang-cl as your main (or support to VC) compiler. Clang can 'theoretically' compile 'anything' (bar bugs in Clang), as long as it's coded in a std that's supported by that version of clang and the code adheres to that std, including but not limited to gcc-extensions and microsoft-extensions (in addition to clang-extensions). Contrary to what Clang claims, Clang is not fast (!!!), but it generates good code and excellent (short) error messages, sometimes better (faster binary) than VC, sometimes worse than VC, it's a crap-shoot, for speed-critical code, compiling bits with VC and bits with Clang after having identified which bits work best which compiler after bench-marking, gives to absolute best result, at least at par with gcc if not better. -- @systemdeg"We value your privacy, click here!" Sod off! - degski "Anyone who believes that exponential growth can go on forever in a finite world is either a madman or an economist" - Kenneth E. Boulding"Growth for the sake of growth is the ideology of the cancer cell" - Edward P. Abbey _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users
On 3/03/2020 08:37, Da Xu wrote:
Thank you guys. Yah, the problem is that the pieces I am writing are all DLLs. And of course, gcc DLL and Msvc DLLs aren't compatible. Therefore, I HAVE to use two compilers.
So, clang might produce data structure layout that might be compatible with gcc?
If you confine your shared memory interface to POD types that use well-known-sized fields, then most compilers will produce a compatible binary layout. (With a few quirks to watch out for, such as default padding and potentially different sizes for "int", "long", "size_t", "time_t", etc. It doesn't hurt to explicitly static_assert(sizeof(T) == known_constant) in both compilers.) It's when you introduce non-POD C++ types that you typically run into trouble. For example, library types such as std::string are not guaranteed to have equivalent binary layout across different compilers (or even different compiler versions).
Thank you for the reply. Right now, I am actually stuck in Boost's segment manager. Its own data structure got scrambled in the memory. I haven't even gotten to where my own data structure gets read. If I do go with the bare metal approach, I will definitely keep these advice in mind. Da. On Monday, March 2, 2020, 06:33:55 PM EST, Gavin Lambert via Boost-users <boost-users@lists.boost.org> wrote: On 3/03/2020 08:37, Da Xu wrote:
Thank you guys. Yah, the problem is that the pieces I am writing are all DLLs. And of course, gcc DLL and Msvc DLLs aren't compatible. Therefore, I HAVE to use two compilers.
So, clang might produce data structure layout that might be compatible with gcc?
If you confine your shared memory interface to POD types that use well-known-sized fields, then most compilers will produce a compatible binary layout. (With a few quirks to watch out for, such as default padding and potentially different sizes for "int", "long", "size_t", "time_t", etc. It doesn't hurt to explicitly static_assert(sizeof(T) == known_constant) in both compilers.) It's when you introduce non-POD C++ types that you typically run into trouble. For example, library types such as std::string are not guaranteed to have equivalent binary layout across different compilers (or even different compiler versions). _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users
participants (4)
-
Da Xu -
degski -
Gavin Lambert -
Stian Zeljko Vrba