filesystem::wpath and Linux
Hello list! Im working on an app wich is to run on windows, linux and OSX. I just switched from using filesystem::path to wpath, to be able to use i18n'ed paths on Windows. However this gave me a bunch of problems on Linux. A simple test case: test.cpp #include <iostream> #include <boost/filesystem/path.hpp> #include <boost/filesystem/convenience.hpp> namespace fs = boost::filesystem; int main() { fs::wpath mypath(L"/tmp/some/Directories"); for(int i = 0; i < 10; i++) { try { std::wcout << "Creating " << mypath << " nr " << i; fs::create_directories(mypath); }catch(std::exception&e) { std::cout << e.what() <<std::endl; } } } And Jamfile: import modules ; BOOST_ROOT = [ modules.peek : BOOST_ROOT ] ; use-project /boost : $(BOOST_ROOT) ; project wpathtest : requirements <library>/boost/filesystem//boost_filesystem/<link>static ; exe test : test.cpp : ; Running this in GDB on Linux with gcc 4.1.1, boost 1.34.1 (.0 too): (gdb) % dev1 ~/testcase$ gdb ./bin/gcc-4.1.1/debug/test GNU gdb Red Hat Linux (6.3.0.0-1.143.el4rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1". (gdb) run Starting program: /home/johan/testcase/bin/gcc-4.1.1/debug/test Creating /tmp/some/Directories nr 0 *** glibc detected *** double free or corruption (out): 0x0891e858 *** Program received signal SIGABRT, Aborted. 0x001cf7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 (gdb) where #0 0x001cf7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x002107a5 in raise () from /lib/tls/libc.so.6 #2 0x00212209 in abort () from /lib/tls/libc.so.6 #3 0x00244a1a in __libc_message () from /lib/tls/libc.so.6 #4 0x0024b2bf in _int_free () from /lib/tls/libc.so.6 #5 0x0024b63a in free () from /lib/tls/libc.so.6 #6 0x00654041 in operator delete () from /usr/lib/libstdc++.so.6 #7 0x0065408d in operator delete[] () from /usr/lib/libstdc++.so.6 #8 0x0804e233 in boost::checked_array_delete<char> (x=0x891e858 "\200 \200\200\200\200��\200\200\200\200�\200\200\200\200�\200\200 \200\200p") at /home/common/boost_1_34_1/boost/checked_delete.hpp:41 #9 0x0804e249 in ~scoped_array (this=0xbfefada0) at /home/common/ boost_1_34_1/boost/scoped_array.hpp:65 #10 0x0804f983 in boost::filesystem::wpath_traits::to_external (ph=@0xbfefaf1c, src=@0xbfefadfc) at /home/common/boost_1_34_1/libs/ filesystem/src/path.cpp:96 #11 0x0804af53 in boost::filesystem::basic_path<std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, boost::filesystem::wpath_traits>::external_file_string (this=0xbfefaf1c) at /home/common/boost_1_34_0/boost/filesystem/ path.hpp:302 #12 0x0804b917 in boost::filesystem::exists<boost::filesystem::basic_path<std::basic_strin g<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, boost::filesystem::wpath_traits> > (ph=@0xbfefaf1c) at /home/common/ boost_1_34_0/boost/filesystem/operations.hpp:279 #13 0x0804ba6d in boost::filesystem::exists (ph=@0xbfefaf1c) at /home/ common/boost_1_34_0/boost/filesystem/operations.hpp:601 #14 0x0804c06d in boost::filesystem::create_directories<boost::filesystem::basic_path<std: :basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, boost::filesystem::wpath_traits> > (ph=@0xbfefaf1c) at /home/common/boost_1_34_0/boost/filesystem/ convenience.hpp:42 #15 0x0804c219 in boost::filesystem::create_directories (ph=@0xbfefaf1c) at /home/common/boost_1_34_0/boost/filesystem/ convenience.hpp:88 #16 0x0804c1b3 in boost::filesystem::create_directories<boost::filesystem::basic_path<std: :basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, boost::filesystem::wpath_traits> > (ph=@0xbfefaf7c) at /home/common/boost_1_34_0/boost/filesystem/ convenience.hpp:51 #17 0x0804c219 in boost::filesystem::create_directories (ph=@0xbfefaf7c) at /home/common/boost_1_34_0/boost/filesystem/ convenience.hpp:88 #18 0x0804c1b3 in boost::filesystem::create_directories<boost::filesystem::basic_path<std: :basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, boost::filesystem::wpath_traits> > (ph=@0xbfefafc0) at /home/common/boost_1_34_0/boost/filesystem/ convenience.hpp:51 #19 0x0804c219 in boost::filesystem::create_directories (ph=@0xbfefafc0) at /home/common/boost_1_34_0/boost/filesystem/ convenience.hpp:88 #20 0x0804a4c0 in main () at test.cpp:14 (gdb) If I instead remove the for/try stuff (only keep fs::wpath and fs::create_directories(mypath) and the std::wcout, I get this: (gdb) run Starting program: /home/johan/testcase/bin/gcc-4.1.1/debug/test Creating /tmp/some/Directories nr terminate called after throwing an instance of 'boost::filesystem::basic_filesystem_error<boost::filesystem::basic_path <std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, boost::filesystem::wpath_traits> >' what(): boost::filesystem::wpath::to_external conversion error Program received signal SIGABRT, Aborted. 0x001cf7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 (gdb) where #0 0x001cf7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x002107a5 in raise () from /lib/tls/libc.so.6 #2 0x00212209 in abort () from /lib/tls/libc.so.6 #3 0x006571bb in __gnu_cxx::__verbose_terminate_handler () from /usr/ lib/libstdc++.so.6 #4 0x00654ed1 in __cxa_call_unexpected () from /usr/lib/libstdc++.so.6 #5 0x00654f06 in std::terminate () from /usr/lib/libstdc++.so.6 #6 0x0065504f in __cxa_throw () from /usr/lib/libstdc++.so.6 #7 0x0804b69c in boost::throw_exception<boost::filesystem::basic_filesystem_error<boost:: filesystem::basic_path<std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, boost::filesystem::wpath_traits> > > (e=@0xbfeed054) at /home/common/ boost_1_34_0/boost/throw_exception.hpp:39 #8 0x0804f669 in boost::filesystem::wpath_traits::to_external (ph=@0xbfeed1cc, src=@0xbfeed0ac) at /home/common/boost_1_34_0/libs/ filesystem/src/path.cpp:94 #9 0x0804ad15 in boost::filesystem::basic_path<std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, boost::filesystem::wpath_traits>::external_file_string (this=0xbfeed1cc) at /home/common/boost_1_34_0/boost/filesystem/ path.hpp:302 #10 0x0804b6d9 in boost::filesystem::exists<boost::filesystem::basic_path<std::basic_strin g<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, boost::filesystem::wpath_traits> > (ph=@0xbfeed1cc) at /home/common/ boost_1_34_0/boost/filesystem/operations.hpp:279 #11 0x0804b82f in boost::filesystem::exists (ph=@0xbfeed1cc) at /home/ common/boost_1_34_0/boost/filesystem/operations.hpp:601 #12 0x0804be2f in boost::filesystem::create_directories<boost::filesystem::basic_path<std: :basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, boost::filesystem::wpath_traits> > (ph=@0xbfeed1cc) at /home/common/boost_1_34_0/boost/filesystem/ convenience.hpp:42 #13 0x0804bfdb in boost::filesystem::create_directories (ph=@0xbfeed1cc) at /home/common/boost_1_34_0/boost/filesystem/ convenience.hpp:88 #14 0x0804bf75 in boost::filesystem::create_directories<boost::filesystem::basic_path<std: :basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, boost::filesystem::wpath_traits> > (ph=@0xbfeed22c) at /home/common/boost_1_34_0/boost/filesystem/ convenience.hpp:51 #15 0x0804bfdb in boost::filesystem::create_directories (ph=@0xbfeed22c) at /home/common/boost_1_34_0/boost/filesystem/ convenience.hpp:88 #16 0x0804bf75 in boost::filesystem::create_directories<boost::filesystem::basic_path<std: :basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, boost::filesystem::wpath_traits> > (ph=@0xbfeed278) at /home/common/boost_1_34_0/boost/filesystem/ convenience.hpp:51 #17 0x0804bfdb in boost::filesystem::create_directories (ph=@0xbfeed278) at /home/common/boost_1_34_0/boost/filesystem/ convenience.hpp:88 #18 0x0804a2f3 in main () at test.cpp:10 The double-delete doesnt look too good, does it? The other problem, this makes wpath pretty useless in crossplatform if it cannot decode itself? Even when its such a simple path.. I've done some googling on wpath but there doesnt seem to be very much usage of it.. Any clues? Thanks! -- Johan Ström Stromnet johan@stromnet.se http://www.stromnet.se/
Johan Ström wrote:
Hello list!
Im working on an app wich is to run on windows, linux and OSX. I just switched from using filesystem::path to wpath, to be able to use i18n'ed paths on Windows. However this gave me a bunch of problems on Linux.
A simple test case:
test.cpp ...
This also failed for me. I'll debug it in the next day or so. Thanks for the report, --Beman
Johan Ström wrote:
Hello list!
Im working on an app wich is to run on windows, linux and OSX. I just switched from using filesystem::path to wpath, to be able to use i18n'ed paths on Windows. However this gave me a bunch of problems on Linux.
I was missing the obvious; you forgot to imbue a locale. Your test program works fine once that is done. See below. Because I didn't know your preferred encoding, I used the UTF-8 codecvt facet Boost.Filesystem uses for testing. Maybe it would be better if Boost.Filesystem defaulted to some likely encoding, such as UTF-8. --Beman
A simple test case:
test.cpp #include <iostream> #include <boost/filesystem/path.hpp> #include <boost/filesystem/convenience.hpp>
#include "libs/filesystem/src/utf8_codecvt_facet.hpp"
namespace fs = boost::filesystem; int main() {
std::locale global_loc = std::locale(); fs::detail::utf8_codecvt_facet utf8_facet; std::locale loc( global_loc, &utf8_facet ); fs::wpath_traits::imbue( loc );
fs::wpath mypath(L"/tmp/some/Directories"); for(int i = 0; i < 10; i++) { try { std::wcout << "Creating " << mypath << " nr " << i; fs::create_directories(mypath); }catch(std::exception&e) { std::cout << e.what() <<std::endl; } } }
Beman Dawes wrote:
std::locale global_loc = std::locale(); fs::detail::utf8_codecvt_facet utf8_facet; std::locale loc( global_loc, &utf8_facet );
Eep, don't do that! Facets are reference-counted. The locale adds a reference when you add the facet, and takes it away when it is destructed. When the count drops to zero, it tries to delete the facet. Either allocate the facet with new and pass it to the locale, or pass 1 to its constructor so that the reference counter is initialized at 1 and never drops to zero. Sebastian Redl
On Nov 10, 2007, at 14:32 , Sebastian Redl wrote:
Beman Dawes wrote:
std::locale global_loc = std::locale(); fs::detail::utf8_codecvt_facet utf8_facet; std::locale loc( global_loc, &utf8_facet );
Eep, don't do that!
Facets are reference-counted. The locale adds a reference when you add the facet, and takes it away when it is destructed. When the count drops to zero, it tries to delete the facet.
Either allocate the facet with new and pass it to the locale, or pass 1 to its constructor so that the reference counter is initialized at 1 and never drops to zero.
Thanks guys! Now it is working fine on Linux (with LANG=en_US.utf8 at least). I put my utf8_facet inside a shared_ptr that lives throught my app so thats solved. However, on OS X it still wont work, not sure why.. When I do fs::wpath_traits::imbue( loc ); I get an "locale::facet::_S_create_c_locale name not valid" exception. I'm running with LANG=en_US.UTF-8 here, which (according to locale in console) is fine. Any clues? Thanks Johan
Johan Ström wrote:
Thanks guys! Now it is working fine on Linux (with LANG=en_US.utf8 at least). I put my utf8_facet inside a shared_ptr that lives throught my app so thats solved. However, on OS X it still wont work, not sure why.. When I do fs::wpath_traits::imbue( loc ); I get an "locale::facet::_S_create_c_locale name not valid" exception. I'm running with LANG=en_US.UTF-8 here, which (according to locale in console) is fine.
Was looking through the boost archives and decided to reply this old message. I had similar (somewhat) problems with boost::filesystem::wpath (Linux x86, gcc 3.4.6 compiler) and locale interaction. The UTF-8 facet from the boost itself worked fine for me but somehow I couldn't get it working with the standard system environment locale. I would just failed to convert strings and valgrind said that there are some uninitialized reads or something. I investigated it but no luck. I stopped my investigations when we decided to limit our product support to UTF-8 locales only. Not sure this helps - just for a record. You can search the boost archives for messages from me with "filesystem" in the subject - there is pretty detailed description of the problem there. -- Alexei Alexandrov
participants (4)
-
Alexei Alexandrov -
Beman Dawes -
Johan Ström -
Sebastian Redl