[boost-TLS] Thread Local Storage (TLS) Question
Hello, I am using multiple threads for I/O operations. The issue I am having is joining these threads on the main thread. It appears that calling join() on a thread with TLS pointers, it results in a hang. I have multiple thread specific pointers declared outside of the function and in the beginning of the method I call reset ( new std::string ) ....etc etc. I also am acquiring some resource handlers as Im using a boost::thread_specific_ptr < ifstream > infile So, when I create the thread in the main thread and join() it I get a hang. The only thing I can think of is that the handlers are not being released so join() waits indefinitely. I read in the documentation that the pointers should be cleaned up on thread exit - but the thread apparently never exits because then resources are not released? I understand that it may not be "conventional" to use a tls pointer for ifstream but I find it a lot better than locking with mutexes. Also, I'm not sure if there are limits as to how many TLS pointers you can actually use but I am currently using about 10 for one thread method. Another thing to note is that when calling detach() on the thread, all of the threads execute properly and I get expected output. However, I need to call join() every x amount of threads due to OS limitations of how many threads can actually be created. I need to read about 5000 files so obviously I can't create 5000 threads as Windows maxes out at 700 I believe. I am reading files off of a RAID setup so I believe this is a situation that can benefit from multiple threads reading files. Basically, I only want to create 4 threads at a time, then join() the previous 4, ensure that they have all completed successfully and continue to create more. Any suggestions or comments on this issue? Thanks -- Kyle Ketterer <reminisc3@gmail.com>
I am using multiple threads for I/O operations. The issue I am having is joining these threads on the main thread. It appears that calling join() on a thread with TLS pointers, it results in a hang.
Could you provide a small self-contained repro, or at least show some code? <...>
Another thing to note is that when calling detach() on the thread, all of the threads execute properly and I get expected output. However, I need to call join() every x amount of threads due to OS limitations of how many threads can actually be created. I need to read about 5000 files so obviously I can't create 5000 threads as Windows maxes out at 700 I believe. I am reading files off of a RAID setup so I believe this is a situation that can benefit from multiple threads reading files.
Creating threads for i/o is not an efficient/scalable approach. Consider using Boost.Asio instead (in particular, see the following example http://www.boost.org/doc/libs/1_55_0/doc/html/boost_asio/example/cpp03/windo...)
Here is some scaled down code of generally what I'm doing: ----------------------------------------------------------------------------------------------- boost::thread_specific_ptr <ifstream> infile; boost::thread_specific_ptr< std::vector<std::string> > file_buffer; boost::thread_specific_ptr<std::string> TempLine; //more ptrs here void ReadFile(wxString FileName) { infile.reset(new ifstream); file_buffer.reset ( new std::vector<std::string> ); TempLine.reset ( new std::string ); //init other ptrs here //open file, etc here (@infile) //read contents into vector line by line while ( ! infile.get()->eof() ) { getline(*infile,*TempLine); file_buffer.get()->push_back(*TempLine); } //other operations on file_buffer below ........... } void CallFunc() { boost::filesystem::path p(TEMP_PATH); //refers to class member boost::filesystem::directory_iterator itr(p); boost::filesystem::directory_entry & entry = *itr; boost::filesystem::directory_iterator end_itr; std::vector< boost::thread* > t; int total_files = std::count_if( directory_iterator(p), directory_iterator(), bind( static_cast<bool(*)(const path&)>(is_regular_file), bind( &directory_entry::path, _1 ) ) ); for (; itr!=end_itr; ++itr) { //read files, create threads t.push_back( new boost::thread(&ReadFile, this, entry.path().filename().string() ) ) ; t->join(); //app freezes here, if i detach() instead of join it's fine //some other stuff } //delete thread pointers, etc ( join() before delete ) } ------------------------------------------------------------------------------------------------ Hope that helps On Sun, Nov 24, 2013 at 12:07 PM, Igor R <boost.lists@gmail.com> wrote:
I am using multiple threads for I/O operations. The issue I am having is joining these threads on the main thread. It appears that calling join() on a thread with TLS pointers, it results in a hang.
Could you provide a small self-contained repro, or at least show some code?
Another thing to note is that when calling detach() on the thread, all of the threads execute properly and I get expected output. However, I need to call join() every x amount of threads due to OS limitations of how many threads can actually be created. I need to read about 5000 files so obviously I can't create 5000 threads as Windows maxes out at 700 I believe. I am reading files off of a RAID setup so I believe this is a situation
<...> that
can benefit from multiple threads reading files.
Creating threads for i/o is not an efficient/scalable approach. Consider using Boost.Asio instead (in particular, see the following example http://www.boost.org/doc/libs/1_55_0/doc/html/boost_asio/example/cpp03/windo... ) _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- Kyle Ketterer <reminisc3@gmail.com> 215-208-8523
On 24 Nov 2013 at 11:16, Kyle Ketterer wrote:
Another thing to note is that when calling detach() on the thread, all of the threads execute properly and I get expected output. However, I need to call join() every x amount of threads due to OS limitations of how many threads can actually be created. I need to read about 5000 files so obviously I can't create 5000 threads as Windows maxes out at 700 I believe. I am reading files off of a RAID setup so I believe this is a situation that can benefit from multiple threads reading files.
You shouldn't use threads to read from many files in parallel - you should use the very well designed Win IOCP API to do asynchronous file operations. There is a new Boost library called AFIO in the peer review queue (http://www.boost.org/community/review_schedule.html) which extends Boost.ASIO with asynchronous file i/o support. If you don't like AFIO, use libuv which is a C library also implementing portable asynchronous file i/o. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
participants (3)
-
Igor R -
Kyle Ketterer -
Niall Douglas