[boost-TLS] Thread Local Storage (TLS) Question
Hello,
I am using multiple threads for I/O operations. The issue I am having is
joining these threads on the main thread. It appears that calling join() on
a thread with TLS pointers, it results in a hang.
I have multiple thread specific pointers declared outside of the function
and in the beginning of the method I call reset ( new std::string ) ....etc
etc. I also am acquiring some resource handlers as Im using a
boost::thread_specific_ptr < ifstream > infile
So, when I create the thread in the main thread and join() it I get a hang.
The only thing I can think of is that the handlers are not being released
so join() waits indefinitely.
I read in the documentation that the pointers should be cleaned up on
thread exit - but the thread apparently never exits because then resources
are not released? I understand that it may not be "conventional" to use a
tls pointer for ifstream but I find it a lot better than locking with
mutexes.
Also, I'm not sure if there are limits as to how many TLS pointers you can
actually use but I am currently using about 10 for one thread method.
Another thing to note is that when calling detach() on the thread, all of
the threads execute properly and I get expected output. However, I need to
call join() every x amount of threads due to OS limitations of how many
threads can actually be created. I need to read about 5000 files so
obviously I can't create 5000 threads as Windows maxes out at 700 I
believe. I am reading files off of a RAID setup so I believe this is a
situation that can benefit from multiple threads reading files.
Basically, I only want to create 4 threads at a time, then join() the
previous 4, ensure that they have all completed successfully and continue
to create more.
Any suggestions or comments on this issue?
Thanks
--
Kyle Ketterer
I am using multiple threads for I/O operations. The issue I am having is joining these threads on the main thread. It appears that calling join() on a thread with TLS pointers, it results in a hang.
Could you provide a small self-contained repro, or at least show some code? <...>
Another thing to note is that when calling detach() on the thread, all of the threads execute properly and I get expected output. However, I need to call join() every x amount of threads due to OS limitations of how many threads can actually be created. I need to read about 5000 files so obviously I can't create 5000 threads as Windows maxes out at 700 I believe. I am reading files off of a RAID setup so I believe this is a situation that can benefit from multiple threads reading files.
Creating threads for i/o is not an efficient/scalable approach. Consider using Boost.Asio instead (in particular, see the following example http://www.boost.org/doc/libs/1_55_0/doc/html/boost_asio/example/cpp03/windo...)
Here is some scaled down code of generally what I'm doing:
-----------------------------------------------------------------------------------------------
boost::thread_specific_ptr <ifstream> infile;
boost::thread_specific_ptr< std::vectorstd::string > file_buffer;
boost::thread_specific_ptrstd::string TempLine;
//more ptrs here
void ReadFile(wxString FileName)
{
infile.reset(new ifstream);
file_buffer.reset ( new std::vectorstd::string );
TempLine.reset ( new std::string );
//init other ptrs here
//open file, etc here (@infile)
//read contents into vector line by line
while ( ! infile.get()->eof() )
{
getline(*infile,*TempLine);
file_buffer.get()->push_back(*TempLine);
}
//other operations on file_buffer below ...........
}
void CallFunc()
{
boost::filesystem::path p(TEMP_PATH); //refers to class member
boost::filesystem::directory_iterator itr(p);
boost::filesystem::directory_entry & entry = *itr;
boost::filesystem::directory_iterator end_itr;
std::vector< boost::thread* > t;
int total_files = std::count_if( directory_iterator(p),
directory_iterator(), bind( static_cast
I am using multiple threads for I/O operations. The issue I am having is joining these threads on the main thread. It appears that calling join() on a thread with TLS pointers, it results in a hang.
Could you provide a small self-contained repro, or at least show some code?
Another thing to note is that when calling detach() on the thread, all of the threads execute properly and I get expected output. However, I need to call join() every x amount of threads due to OS limitations of how many threads can actually be created. I need to read about 5000 files so obviously I can't create 5000 threads as Windows maxes out at 700 I believe. I am reading files off of a RAID setup so I believe this is a situation
<...> that
can benefit from multiple threads reading files.
Creating threads for i/o is not an efficient/scalable approach. Consider using Boost.Asio instead (in particular, see the following example http://www.boost.org/doc/libs/1_55_0/doc/html/boost_asio/example/cpp03/windo... ) _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
--
Kyle Ketterer
On 24 Nov 2013 at 11:16, Kyle Ketterer wrote:
Another thing to note is that when calling detach() on the thread, all of the threads execute properly and I get expected output. However, I need to call join() every x amount of threads due to OS limitations of how many threads can actually be created. I need to read about 5000 files so obviously I can't create 5000 threads as Windows maxes out at 700 I believe. I am reading files off of a RAID setup so I believe this is a situation that can benefit from multiple threads reading files.
You shouldn't use threads to read from many files in parallel - you should use the very well designed Win IOCP API to do asynchronous file operations. There is a new Boost library called AFIO in the peer review queue (http://www.boost.org/community/review_schedule.html) which extends Boost.ASIO with asynchronous file i/o support. If you don't like AFIO, use libuv which is a C library also implementing portable asynchronous file i/o. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
participants (3)
-
Igor R -
Kyle Ketterer -
Niall Douglas