RE: [boost] Thread: Why is my prog aborting after 283 or so threads?
I went ahead and submitted a bug report in the Sourceforge bug tracker for this issue: [ 1075456 ] Boost::thread under Linux crashes after 200 to 400 threads I made it a high priority, since it's a program crash. I attached my program to reproduce the problem, so hopefully the person who fixes this will be able to reproduce the crash pretty easily. -- Tim -----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Tim Laplaca Sent: Monday, November 29, 2004 9:29 AM To: boost@lists.boost.org Subject: RE: [boost] Thread: Why is my prog aborting after 283 or so threads? Thanks for the link. I'm not sure it applies though; the prog isn't spawning the threads simultaneously, so it shouldn't be taking many resources. I've found that even if I only have one thread spawned at one time (where the previous thread has died out before the new one is spawned) it still aborts at the same place. Over the holiday I pared my program down to just the thread spawning, and on my Mandrake setup at home it dies at the nice even (computer) number of 256 threads. The same pared-down prog dies on my work Fedora Core 3 machine at 405. Under Windows, it doesn't ever seem to die. I'm getting pretty convinced this is a bug in the boost library that manifests itself only under Linux, and not my code. Perhaps resources are not being properly released when the threads are dying out. Since the prog is pretty small, I'll include it here. It spawns a thread every second, and each thread only last one second. #include <iostream> #include <string> #include <stdlib.h> // for multithreading #include <boost/thread/mutex.hpp> #include <boost/thread.hpp> #include <boost/thread/xtime.hpp> #include <boost/bind.hpp> int totalthreads = 0; boost::mutex io_mutex; // iostreams boost::mutex numthreads_mutex; // for changing value of numthreads boost::mutex threadid_mutex; // for manipulating thread_id using namespace std; void threadfunc() { // wait some secs boost::xtime xt; boost::xtime_get(&xt, boost::TIME_UTC); xt.sec += 1; boost::thread::sleep(xt); { boost::mutex::scoped_lock scoped_lock(io_mutex); std::cout << " Thread Exiting. " << std::endl; } } main() { boost::thread_group thrds; boost::xtime xt; int maxthreads=99999; // ******************************** // Main thread-spawning loop for (int i=0; i<=maxthreads; i++) { totalthreads++; { boost::mutex::scoped_lock scoped_lock(io_mutex); std::cout << "MAIN: Thread:" << totalthreads << " being spawned... " << std::endl; } thrds.create_thread( &threadfunc); // wait some seconds boost::xtime_get(&xt, boost::TIME_UTC); xt.sec += 1; boost::thread::sleep(xt); } // end thread-spawning for loop thrds.join_all(); std::cout << "Done." << std::endl; } Here's the Makefile I build it with: lib = /usr/local/lib/libboost_thread-gcc-mt-1_32.so threadtest : threadtest.o g++ $(lib) $(lib2) -o threadtest threadtest.o -ldl -lc -lpthread threadtest.o : threadtest.cpp g++ -Wall threadtest.cpp -I/usr/local/include/boost-1_32 -pthread -c -w -O -o threadtest.o -----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Alexander Nasonov Sent: Friday, November 26, 2004 2:07 PM To: boost@lists.boost.org Subject: Re: [boost] Thread: Why is my prog aborting after 283 or so threads? Tim Laplaca wrote:
I wrote a prog that spawns threads (RED HAT Fedora Core 2), and after spawning 283 or so threads, it stops running and 'Aborted' is printed.
Take a look at: http://www.kegel.com/c10k.html#threaded and check your sizes. -- Alexander Nasonov _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Mon, 29 Nov 2004 13:57:00 -0500, Tim Laplaca <tlaplaca@voiceglo.com> wrote:
I went ahead and submitted a bug report in the Sourceforge bug tracker for this issue:
[ 1075456 ] Boost::thread under Linux crashes after 200 to 400 threads
I made it a high priority, since it's a program crash. I attached my program to reproduce the problem, so hopefully the person who fixes this will be able to reproduce the crash pretty easily.
-- Tim
I've compiled the test program and can confirm the behavior on Red Hat Advanced Server 3.0 and Debian GNU/Linux running stock kernel 2.6.7, though I got up over 1100 threads on each of these before the program died. The problem isn't one of threads so much as one of memory however. The program just runs out of memory because you are not join'ing the exited threads. According to the manual page for pthread_join on my Linux box: When a joinable thread terminates, its memory resources (thread descriptor and stack) are not deallocated until another thread performs pthread_join on it. Therefore, pthread_join must be called once for each joinable thread created to avoid memory leaks. Indeed, changing the code to use individual threads instead of a thread group and joining each after the sleep call fixes the leak and the crash. I've successfully spawned tens of thousands of threads on Linux with this change in place. -- Caleb Epstein caleb dot epstein at gmail dot com
At 02:30 PM 11/29/2004, Caleb Epstein wrote:
The problem isn't one of threads so much as one of memory however. The program just runs out of memory because you are not join'ing the exited threads.
Caleb, could you please do one more thing on this: read the docs and see if your answer is obvious from the docs. If not, could you please submit a path clarifying the docs the need to join. Thanks, --Beman
Beman Dawes wrote:
At 02:30 PM 11/29/2004, Caleb Epstein wrote:
The problem isn't one of threads so much as one of memory however. The program just runs out of memory because you are not join'ing the exited threads.
Caleb, could you please do one more thing on this: read the docs and see if your answer is obvious from the docs. If not, could you please submit a path clarifying the docs the need to join.
I guess he's running out of resources since .join() is not being called in the example program. Regards, m
On Mon, 29 Nov 2004 16:41:49 -0500, Beman Dawes <bdawes@acm.org> wrote:
Caleb, could you please do one more thing on this: read the docs and see if your answer is obvious from the docs. If not, could you please submit a path clarifying the docs the need to join.
FYI I've managed to produce this OOM condition with Solaris pthreads, so its not a Linux or Boost issue. Attached is a simple program that probes this limit using only pthreads calls. On Solaris I hit the memory limit after 2,714 threads and on Linux 2.4.21-9.ELsmp I hit it at 1,020. I don't think the memory leak issue is discussed anywhere in the Boost.Threads documentation, or if it is I'm unable to find it. Because threads are automatically detached in the thread dtor, the only way for this leak to take place is by creating threads on the heap and not deleting/joining them, which is what the OP was doing. There is the following text in http://boost.org/doc/html/thread.html, under Description: "A thread of execution is said to be "finished" or to have "finished execution" when its initial function returns or is terminated. This includes completion of all thread cleanup handlers, and completion of the normal C++ function return behaviors, such as destruction of automatic storage (stack) objects and releasing any associated implementation resources." Perhaps the following paragraph could be added in another callout at the bottom of this section or below the "thread construct/copy/destruct" section: "With some platform-specific thread libraries (notably POSIX threads), the stack space of a joinable thread is not automatically cleaned up until it has been joined. When thread objects are allocated on the heap (e.g. via operator new or thread_group::create), it is the responsibility of the user of Boost.Threads to ensure these threads have been either detached or joined in order to avoid memory leaks." Also, here are some formatting oddities I noticed in the documentation from my reading just now: On page http://boost.org/doc/html/threads/rationale.html, in the Comparison section: * 2. Comparison: creation of a thread that's later joined create_thread(&bar);thrd->join(); should be create_thread(&bar); thrd->join(); * 4. Comparison: creation of several threads in a loop which are later joined. for (int i= 0; i<NUM_THREADS; ++i)threads[i]->join(); should be for (int i= 0; i<NUM_THREADS; ++i) threads[i]->join(); -- Caleb Epstein caleb dot epstein at gmail dot com
At 06:07 PM 11/29/2004, Caleb Epstein wrote:
There is the following text in http://boost.org/doc/html/thread.html, under Description:
"A thread of execution is said to be "finished" or to have "finished execution" when its initial function returns or is terminated. This includes completion of all thread cleanup handlers, and completion of the normal C++ function return behaviors, such as destruction of automatic storage (stack) objects and releasing any associated implementation resources."
Perhaps the following paragraph could be added in another callout at the bottom of this section or below the "thread construct/copy/destruct" section:
"With some platform-specific thread libraries (notably POSIX threads), the stack space of a joinable thread is not automatically cleaned up until it has been joined. When thread objects are allocated on the heap (e.g. via operator new or thread_group::create), it is the responsibility of the user of Boost.Threads to ensure these threads have been either detached or joined in order to avoid memory leaks."
Also, here are some formatting oddities I noticed in the documentation from my reading just now:
On page http://boost.org/doc/html/threads/rationale.html, in the Comparison section:
* 2. Comparison: creation of a thread that's later joined
create_thread(&bar);thrd->join();
should be
create_thread(&bar); thrd->join();
* 4. Comparison: creation of several threads in a loop which are later joined.
for (int i= 0; i<NUM_THREADS; ++i)threads[i]->join();
should be
for (int i= 0; i<NUM_THREADS; ++i) threads[i]->join();
I guess it is really Michael Glassford's call, but those changes seem appropriate to me. Michael? --Beman
"Beman Dawes" <bdawes@acm.org> wrote in message news:6.0.3.0.2.20041130115430.02f39928@mailhost.esva.net...
At 06:07 PM 11/29/2004, Caleb Epstein wrote:
There is the following text in http://boost.org/doc/html/thread.html, under Description:
"A thread of execution is said to be "finished" or to have "finished execution" when its initial function returns or is terminated. This includes completion of all thread cleanup handlers, and completion of the normal C++ function return behaviors, such as destruction of automatic storage (stack) objects and releasing any associated implementation resources."
Perhaps the following paragraph could be added in another callout at the bottom of this section or below the "thread construct/copy/destruct" section:
"With some platform-specific thread libraries (notably POSIX threads), the stack space of a joinable thread is not automatically cleaned up until it has been joined. When thread objects are allocated on the heap (e.g. via operator new or thread_group::create), it is the responsibility of the user of Boost.Threads to ensure these threads have been either detached or joined in order to avoid memory leaks."
Also, here are some formatting oddities I noticed in the documentation from my reading just now:
On page http://boost.org/doc/html/threads/rationale.html, in the Comparison section:
* 2. Comparison: creation of a thread that's later joined
create_thread(&bar);thrd->join();
should be
create_thread(&bar); thrd->join();
* 4. Comparison: creation of several threads in a loop which are later joined.
for (int i= 0; i<NUM_THREADS; ++i)threads[i]->join();
should be
for (int i= 0; i<NUM_THREADS; ++i) threads[i]->join();
I guess it is really Michael Glassford's call, but those changes seem appropriate to me.
Michael?
Sorry, I must have missed Caleb's post. His suggestions look good to me; I'll make them. Mike
participants (5)
-
Beman Dawes -
Caleb Epstein -
Martin Wille -
Michael Glassford -
Tim Laplaca