[threadpool] sub_tasks and active waiting

Hi, How can I define pool so the following code do not deadlocks? ???? pool; int fib_seq(int n) { if (n <= 1) return n; else return fib_seq(n−1) + fib_seq(n−2); } void fib_par(int n) { if (n <= 5) // granularity ctl return fib_seq(n); else { tp::task<int> t1(pool.submit(bind(fib_par, n-1)); tp::task<int> t2(pool.submit(bind(fib_par, n-2)); return t1.get_future().get() + t2.get_future().get(); } } int main() { return fib_par(10); } Well, we can see that the number worker threads depends on the fib_par parameter. As soon as the fib_par function waits on its sub_tasks t1 and t2, the worker thread cannot handle other tasks, so we need a thread available for fib_par(9) and fib_par(8). When this worker thread call fib_par(9) after launching fib_par(8) and fib_par(7) will be blocked to manage other task, ... Evidently this do not scale very well, so we can not use the threadpool library to address this kind of problems, in which the tasks is split on sub-task launched to be run in parallel, and the the parennt task waits on its child sub-task to do somthing. IMO, what we need is that instead of creating a task we create a subtask. When the function will wait until the results of its subtasks are ready, we don't need to block the worker thread, instead we can implement replace the pasive wait by an active one, which will check if the waited task is ready and if this is not the case it will do a scheduling step on the associated thread pool and check again. void fib_par(int n) { if (n <= 5) // granularity ctl return fib_seq(n); else { tp::sub_task<int> t1(pool.submit_subtask(bind(fib_par, n-1)); tp::sub_task<int> t2(pool.submit_subtask(bind(fib_par, n-2)); return t1.get() + t2.get(); } } The submison of a sub_task can be done differently of a task, because there is a parent task that will be waiting for the end of the sub_task, so maybe we can add it on LIFO order. Do you think that this is an interesting feature for your library? could it be integrated in your library without disruption? Best, ______________________ Vicente Juan Botet Escribá

Am Freitag, 12. September 2008 08:17:55 schrieb vicente.botet:
int fib_seq(int n) { if (n <= 1) return n; else return fib_seq(n−1) + fib_seq(n−2); } void fib_par(int n) { if (n <= 5) // granularity ctl return fib_seq(n); else { tp::task<int> t1(pool.submit(bind(fib_par, n-1)); tp::task<int> t2(pool.submit(bind(fib_par, n-2)); return t1.get_future().get() + t2.get_future().get(); } }
The recursive fibonacci algorithm is horrible ineficient - better use this one: inline int fibonacci( int n) { if ( n == 0) return 0; if ( n == 1) return 1; int k1( 1), k2( 0); for ( int i( 2); i <= n; ++i) { int tmp( k1); k1 = k1 + k2; k2 = tmp; } return k1; }
Well, we can see that the number worker threads depends on the fib_par parameter. As soon as the fib_par function waits on its sub_tasks t1 and t2, the worker thread cannot handle other tasks, so we need a thread available for fib_par(9) and fib_par(8). When this worker thread call fib_par(9) after launching fib_par(8) and fib_par(7) will be blocked to manage other task, ... Evidently this do not scale very well, so we can not use the threadpool library to address this kind of problems, in which the tasks is split on sub-task launched to be run in parallel, and the the parennt task waits on its child sub-task to do somthing.
The problem is caused by this code: return t1.get_future().get() + t2.get_future().get(); Because it blocks until t1 and t2 are ready -> dead lock possible. Instead you can do something like: tp::task< int > t1( pool.submit( boost::bind( & fib_par, n - 1) ) ); tp::task< int > t2( pool.submit( boost::bind( & fib_par, n - 2) ) ); boost::future< void > f( boost::op( t1.get_future() ) && boost::op( t2.get_future() ) ); pool.chained_submit( boost::bind( & add, t1.get_future(), t2.get_future() ), f); Oliver

----- Original Message ----- From: <k-oli@gmx.de> To: <boost@lists.boost.org> Sent: Saturday, September 13, 2008 9:05 PM Subject: Re: [boost] [threadpool] sub_tasks and active waiting
Am Freitag, 12. September 2008 08:17:55 schrieb vicente.botet:
int fib_seq(int n) { if (n <= 1) return n; else return fib_seq(n−1) + fib_seq(n−2); } void fib_par(int n) { if (n <= 5) // granularity ctl return fib_seq(n); else { tp::task<int> t1(pool.submit(bind(fib_par, n-1)); tp::task<int> t2(pool.submit(bind(fib_par, n-2)); return t1.get_future().get() + t2.get_future().get(); } }
The recursive fibonacci algorithm is horrible ineficient - better use this one:
this was only one example. Take the sort example if you want.
inline int fibonacci( int n) { if ( n == 0) return 0; if ( n == 1) return 1; int k1( 1), k2( 0); for ( int i( 2); i <= n; ++i) { int tmp( k1); k1 = k1 + k2; k2 = tmp; } return k1; }
Well, we can see that the number worker threads depends on the fib_par parameter. As soon as the fib_par function waits on its sub_tasks t1 and t2, the worker thread cannot handle other tasks, so we need a thread available for fib_par(9) and fib_par(8). When this worker thread call fib_par(9) after launching fib_par(8) and fib_par(7) will be blocked to manage other task, ... Evidently this do not scale very well, so we can not use the threadpool library to address this kind of problems, in which the tasks is split on sub-task launched to be run in parallel, and the the parennt task waits on its child sub-task to do somthing.
The problem is caused by this code: return t1.get_future().get() + t2.get_future().get(); Because it blocks until t1 and t2 are ready -> dead lock possible.
Instead you can do something like:
tp::task< int > t1( pool.submit( boost::bind( & fib_par, n - 1) ) ); tp::task< int > t2( pool.submit( boost::bind( & fib_par, n - 2) ) ); boost::future< void > f( boost::op( t1.get_future() ) && boost::op( t2.get_future() ) ); pool.chained_submit( boost::bind( & add, t1.get_future(), t2.get_future() ), f);
What boost::op stands for? Which will be the returned value? What about the sub-tasks feature? Vicente

k-oli@gmx.de writes:
Am Freitag, 12. September 2008 08:17:55 schrieb vicente.botet:
Well, we can see that the number worker threads depends on the fib_par parameter. As soon as the fib_par function waits on its sub_tasks t1 and t2, the worker thread cannot handle other tasks, so we need a thread available for fib_par(9) and fib_par(8). When this worker thread call fib_par(9) after launching fib_par(8) and fib_par(7) will be blocked to manage other task, ... Evidently this do not scale very well, so we can not use the threadpool library to address this kind of problems, in which the tasks is split on sub-task launched to be run in parallel, and the the parennt task waits on its child sub-task to do somthing.
The problem is caused by this code: return t1.get_future().get() + t2.get_future().get(); Because it blocks until t1 and t2 are ready -> dead lock possible.
Not if the thread pool can detect this scenario and handle it. For example, my prototype can handle void recursive_task(jss::thread_pool* pool,int level,int count) { { std::lock_guard<std::mutex> lk(iom); std::cout<<"level "<<level<<", count"<<count<<std::endl; } if(level) { jss::unique_future<void> f1(pool->submit_task_returning<void>(boost::bind(boost::type<void>(),&recursive_task,pool,level-1,1))); jss::unique_future<void> f2(pool->submit_task_returning<void>(boost::bind(boost::type<void>(),&recursive_task,pool,level-1,2))); jss::unique_future<void> f3(pool->submit_task_returning<void>(boost::bind(boost::type<void>(),&recursive_task,pool,level-1,3))); f1.get(); f2.get(); f3.get(); } } just fine, even with only one worker thread. Of course, if you nest too deeply it runs out of stack space :-( Anthony -- Anthony Williams | Just Software Solutions Ltd Custom Software Development | http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL
participants (3)
-
Anthony Williams
-
k-oli@gmx.de
-
vicente.botet