[thread] Dividing into Boost.Thread and Boost.Sync

Hi, In our private conversation Vicente J. Botet Escriba, Tim Blechmann and myself have formed a proposal to split the current Boost.Thread library into two libraries: Boost.Thread and Boost.Sync. The former will retain its thread management features, such as thread class, futures and TLS. The latter will contain all thread synchronization components, such as: - locks - mutexes - condition variables - call_once Mutexes and condition variables in Boost.Sync will not support thread interruption and will be as simple and lightweight as possible. For example, one of the benefits I'd like to achieve with this is that synchronization primitives should become decoupled from the timing libraries. I'm not sure we'll be able to make the whole library header-only, but we will try to. If some component is header-only, it will not require linking to Boost.Sync. We plan to add new components to Boost.Sync after the extraction is made. The current candidates for addition are: - spin_mutex and yield_spin_mutex. There are several implementations in Boost, namely in Boost.SmartPtr, Boost.Log and Boost.Atomic. The plan is to refine them into two flavors of spin mutex and move to Boost.Sync and then switch these libraries to the common implementation, if possible. - once blocks from Boost.Log. - portable Windows-style events. There is a prototype implementation in Boost.Log. Boost.Thread will depend on Boost.Sync and will retain headers and possible glue code for backward compatibility. For instance, I think it will have to implement interruptible condition variable. Comments are welcome.

Future/promise, packaged_task, async and c++xy extensions would then be moved in Boost.Async ? I got no problem with the split, I was thinking about something vaguely similar recently. Joel Lamotte

On Saturday 17 August 2013 13:35:57 Klaim - Joël Lamotte wrote:
Future/promise, packaged_task, async and c++xy extensions would then be moved in Boost.Async ?
We haven't discussed Boost.Async, just Boost.Sync for now. It seems possible, although I wonder what will be left in Boost.Thread then? Basic thread management?

On Sat, Aug 17, 2013 at 1:41 PM, Andrey Semashev
We haven't discussed Boost.Async, just Boost.Sync for now. It seems possible, although I wonder what will be left in Boost.Thread then? Basic thread management?
Sorry that was a typo, I meant Boost.Sync yes. It's true that Boost.Thread would host only boost::thread and interruptions then, so the split I was wondering about might not really helpful, nevermind. I was just wondering because future/promise/async are, to me, synchronization mechanisms, but high-level compared to the ones you already want to separate, so I would have expected to find them with other "synchronization" mechanisms, OR in a high-level synchronization mechanism library. Future/promise/aysnc are all packed into <future> in the standard library by the way, not sure if it's the best grouping but it's notable data. It's just some thoughts so don't take my comment as suggestion or preference, i was just wondering. In my current code base I use almost all current Boost.Thread constructs so the split wouldn't impact me immediately I think. Joel Lamotte

We haven't discussed Boost.Async, just Boost.Sync for now. It seems possible, although I wonder what will be left in Boost.Thread then? Basic thread management?
Sorry that was a typo, I meant Boost.Sync yes. It's true that Boost.Thread would host only boost::thread and interruptions then, so the split I was wondering about might not really helpful, nevermind. I was just wondering because future/promise/async are, to me, synchronization mechanisms, but high-level compared to the ones you already want to separate, so I would have expected to find them with other "synchronization" mechanisms, OR in a high-level synchronization mechanism library. Future/promise/aysnc are all packed into <future> in the standard library by the way, not sure if it's the best grouping but it's notable data.
i'd rather have boost.sync provide only low-level threading classes. especially it should not depend on boost.thread to avoid cyclic dependencies. we already have the awkward situation that sync will depend on atomic, which will depend on spinlocks defined in sync ...

On Saturday 17 August 2013 13:54:57 Klaim - Joël Lamotte wrote:
On Sat, Aug 17, 2013 at 1:41 PM, Andrey Semashev
wrote: We haven't discussed Boost.Async, just Boost.Sync for now. It seems possible, although I wonder what will be left in Boost.Thread then? Basic thread management?
Sorry that was a typo, I meant Boost.Sync yes. It's true that Boost.Thread would host only boost::thread and interruptions then, so the split I was wondering about might not really helpful, nevermind. I was just wondering because future/promise/async are, to me, synchronization mechanisms, but high-level compared to the ones you already want to separate, so I would have expected to find them with other "synchronization" mechanisms, OR in a high-level synchronization mechanism library. Future/promise/aysnc are all packed into <future> in the standard library by the way, not sure if it's the best grouping but it's notable data.
We don't plan to extract futures and related stuff, it's too coupled with Boost.Thread. The main goal of Boost.Sync is to provide lightweight building blocks for low level thread synchronization and futures are beyond that.

On Sat, Aug 17, 2013 at 2:47 PM, Andrey Semashev
We don't plan to extract futures and related stuff, it's too coupled with Boost.Thread. The main goal of Boost.Sync is to provide lightweight building blocks for low level thread synchronization and futures are beyond that.
Ok!

2013/8/17 Andrey Semashev
We don't plan to extract futures and related stuff, it's too coupled with Boost.Thread. The main goal of Boost.Sync is to provide lightweight building blocks for low level thread synchronization and futures are beyond that.
futures usually depend only on mutex and condition_variable - at least it is possible to implement it this way. I would vote to decouple boost.future from boost.thread (so it could be used from other execution primitives than threads).

On Saturday 17 August 2013 17:58:52 Oliver Kowalke wrote:
2013/8/17 Andrey Semashev
We don't plan to extract futures and related stuff, it's too coupled with Boost.Thread. The main goal of Boost.Sync is to provide lightweight building blocks for low level thread synchronization and futures are beyond that.
futures usually depend only on mutex and condition_variable - at least it is possible to implement it this way. I would vote to decouple boost.future from boost.thread (so it could be used from other execution primitives than threads).
You can use future and promise to pass asynchronous values, true. But still, its primary use is with threads. Things like async() and future.then() use threads to carry out actions. Even if you don't use these features, you can still use futures from Boost.Thread. Note that I'm not saying there's no point in loosening this dependency of futures on threads. I'm saying that Boost.Sync doesn't look like a proper place for futures.

2013/8/17 Andrey Semashev
You can use future and promise to pass asynchronous values, true. But still, its primary use is with threads. Things like async() and future.then() use threads to carry out actions. Even if you don't use these features, you can still use futures from Boost.Thread.
I mean using fibers instead threads
Note that I'm not saying there's no point in loosening this dependency of futures on threads. I'm saying that Boost.Sync doesn't look like a proper place for futures.
maybe boost.future?

On Saturday 17 August 2013 20:48:49 Oliver Kowalke wrote:
2013/8/17 Andrey Semashev
You can use future and promise to pass asynchronous values, true. But still, its primary use is with threads. Things like async() and future.then() use threads to carry out actions. Even if you don't use these features, you can still use futures from Boost.Thread.
I mean using fibers instead threads
You can't safely use regular mutexes and condition variables with fibers, can you? If you block, you won't switch to another fiber, which you may be waiting for.
Note that I'm not saying there's no point in loosening this dependency of futures on threads. I'm saying that Boost.Sync doesn't look like a proper place for futures.
maybe boost.future?
Maybe. That's along the lines of the Boost.Async suggestion.

2013/8/17 Andrey Semashev
You can't safely use regular mutexes and condition variables with fibers, can you? If you block, you won't switch to another fiber, which you may be waiting for.
it's possible - I'll publish my fiber lib soon (contains mutex, condition_variable, future, ...)

On Sat, Aug 17, 2013 at 8:39 PM, Andrey Semashev
Things like async() and future.then() use threads to carry out actions. Even if you don't use these features, you can still use futures from Boost.Thread.
That being said, I have custom code that works like async and future but take a scheduler callable (like the coming implementation) and I use work queues (associated with a thread or an update task pushed in a task scheduler) instead of threads or fiber. I'm not sure to understand how future and async really requires thread itself. to me it's only building synchronization structures between two potentially concurrent systems, whatever the way concurrency is achieved. At some point there is threads running, but it still look independant technically. Joel Lamotte

Le 17/08/13 17:58, Oliver Kowalke a écrit :
2013/8/17 Andrey Semashev
We don't plan to extract futures and related stuff, it's too coupled with Boost.Thread. The main goal of Boost.Sync is to provide lightweight building blocks for low level thread synchronization and futures are beyond that.
futures usually depend only on mutex and condition_variable - at least it is possible to implement it this way. Boost.Thread future follows the C++11 standard and the ..._at_thread_exit functions depend deeply on the thread class.
The async function such as it is defined today depends also on boost::thread. Of course, we can provide an async function that has an execution context as parameter that doesn't depend on Boost::thread.
I would vote to decouple boost.future from boost.thread (so it could be used from other execution primitives than threads).
I think we can provide a minimal future abstraction that can be used with other execution contexts, but it would need to have as parameter the associated mutex and condition_variable abstraction, otherwise I don't see how we can do it. Maybe you have found how. If yes, I will be interested in knowing how? Best, Vicente

2013/8/18 Vicente J. Botet Escriba
I think we can provide a minimal future abstraction that can be used with other execution contexts, but it would need to have as parameter the associated mutex and condition_variable abstraction, otherwise I don't see how we can do it.
That's exactly what I've had in mind. namespace boost { namespace futures { template< typename R, typename Mtx, typename Cnd > class future; }} boost.thread: template< typename R > typedef futures::future< R, mutex, condition_variable > future; // uses threads thread, mutex and condition_vairable ,... are defined in the 'boost' namespace (which is an exception to other boost-libs). boost.fiber: namespace boost { namespace fibers { template< typename R > typedef futures::future< R, mutex, condition_variable > future; // uses fibers }}

On 08/18/2013 11:31 PM, Vicente J. Botet Escriba wrote:
I think we can provide a minimal future abstraction that can be used with other execution contexts, but it would need to have as parameter the associated mutex and condition_variable abstraction, otherwise I don't see how we can do it. Maybe you have found how. If yes, I will be interested in knowing how?
It the most extreme case where no synchronization is needed, we have Boost.Expected (once it is finalized.)

Le 19/08/13 13:53, Bjorn Reese a écrit :
On 08/18/2013 11:31 PM, Vicente J. Botet Escriba wrote:
I think we can provide a minimal future abstraction that can be used with other execution contexts, but it would need to have as parameter the associated mutex and condition_variable abstraction, otherwise I don't see how we can do it. Maybe you have found how. If yes, I will be interested in knowing how?
It the most extreme case where no synchronization is needed, we have Boost.Expected (once it is finalized.)
Your are almost right. There are some differences between boost::expected and a future not synchronized. In particular: * future is default constructible, while expected would ensure the non-empty guaranty as boost::variant does, that is expected provides an immediate value, while future provides, well, a future value. * futures pairwise with promises, while expected is a stand alone class, * futures are movables-only, while expected can be copy_constructible, * ... Best, Vicente

We plan to add new components to Boost.Sync after the extraction is made. The current candidates for addition are:
semaphores as in [1]. [1] https://svn.boost.org/trac/boost/ticket/8797
participants (6)
-
Andrey Semashev
-
Bjorn Reese
-
Klaim - Joël Lamotte
-
Oliver Kowalke
-
Tim Blechmann
-
Vicente J. Botet Escriba