Howard Hinnant wrote: [...]

In essence, we're asking the C community to standardize the semantics of try/finally (no catch, no throw)

Nope. It's more like http://www.ohse.de/uwe/articles/gcc-attributes.html#var-cleanup or, in pthread/C++ speak, just /* * See http://www.cuj.com/documents/s=8000/cujcexp1812alexandr/ */ #define pthread_cleanup_push(routine, arg) \ { \ scope_guard_impl const & _guard = make_scope_guard(arg, routine); #define pthread_cleanup_pop(execute) \ if (!execute) _guard.dismiss(); \ } regards, alexander.

Howard Hinnant wrote:

On Mar 25, 2007, at 12:03 AM, Emil Dotchevski wrote:

...

On Windows, you can have two joins on the same thread. Copyable thread handles can support this behavior directly. How does one achieve the same thing with N2184::thread semantics?

You put a N2184::thread on the heap, with a condition variable, mutex and reference count accompanying it. You then create a copyable handle to this struct which manages the lifetime with the reference count, and access to the thread (and other state such as whether the thread as ended or not).

It probably should be noted here that a Windows implementation of N2184 can support multiple concurrent joins (and try/timed joins) directly without much effort. Based on a cursory look it seems that Anthony's implementation does (minus the try/timed joins, of course, but they appear trivial to add). As a motivating example for try joins, you might consider doing a prototype of a thread pool that dynamically varies the number of its threads. (Spawning new threads when a task is waiting more than a set amount of time avoids deadlocks in the general case.) One would like the 'extra' threads to terminate after a set amount of inactivity, and one would also like to periodically sweep the pool container to remove these terminated threads from it. I'll probably need to produce such a thread pool based on N2178::handle anyway to address the potential deadlock in my implementation of N2185. Here's a link that discusses the problem with fixed thread pools: http://www.bluebytesoftware.com/blog/PermaLink,guid,ca22a5a8-a3c9-4ee8-9b41-...

Howard Hinnant wrote:

N2184 takes the minimalist approach. It is easier to add functionality (and the associated expense) than to subtract it. There are many things the N2184::thread does not directly support. I believe this is a feature, not a bug.

This is true under the assumption that we need to target the least common denominator. Try/timed joins for example come for free on Windows. So you are imposing an unnecessary mutex+cv overhead for everyone wanting to use try/timed joins there. I've decided to adopt a different approach and suggest a way to equalize ('harmonise' in EU terms :-) ) the platforms via the join2 extensions. In a perfect world, this would lead to everyone enjoying _zero overhead_ try/timed joins in a few years once pthread implementors adopt the extensions. Yes, I agree that this can be considered idealistic; but the other option is to not even give them a chance to offer the functionality as there would be no portable C++ way to take advantage of it.

Yuval Ronen wrote:

Peter Dimov wrote:

...

Howard Hinnant wrote:

...

N2184 takes the minimalist approach. It is easier to add functionality (and the associated expense) than to subtract it. There are many things the N2184::thread does not directly support. I believe this is a feature, not a bug.

This is true under the assumption that we need to target the least common denominator. Try/timed joins for example come for free on Windows. So you are imposing an unnecessary mutex+cv overhead for everyone wanting to use try/timed joins there.

I've decided to adopt a different approach and suggest a way to equalize ('harmonise' in EU terms :-) ) the platforms via the join2 extensions. In a perfect world, this would lead to everyone enjoying _zero overhead_ try/timed joins in a few years once pthread implementors adopt the extensions.

Yes, I agree that this can be considered idealistic; but the other option is to not even give them a chance to offer the functionality as there would be no portable C++ way to take advantage of it.

I'm far from being an expert on the Windows internals, but maybe what seems to "zero overhead" try/timed join is not really zero overhead, and the overhead is there, just hidden somewhere inside. If that's the case, then the "idealistic" way to go is to get Windows to supply a true zero overhead join.

Basically you're just waiting for a thread handle to become signalled, much as waiting for an event, mutex or whatever. No more overhead than that, AFAIK. / Johan

Yuval Ronen wrote:

Johan Nilsson wrote:

...

Yuval Ronen wrote:

...

...

Howard Hinnant wrote:

...

N2184 takes the minimalist approach. It is easier to add functionality (and the associated expense) than to subtract it. There are many things the N2184::thread does not directly support. I believe this is a feature, not a bug. This is true under the assumption that we need to target the least common denominator. Try/timed joins for example come for free on Windows. So you are imposing an unnecessary mutex+cv overhead for everyone wanting to use try/timed joins there.

I've decided to adopt a different approach and suggest a way to equalize ('harmonise' in EU terms :-) ) the platforms via the join2 extensions. In a perfect world, this would lead to everyone enjoying _zero overhead_ try/timed joins in a few years once pthread implementors adopt the extensions.

Yes, I agree that this can be considered idealistic; but the other option is to not even give them a chance to offer the functionality as there would be no portable C++ way to take advantage of it. I'm far from being an expert on the Windows internals, but maybe what seems to "zero overhead" try/timed join is not really zero overhead, and

Peter Dimov wrote: the overhead is there, just hidden somewhere inside. If that's the case, then the "idealistic" way to go is to get Windows to supply a true zero overhead join.

Basically you're just waiting for a thread handle to become signalled, much as waiting for an event, mutex or whatever. No more overhead than that, AFAIK.

One overhead I can think of is the need to call CloseHandle after the WaitForSingleObject, which means another OS call (kernel?).

Yes, kernel.

This doesn't exist on pthreads where calling pthread_join is enough. So in case of a single joiner, it seems pthreads is better, at least with respect to the CloseHandle call (and there might be other issues). Is the CloseHandle overhead significant?

When the last handle to the thread is closed, the kernel thread structure must be cleaned-up. Whether that results in significant overhead or not depends on the implementation. It could possibly be as simple as moving the structure pointer to a "free list". I tried to search through my copy of "Inside Windows 2000", but didn't find anything fast enough to give you a reply ... Still, yes, an extra kernel mode transition.

I really don't know. If it's not much, and there's really a way to implement zero-overhead multi/try/times joins, then that would certainly change my perspective.

Is it possible to implement join reliably/effectively under Win32 without resorting to kernel objects? All kernel objects still need to have their references (Win32 handles etc) closed to avoid resource leakage during process lifetime. / Johan

A few days late, but... Peter Dimov wrote:

Yuval Ronen wrote:

...

One overhead I can think of is the need to call CloseHandle after the WaitForSingleObject, which means another OS call (kernel?). This doesn't exist on pthreads where calling pthread_join is enough.

The overhead in this case is not measured in cycles or kernel transitions (remember that you called a blocking function because you have nothing better to do with the CPU).

The kernel transitions take CPU cycles, which means these cycles won't be available to any thread, not just the one that joined. AFAIU, all threads in all process would suffer from this overhead, because CPU cycles were wasted. On the other hand, one might say that if the total cost of creating + destroying a thread is much more than a single kernel transition (is it?), then adding one such transition is negligible.

It's measured in kernel objects; these come from the nonpaged pool and are a (relatively) limited resource. Since (today) a cv under Windows is usually ~3 kernel objects, a mutex one more, adding a mutex+cv to every thread increases its kernel footprint five times.

But are all kernel objects created equal? If a thread object is much "heavier" than a mutex or c/v object, (which means, for instance, there's much fewer of it), than the cost if much less than five times. As I said, I have no idea if that's really the case. Just raising questions... (Oh, and BTW, this cost doesn't need to be added to every thread, only to those who need multi joins)

Howard Hinnant wrote:

layer (and that has been my sense from the boost community as well: "I want boost + cancellation").

May I drop in an innocent (or blaspheming?) question? Why do we want cancellation? Which kind of cancellation do we want (if at all)? Do we really need true async cancellation, or suffices to be able to control a thread at certain (well defined) points. Wouldn't 90% of the users be happy if they just be able to instruct a thread to stop while it is in an (idling) wait loop without need to resort to custom (boiler plate) IPC mechansims? Roland

On Mar 26, 2007, at 3:16 PM, Roland Schwarz wrote:

Howard Hinnant wrote:

...

layer (and that has been my sense from the boost community as well: "I want boost + cancellation").

May I drop in an innocent (or blaspheming?) question?

Please do! :-)

Why do we want cancellation?

Good question. It is part of the pthreads model and has been requested a lot from clients of boost::thread. It arguably (and I didn't realize this at first) provides the needed semantics of ~thread(): thread::~thread() { if (joinable()) { cancel(); detach(); } } I.e. if the parent thread leaves the room unexpectedly, what do you want to happen to the child thread? boost::thread simply detaches. And upon reflection, I think that can delay resource cleanup of the child thread for an unreasonably long time.

Which kind of cancellation do we want (if at all)?

In pthreads vocabulary: only deferred. Nobody even wants to talk about asynchronous cancellation. But some, because of the pthreads history, would prefer us to use a different name for "cancel" so that there is no such confusion.

Do we really need true async cancellation, or suffices to be able to control a thread at certain (well defined) points.

<nod> Only the latter.

Wouldn't 90% of the users be happy if they just be able to instruct a thread to stop while it is in an (idling) wait loop without need to resort to custom (boiler plate) IPC mechansims?

That is essentially what we're after. N2184 recommends very few cancellation points: • void std::this_thread::cancellation_point() • void std::this_thread::sleep(std::nanoseconds rel_time) • void std::thread::join() • void std::thread::operator()() • void std::condition<Lock>::wait(Lock&) • template<class Predicate> void std::condition<Lock>::wait(Lock&, Predicate) • bool std::condition<Lock>::timed_wait(Lock&, std::nanoseconds) • template<class Predicate> void std::condition<Lock>::timed_wait(Lock&, std::nanoseconds, Predicate) N2178 has the same outlook but recommends considerably more cancellation points (the entire posix set if I understand correctly). -Howard

On Mar 26, 2007, at 3:47 PM, Peter Dimov wrote:

Howard Hinnant wrote:

...

N2178 has the same outlook but recommends considerably more cancellation points (the entire posix set if I understand correctly).

N2178 doesn't recommend a specific set of cancelation points. I expect that the standard would need to define a list of mandatory C++ cancelation points (potentially covering more than just the threading portion) and a list of optional cancelation points (which would be similar to the POSIX list of optional cancelation points with a few subtractions such as fclose.)

Thanks for the clarification. If it helps (the readers) here's a link to the posix cancellation points (and optional points): http://www.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_09.html#t... The rationale for my conclusion about N2178 was that once we hand control over to pthread_cancel, we're pretty much in the hands of the posix committee (or at least the posix implementors). We may be able to make some agreements with them. -Howard

on Mon Mar 26 2007, Roland Schwarz <roland.schwarz-AT-chello.at> wrote:

What I really meant by "do we need cancellation" is the inherent meaning of cancellation: once started you can't stop it.

That's not inherent. You're stepping into a minefield here...

Your example seems to imply this, but I don't think this necessarily is the case.

"Everybody" in the C++ community wants cancellation to be an ordinary, stoppable, C++ exception, thrown only synchronously, at well-defined cancellation points. Anything else makes writing cancellation-safe code basically untenable. In fact, the only person I know of in *any* community who insists that cancellation must be unstoppable is Ullrich Drepper.

Another point (not a strict technical one though): Developing a mechanism that could be used for cancellation but strictly speaking _is_ _not_ cancellation could end the stalling discussions about cancellation, which are lasting almost forever now.

I suggest that trying to use meanings of "cancellation" other than the one(s) used in the pthread standard at this point can only confuse things, which hardly seems like a way to un-stall the process. -- Dave Abrahams Boost Consulting www.boost-consulting.com

on Tue Mar 27 2007, Alexander Terekhov <terekhov-AT-web.de> wrote:

David Abrahams wrote: [...]

...

"Everybody" in the C++ community wants cancellation to be an ordinary, stoppable, C++ exception, thrown only synchronously, at well-defined cancellation points.

Or asynchronously from well-defined async-cancel-safe regions (ideally with async-cancel-safety of code inside async_cancel {} regions statically checked by compiler).

Yes, given more C++ core language changes, async cancel safety could be seen as viable under carefully controlled circumstances.

...

Anything else makes writing cancellation-safe code basically untenable. In fact, the only person I know of in *any* community who insists that cancellation must be unstoppable is Ullrich Drepper.

Perhaps someone should try to explain to him that nobody really wants to allow longjmp() from pthread_cleanup_push()'s handlers.

Be my guest Alexander; you're just the man for the job! -- Dave Abrahams Boost Consulting www.boost-consulting.com

Roland Schwarz <roland.schwarz@chello.at> writes:

David Abrahams wrote:

...

I suggest that trying to use meanings of "cancellation" other than the one(s) used in the pthread standard at this point can only confuse things, which hardly seems like a way to un-stall the process.

I might be completely wrong, but doesn't the pthread standard require a thread to stop once it has been cancelled? The thread may decide to defer the request, but not to deny. Correct?

Well, it may defer the request forever by never enabling cancellation, or the code may register a cleanup handler that essentially restarts processing.

So what we are discussing (and what I want to suggest too) is to let the thread decide if it honours the cancellation request.

Yes.

My point was just to give it a different name then, since this basically is a communication channel that can be used for other purposes too.

Fair enough. If you make it generic (e.g. t.raise(some_exception)), then you have to write code to deal with arbitrary exceptions being thrown from what essentially amounts to arbitrary points. t.raise(std::bad_alloc()); namespace{ struct type_with_unknown_name{}; } t.raise(type_with_unknown_name()); Aside from the exception-transportation problems, I'm not keen on this. t.request_cancel() says what we really mean, and t.cancel() strikes me as acceptable shorthand. Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL

On Mar 26, 2007, at 12:44 PM, Peter Dimov wrote:

Howard Hinnant wrote:

...

Do we want such sharing *implicitly*? By default? In the *foundation* class?

There is a difference between not wanting sharing implicitly by default and actively not supporting it by making all operations non-const and not thread safe.

Fair point. Perhaps we should go back to a more fundamental question: std::thread t(f); ... t.cancel(); // Logically is this a const or non-const operation on the thread executing t/f? To answer that question probably involves the very definition of cancellation. The "t.cancel()" can reasonably be interpreted in two different ways: 1. Owner to t: I'm no longer interested in your computation (but somebody else might be). 2. Owner to t: I want you to clean-up and stop as soon as possible. I believe the shared ownership handle will actually have to support both semantics (the first belongs in the handle's destructor). The sole ownership handle outlined in N2184 (directly) supports only the second interpretation. And the second interpretation seems to me to be a non-const operation on t. It must set thread state in t (either external or internal to the OS thread). void foo(const thread& t); void bar() { std::thread t(f); ... foo(t); ... // Is it useful to be able to assume things about t here? For example: // t hasn't been canceled, or moved from. // Perhaps foo() tested t's identity or queried it for a pending cancellation } -Howard

On Mar 26, 2007, at 1:56 PM, Stefan Seefeld wrote:

Howard Hinnant wrote:

...

Fair point. Perhaps we should go back to a more fundamental question:

std::thread t(f); ... t.cancel(); // Logically is this a const or non-const operation on the thread executing t/f?

To answer that question probably involves the very definition of cancellation. The "t.cancel()" can reasonably be interpreted in two different ways:

1. Owner to t: I'm no longer interested in your computation (but somebody else might be). 2. Owner to t: I want you to clean-up and stop as soon as possible.

Isn't the first point more in line with what in POSIX threads is described by a 'detach' ? (once a thread is detached, it can't be joined any more. It still can be canceled.

I was thinking about that when I wrote this list and decided not quite. Detach is more reasonably interpreted as: I want you to continue to work independently. Clean up after yourself when you're done. I.e. with 1 I'm trying to convey that I don't care whether the child thread lives or dies, does more work or not. Whereas with 2 I'm trying to say that I don't want the child to do any more work. And with detach I'm trying to say, I want you to continue your work, just I'm going elsewhere, turn out the lights when you're done. In the pthreads world, there is no direct support for 1. And 2 maps to pthread_cancel. -Howard

On Mar 26, 2007, at 2:04 PM, Peter Dimov wrote:

Howard Hinnant wrote:

...

Fair point. Perhaps we should go back to a more fundamental question:

std::thread t(f); ... t.cancel(); // Logically is this a const or non-const operation on the thread executing t/f?

I prefer to first ask "does it matter"? If you can't observe the cancelation state of t and if you are assured that t.cancel is valid no matter what else is going on with t, what difference does a const/non-const label make to the external observer?

(Internally there is a flag being set, of course.)

I believe it matters because other threads can detect whether the child thread has been canceled or not. Canceled may be a valid state. But it is a different and detectable state. Here's a simple HelloWorld demonstrating it: #include "thread" #include <iostream> std::thread t; std::mutex mut; std::condition<std::exclusive_lock<std::mutex> > cv; bool work_done = false; void f() { std::this_thread::sleep(2); { std::exclusive_lock<std::mutex> lk(mut); work_done = true; } cv.notify_all(); } void f2() { std::this_thread::sleep(1); // t.cancel(); } int main() { t = std::thread(f); std::thread t2(f2); std::exclusive_lock<std::mutex> lk(mut); while (!work_done) cv.wait(lk); std::cout << "Success!\n"; } This prints out "Success!" as it is now. But if you comment out "t.cancel()" in f2 then main hangs forever. f2 did something logically non-const to the separate thread f which in turn caused the main thread to react differently. This seems to me like f2 is doing something non-const. -Howard

On Mar 26, 2007, at 2:47 PM, Peter Dimov wrote:

But my question is more along the lines of: why does the const/non- const label matter when we decide on the precondition and thread safety of t.cancel? If t.cancel is always valid on a valid t, this makes sharing possible and safe, regardless of whether thread::cancel is declared const.

I guess my answer would go back to this snippet: void foo(const thread& t); void bar() { std::thread t(f); ... foo(t); ... } Because of the const qualifier on the thread& which foo takes as a parameter, and knowing that the const interface of thread is a subset of the non-const interface, we can make useful assumptions about t after the call to foo() (such as t hasn't been canceled). This might be particularly helpful if foo is a virtual function. -Howard

On Mar 26, 2007, at 3:24 PM, Peter Dimov wrote:

Or, stated differently,

Why should a thread::cancel that is declared non-const necessarily be made thread and sharing unsafe?

The sole-ownership model generally implies that only one thread has access to the std::thread (i.e. like std::fstream to the file on disk, or lock<mutex> to the mutex. There are somethings which *must* be thread safe anyway: t.cancel() must be made safe with respect to this_thread::cancellation_requested() since we have one thread writing to the same value that another thread is trying to read. t.cancellation_requested() must be thread safe with respect to the innards of this_thread::cancellation_point() for the same reasons but in reverse (the latter is writing while the former is reading). However I don't want to add cost to support idioms which are inherently unsafe in the pthread model anyway, for example: std::thread t(f); // global thread A thread B t.join() t.cancel() In pthreads this is a race. Bad things can happen if thread A gets there first. If an implementation of std::thread happens to tolerate this race, I have no problem with that. But I don't want to require all implementations to add extra synchronization to make this race well defined. N2184 is attempting to just model pthreads semantics. Alternatively if we have: thread A thread B t.cancel() t.cancel() then this should be a harmless race, both in pthreads and in N2184. The implementation is already having to use atomics (or whatever) to set the cancellation pending flag to protect it against reads in the t thread. But: thread A thread B t.detach() t.cancel() is again a race condition in both pthreads and N2184 (N2184 does a lousy job of clarifying all this). So bottom line: N2184 isn't aiming to go out of its way to make sharing unsafe. It is attempting to add just enough cost to support pthreads semantics and no more. -Howard

On Mar 26, 2007, at 2:04 PM, Peter Dimov wrote:

...

Howard Hinnant wrote:

...

Fair point. Perhaps we should go back to a more fundamental question:

std::thread t(f); ... t.cancel(); // Logically is this a const or non-const operation on the thread executing t/f?

I prefer to first ask "does it matter"? If you can't observe the cancelation state of t and if you are assured that t.cancel is valid no matter what else is going on with t, what difference does a const/non-const label make to the external observer?

(Internally there is a flag being set, of course.)

I believe it matters because other threads can detect whether the child thread has been canceled or not. Canceled may be a valid state. But it is a different and detectable state.

Perhaps shared_ptr/weak_ptr is a good analogy. I can have shared_ptr<int const>, and if this is the last reference to expire it'll delete the int (despite that mutable operations on the int are disallowed.) I can also have a weak_ptr<int const> to the same object, and I can detect the fact that it has been destroyed. Yet the object has no "destroyed" state. Emil Dotchevski

Howard Hinnant wrote:

1. Owner to t: I'm no longer interested in your computation (but somebody else might be). 2. Owner to t: I want you to clean-up and stop as soon as possible.

Aren't we trying to ask for too much? If we were able to thread t(); ... t.raise(exception); throw an exception at a thread, the thread itself can take whatever action is sensible at this point. It could just change its current path of control, it could stop (after having reestablished all invariants), it can propagate the exception to its "parent". Just let the user decide what is the best strategy for the particular thread. This scheme, while not mentioning cancellation at all is able to deliver almost everything cancellation also could. The difference "throwing an exception at a thread" is not an all or nothing decision. Roland

Howard Hinnant wrote:

t.cancel() is the same as t.raise(std::thread_canceled)

But semantics of cancel is different, no? Cancel is expected to bring the thread down, yes? Not so raise, it just is sending kind of a signal to the thread. (The slot being catch clause).

Beman and Peter have been active in the area of propagating exceptions from t back through the join. You are the first (that I know of) to suggest the generalized propagation the other way. :-) Do you have use cases other than: t.raise(std::thread_canceled)? Any implementation strategies?

I implemented a small prototype and used it already in an application. I was not using exceptions at this time, as throwing the exceptions I considered not the hard part. I was seeing this syntactic sugar at that time. More interesting was how to establish the interruptible (or "alertable") state. Al though at that time I implemented it as a strict add on to boost thread, because I just wanted to experiment with the usability of the interface. It could be improved a lot I believe if it were tighter integrated into boost:thread. (I needed to resort to notify_all internally e.g, where the program logic would only demand for notify_one.) I uploaded it to the vault under a name "alertable thread" if my memory is not fading... Roland

Howard Hinnant wrote:

I swear (lays hand on backup disk) on my backup disk, all N2184 (and N2178 as I understand it) wants of C++ cancellation is to request that the target thread (whenever it gets around to it, if ever) throw a normal C++ exception, which the target thread is free to subsequently catch, swallow and digest. :-)

To be honest. I did not yet had time to study N2184 and 2178. But if this is true, wouldn't this be in contradiction to pthread semantics? Wouldn't this users rather confuse even more? Sorry if this sounds innocent. I really mean this as a question. Roland

On Mar 26, 2007, at 6:10 PM, Stefan Seefeld wrote:

Roland Schwarz wrote:

...

Howard Hinnant wrote:

...

I swear (lays hand on backup disk) on my backup disk, all N2184 (and

...

N2178 as I understand it) wants of C++ cancellation is to request that

...

the target thread (whenever it gets around to it, if ever) throw a

...

normal C++ exception, which the target thread is free to subsequently

...

catch, swallow and digest. :-)

To be honest. I did not yet had time to study N2184 and 2178. But if this is true, wouldn't this be in contradiction to pthread semantics? Wouldn't this users rather confuse even more?

Indeed, I think these are two very different things:

1) Request a thread cancelation.

This is what in POSIX threads is called pthread_cancel(), and what you presumably mean by "t.raise(cancel_exception)".

2) Notify that a thread has been canceled.

This is what would presumably result in a thread_canceled exception being thrown.

As far as I understand, the desired semantics here would be 'deferred', i.e. 1) is really only a request, i.e. at the point this call returns there is no guarantee that the cancellation has been terminated (or even initialized).

Note that you suggested "t.raise(std::thread_canceled)", which sounds misleading, since 'canceled' suggests this is raised at the point where the cancelation is finished, not requested.

For that matter I don't really see the point in using an exception to signal the 'cancel request' to the target thread, while the use of an exception in 2) makes a lot of sense, since the main discussion (as far as POSIX threads & C++ is concerned) is about the cleanup, i.e. the missing stack unwinding semantics.

(Part of the discussion is about whether the exception can be caught without being rethrown in handlers such as 'catch (...) {}', or whether catch(...) should see thread_canceled at all, etc.)

Beman was supposed to pipe up by now. :-) He has suggested "request_cancel()" or "request_cancellation()" to replace "cancel()" to make it absolutely clear this is a request, not an imperative. I prefer the shorter "cancel()", but it is definitely not an "over my dead body" issue for me (nor is it I think for Beman). I am not aware of anyone in C++ circles who is currently recommending anything other than the "request" semantics. There are people on the posix committee however who think we're nuts for wanting to allow threads to catch their own cancellation exception. Thus the problem with delegating to pthread_cancel... My killer use case for the catchable cancel is a thread_pool library: Such a library would execute a small number of threads, and hold a queue of waiting tasks which get assigned to a thread when one is available. Clients of thread_pool can insert tasks into the queue, and when they do so they receive back a handle (which I'll call future) through which they can join with, or cancel, the inserted task. If the client decides to cancel the task, this translates into an exception being thrown in the running thread. Up in the start function which wraps/adapts the user-supplied function, that exception is caught, and the still running thread is returned to idle status in the thread_pool, ready to accept another task from the pool's queue. One might wonder if thread cancellation is the right semantics for this action. Perhaps some other exception could be thrown. The disadvantage with this is that the task might need to catch/rethrow the cancellation exception during clean up as it is being canceled. It would be a shame if it needed to watch for two different flavors of cancellation to do this (one kind if you're executing in a regular thread, another kind if you're executing in a thread pool). One could design the thread_pool to let such canceled tasks continue to cancel the underlying thread. The thread_pool could be alerted to this fact and simply start a new thread to replace the canceled one. Imho, this would be inelegant, and inefficient by comparison. Fwiw, here's a link to the cancellation description in N2184: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2184.html#Examples Search for "Canceling threads" once you hit the link. It is little more than a page of examples and description (not a formal spec). -Howard

Roland Schwarz wrote:

Howard Hinnant wrote:

...

1. Owner to t: I'm no longer interested in your computation (but somebody else might be). 2. Owner to t: I want you to clean-up and stop as soon as possible.

Aren't we trying to ask for too much?

If we were able to

thread t(); ... t.raise(exception);

throw an exception at a thread, the thread itself can take whatever action is sensible at this point.

Not sure what "to throw at" means in the context of C++ exceptions. (I'm afraid we are deviating from the original topic, though.) With exception-throwing comes stack-unwinding, so the thrower has to be on top of the call stack, not someone outside (or even in another thread). The point really is that we want a means to interrupt an otherwise non-interruptible (i.e. blocking) call, so this requires support from the system layer. Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...

Howard Hinnant wrote:

I was specifically trying to indicate a simplified (and safer) interface.

[...]

Do we want such sharing *implicitly*? By default? In the *foundation* class?

I don't know if you remember it, but a while ago I tried to convince you that an N2178 handle is conceptually a lower level primitive than an N2184 thread. This is extremely hard to sell to someone who has the implementations of both before his eyes and can plainly see that implementation-wise, N2184::thread is closer to pthread_t than N2178::handle. But if you consider both on an abstract level, it's easier to accept. An N2184 thread can be built on top of an N2178::handle by just wrapping it and disabling copyability. And an N2178::handle is obviously more powerful and more unsafe, as opposed to N2184::thread which is simplified and safer. I have absolutely no problems with the idea that the default and recommended primitive needs to be N2184::thread (minus cancelation_requested plus thread safety)... as long as we also deliver a powerful interface for the power users.

Howard Hinnant wrote:

When it comes right down to it, we have two valid and reasonable OS threading models before us:

1. pthreads 2. Windows

Which one do we model with std::thread (or std::thread::handle)?

Neither. We model a synthetic model that has the best qualities of each, ...

N2184 chose pthreads and N2178 chose Windows (to be fair, N2178 chose Windows with some pthreads mixed in).

... accepting the inevitable downside that the pthread people would label it Windows with some pthreads mixed in, while the Windows people would label it pthreads with some Windows mixed in.

Wow. I leave things for a few days and there's loads of messages to deal with. Re cancellation: Everyone I've ever spoken to about C++ having standard thread support has said "excellent, that'll mean we'll have proper cancellation support" or similar. IMO, we can't afford to not have cancellation. Cancellation must operate as-if it's an exception, so stacks get unwound correctly. It must also get caught by catch(...) so catch(...) { cleanup(); throw;} works correctly. In which case, it must be stoppable --- I don't want to introduce an exception that automatically gets rethrown, as that would be a dangerous precedent. All of which basically means it must be a real C++ exception, so why not give it a name (std::thread_canceled)? If this means we can't interoperate with pthread_cancel, so be it. I'd rather pthread_cancel threw std::thread_canceled, though, as that makes it easier to deal with mixed C/C++ code, and allows for more cancellation points. Re single/shared ownership and const/non-const operations I think that cancel/join/detach are non-const operations, as they conceptually modify the state of the thread. I also think this is orthogonal to single/shared ownership. I happen to currently favour single ownership, but that is also separate from shared access --- just because multiple threads or functions can access the std::thread object doesn't mean they "own" it. Single ownership does mean that you can pass a const ref when you want someone to be able to observe the thread but not modify it. I think that all operations on std::thread should be safe to execute concurrently, and that they should have defined functionality in all cases, which should probably be a no-op. Thread A Thread B Comment t.cancel() t.cancel() harmless race: thread may be cancelled twice, if first cancel is swallowed before second cancel raised. If cancel flag already raised, second is no-op. t.join() t.cancel() harmless race: if join runs first, cancel is no-op t.join() t.join() harmless race: Both joins will wait until the thread is done. If one join comes first, the thread is already done when the second tries to join, so no-op. t.detach() t.join() harmless race. Either thread is detached first, so join is no-op, or thread is joined first, so detach is no-op. t.detach() t.detach() harmless race. Thread is detached. second detach is no-op. t.detach() t.cancel() harmless race: if detach runs first, cancel is no-op In my code, this is done by ref counting the thread state. The thread itself has one reference, and the std::thread object has another. Each modifying operation takes a local ref to the shared state, modifies it, and then releases the ref. In the case of join and detach it also clears the ref in the std::thread object, but that's OK too, as the other threads that might do the modifying have their own refs, and the internal state is not pulled out from under them. Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL

On Mar 27, 2007, at 9:20 AM, Howard Hinnant wrote:

But now add Thread B into this mix which calls t.detach():

t Thread A Thread B ... t.detach()

<sigh> sorry about the wrapping. I'm going to buy a gun and shoot my email client. Anybody got directions to the nearest Walmart? ;-) -Howard

On Mar 27, 2007, at 9:45 AM, Peter Dimov wrote:

Howard Hinnant wrote:

[ detach vs join ]

...

This is a departure from pthread semantics that I am uncomfortable with. If I accidently get myself into this kind of race. I think I'd prefer some noise (an assert, or exception, or crash).

I'm not entirely sure what is the purpose of detach as it's unique to N2184, but if it's only used to indicate "do not cancel on ~thread, please", there is no need for it to actually detach anything (invalidate the pthread_t handle), just set a "don't cancel" flag. The race you describe is then harmless.

The purpose of detach is to model the semantics of pthread_detach. pthread_detach is described as: The pthread_detach() function is used to indicate to the implementation that storage for the thread thread can be reclaimed when that thread terminates. If thread has not terminated, pthread_detach() will not cause it to terminate. The effect of multiple pthread_detach() calls on the same target thread is unspecified. I am exploring tightening up those semantics up just a bit with Anthony's:

t.detach() t.detach() harmless race. Thread is detached. second detach is no-op.

(watch out for the inevitable, unintentional text wrapping) -Howard

"Peter Dimov" <pdimov@mmltd.net> writes:

In implementation terms:

class thread { pthread_t handle_; bool detached_;

public:

thread(): detached_( false ) {}

void detach() { atomic_store( &detached_, true ); }

// (try-/timed-) 'const' joins

~thread() { if( !detached_ ) cancel(); pthread_detach( handle_ ); } };

Is this not a conforming N2184 thread? If not, why?

As I understand it: No, because t.joinable() would still be true, and you could still cancel and join the thread. Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL

On Mar 27, 2007, at 10:21 AM, Peter Dimov wrote:

In implementation terms:

class thread { pthread_t handle_; bool detached_;

public:

thread(): detached_( false ) {}

void detach() { atomic_store( &detached_, true ); }

// (try-/timed-) 'const' joins

~thread() { if( !detached_ ) cancel(); pthread_detach( handle_ ); } };

Is this not a conforming N2184 thread? If not, why?

Thanks for the pseudo code. I was working on pseudo code to show as well but I'm an hour behind you. Below is *pseudo* code. I've ignored the necessary atomics among other details in an effort for clarity (hopefully I haven't ignored so much as to make it confusing). Comments attempt to illuminate our differences. I believe one of the big differences is : How soon after join() can the thread clean up resources? I.e. the model I've been working on has an implicit detach() inside of join(), for the purpose of cleaning up the resources immediately instead of waiting to ~thread() for resource cleanup. My apologies in advance for the inappropriate wrapping. I've tried to keep my line lengths very short... pthread_t handle_; thread_local_data* tl; detach() { // If has already been detached, that means tl may already be released // therefore not safe to go through twice. // But maybe ok to ignore subsequent detach? if (handle_ == 0) throw ?; tl->wait_on_owner = false; tl->cv2.notify_all(); handle_ = 0; } cancel() { // Canceling a non-existent thread seems harmless if (handle_ != 0) tl->cancel_pending = true; } join() { // Don't want to pretend to join with non-existent // (or detached) thread, make noise if (handle_ == 0) throw ?; while (!tl->done) tl->cv1.wait(); detach(); // point of contention with N2178, makes join non-const, // and disables multi-join. Without it, resources not // released until explicit detach() or ~thread() } ~thread() { if (handle_ != 0) { cancel(); // harmless if already canceled detach(); // we haven't detached or joined yet else handle_ is 0 } } start() { allocate_all_resources(); // tl != 0 tl->done = false; tl->wait_on_owner = true; tl->cancel_pending = false; try { f(); } catch (thread_canceled&) { } catch (...) { terminate(); } tl->done = true; tl->cv1.notify_all(); while (tl->wait_on_owner) tl->cv2.wait(); release_all_resources(); // tl = 0 } -Howard

On Mar 27, 2007, at 11:47 AM, Peter Dimov wrote:

...

join() { // Don't want to pretend to join with non-existent // (or detached) thread, make noise if (handle_ == 0) throw ?; while (!tl->done) tl->cv1.wait(); detach(); // point of contention with N2178, makes join non- const, // and disables multi-join. Without it, resources not // released until explicit detach() or ~thread() }

... we can still remove this detach() call from join() and the user is free to get the old semantics back by just calling detach() explicitly after join() (whereas the reverse is not true).

Ok, I'm still having pain from realloc doing too much. :-) Maybe this is a safe spot to tweak the pthreads semantics since ~thread() *must* catch any leaks and C doesn't have that option. I've tweaked the pseudo code as below and am continuing to look for problems with it. I believe this comes closer to Anthony's earlier post. And note the const qualifier I put on join!!! :-) (reserving the right to waffle back...) pthread_t handle_; thread_local_data* tl; detach() { // If has already been detached, that means tl may already be released // therefore not safe to go through twice. // But maybe ok to ignore subsequent detach? if (handle_ != 0) { tl->wait_on_owner = false; tl->cv2.notify_all(); handle_ = 0; } } cancel() { // Canceling a non-existent thread seems harmless if (handle_ != 0) tl->cancel_pending = true; } join() const { // Don't want to pretend to join with non-existent // (or detached) thread, make noise if (handle_ == 0) throw ?; while (!tl->done) tl->cv1.wait(); } thread::id get_id() const {return thread::id(handle_);} ~thread() { cancel(); // harmless if already canceled detach(); // harmless if already detached } start() { allocate_all_resources(); // tl != 0 tl->done = false; tl->wait_on_owner = true; tl->cancel_pending = false; try { f(); } catch (thread_canceled&) { } catch (...) { terminate(); } tl->done = true; tl->cv1.notify_all(); while (tl->wait_on_owner) tl->cv2.wait(); release_all_resources(); // tl = 0 } -Howard

On Mar 27, 2007, at 12:00 PM, Peter Dimov wrote:

Howard Hinnant wrote:

...

I believe one of the big differences is : How soon after join() can the thread clean up resources? I.e. the model I've been working on has an implicit detach() inside of join(), for the purpose of cleaning up the resources immediately instead of waiting to ~thread() for resource cleanup.

One question that comes to mind here - which I've been unable to effectively communicate in my other post - is:

What use case exists for not destroying the thread object after it's been detached (or joined+detached) under the current N2184 semantics?

Perhaps a deamon spawner? Perhaps one that has other communication lines with the detached thread? vector<function<void()>> queue; std::thread launch; while (!queue.empty()) { launch = std::thread(queue.back()); queue.pop_back(); register(launch.get_id()); launch.detach(); } The start function in the queue may unregsiter(id) when the thread ends. We could've put launch inside the loop and thus done without the explicit detach. But I'm lacking motivation to disallow the above pattern. Maybe the detach is conditional and the thread needs scope external to the loop: vector<function<void()>> queue; std::thread launch; while (!queue.empty()) { launch = std::thread(queue.back()); queue.pop_back(); register(launch.get_id()); if (something) break; launch.detach(); } // work with undetached launch thread here -Howard

On Mar 27, 2007, at 12:35 PM, Peter Dimov wrote:

Once there, we may (or may not) opt to actually make the thread object useful after detach because this pattern currently has no other uses. This leads us directly to the 'detach suppresses cancel but has no other effects' model, where you can still query the id of the detached thread, among other things, if you hold onto the object for a while.

What would join() do after detach() in this model? -Howard

On Mar 27, 2007, at 12:52 PM, Peter Dimov wrote:

Howard Hinnant wrote:

...

On Mar 27, 2007, at 12:35 PM, Peter Dimov wrote:

...

Once there, we may (or may not) opt to actually make the thread

...

object useful after detach because this pattern currently has no

...

other uses. This

...

leads us directly to the 'detach suppresses cancel but has no other

...

effects'

...

model, where you can still query the id of the detached thread,

...

among other things, if you hold onto the object for a while.

What would join() do after detach() in this model?

What it always does. Wait for the thread to end, then return.

Ok, thanks I think understand now. New use case: vector<thread> v; ... fill with threads ... for (auto i = v.begin(), e = v.end(); i != e; ++i) if (i_feel_like_it) i->detach(); ... // need new thread now, go look for a detached one and use it auto i = v.begin(); for (auto e = v.end(); i != e; ++i) if (i->is_detached()) // N2184 spells this i->get_id() == thread::id() break; if (i != v.end()) *i = std::thread(f); // recycle detached thread else v.push_back(std::thread(f)); In this use case, you're still recycling threads, as in my previous use case. But here there is an opportunity for many threads to not get recycled for a long time, and perhaps forever. <sigh> And this use case is making me nervous about join not implicitly detaching... -Howard

On Mar 27, 2007, at 1:44 PM, Peter Dimov wrote:

...

New use case:

vector<thread> v; ... fill with threads ... for (auto i = v.begin(), e = v.end(); i != e; ++i) if (i_feel_like_it) i->detach(); ... // need new thread now, go look for a detached one and use it auto i = v.begin(); for (auto e = v.end(); i != e; ++i) if (i->is_detached()) // N2184 spells this i->get_id() == thread::id() break; if (i != v.end()) *i = std::thread(f); // recycle detached thread else v.push_back(std::thread(f));

I was thinking of something along these lines, but it still seems contrived to me. There is no reason to reuse an std::thread _object_ because you gain no efficiency from that (as opposed to reusing a thread).

What if the above use case uses vector<MyClass> where MyClass contains a lot of expensive resources (or other data that needs to be kept around for one reason or another) *plus* a std::thread? if (i != v.end()) i->install_new_thread(std::thread(f)); I.e. the fact that the thread is recycled instead of erased and reconstructed here is due to reasons which have nothing to do with with std::thread itself. -Howard

On Mar 27, 2007, at 2:40 PM, Peter Dimov wrote:

Howard Hinnant wrote:

...

What if the above use case uses vector<MyClass> where MyClass contains a lot of expensive resources (or other data that needs to be kept around for one reason or another) *plus* a std::thread?

Well...

void old_detach() { new_detach(); *this = std::thread(); }

<shrug> I don't see sufficient motivation for new_detach(). I'm still trying to model pthreads semantics. I feel underqualified to invent and standardize new semantics for threading. Having two flavors of join (that I just suggested) is pretty radical for old staid conservative me. I'm not sure how much more excitement I can take! :-) -Howard

Howard Hinnant wrote:

I'm beginning to think:

multijoin() const { // Don't want to pretend to join with non-existent // (or detached) thread, make noise if (handle_ == 0) throw ?; while (!tl->done) tl->cv1.wait(); }

join() { multijoin(); detach(); }

I think it should be more like: multijoin() const // try, timed same { if( handle_ != 0 ) // the NULL thread is considered terminated while (!tl->done) tl->cv1.wait(); } join() { if( handle_ == 0 ) assert, throw acc. to taste multijoin(); detach(); *this = std::thread(); } I still think that the user would be free to build destructive join/detach upon the provided interface and that we don't really have to provide them.

I'm coming full circle... As soon as I start allowing multiple threads to access std::thread at once, I need to keep a mutex around for the lifetime of the std::thread, whether or not it is already joined or even detached: class thread { pthread_t handle_; // 0 if detached thread_local_data* tl_ // 0 if detached { mutex mut; // synchronize between parent and child threads condition cv; bool done; bool wait_on_owner; bool cancel_pending; bool cancel_enabled; } mutex another_mut; // synchronize with multi-parent access, always here }; I can never get rid of another_mut, even after thread detach, even after: t = std::thread(); I believe I'll even have to lock t.another_mut during the above move assign, making move assign unacceptably slow. This is the wrong class to allow multiple threads to touch at once. Even if I load it with another_mut and bring move assignment to a crawl, simultaneous access to join() and detach() is still a race. Consider: vector<thread> v(100); v.reserve(101); v.insert(v.begin(), std::thread(one_more_thread)); To accomplish that insert we just had to lock/unlock 100 mutexes. I actually did this with std::string many years ago, and I *still* have the bruises from my customers! Either that or we go with detach() being a no-op as Peter suggests and we keep the thread state around forever. And then we still just did 100 atomic operations to get that insert.

I think that all operations on std::thread should be safe to execute concurrently, and that they should have defined functionality in all cases, which should probably be a no-op.

Sorry, I just do not believe this is a good idea. It completely destroys vector<thread> performance for what is at best a corner use case and can be handled with layered-on classes or otherwise applied external synchronization if desired. The whole beauty of the sole- ownership model is how light and agile it is to manipulate. -Howard

Howard Hinnant wrote:

I'm coming full circle...

As soon as I start allowing multiple threads to access std::thread at once, I need to keep a mutex around for the lifetime of the std::thread, whether or not it is already joined or even detached:

[...] Here's how one possible N2184 thread looks to me: class thread { private: n2178::handle handle_; thread( thread const & ); thread& operator=( thread const & ); public: thread() {} template<class F> explicit thread( F f ): handle_( n2178::create( f ) ) {} thread( thread && rhs ): handle_( std::move( rhs.handle_ ) ) { } thread& operator=( thread && rhs ) { handle_ = std::move( rhs.handle_ ); return *this; } void swap( thread& rhs ) { handle_.swap( rhs.handle_ ); } void join() const { n2178::join( handle_ ); } bool try_join() const { return n2178::try_join( handle_ ) != EBUSY; } bool timed_join( timespec const & abstime ) const { return n2178::timed_join( handle_, abstime ) != ETIMEDOUT; } void cancel() // const acc. to taste { n2178::cancel( handle_ ); } }; For cancel on destroy, add: bool cancel_on_destroy_; // initially true ~thread() { if( cancel_on_destroy_ ) cancel(); } void detach() { atomic_store( &cancel_on_destroy_, true ); } There appears to be no need for a mutex in this prototype, unless I've missed something. Even if the actual implementation uses no n2178::handle, the above sketch might still be useful as an illustration of the semantics and thread safety that I have in mind. The omission of joinable() and cancel_requested() is intentional, the omission of the rest of the members is for brevity, although hardware_concurrency would probably be better off as a free function. To get back to an earlier statement of yours:

This just leaves me with the feeling of implementing unique_ptr in terms of shared_ptr, at least on a pthreads platform.

Under my implementation at least, it's more like implementing intrusive_ptr<thread_data> where the refcount can be at most 2 (std::thread and the thread itself both holding a ref) in terms of intrusive_ptr<thread_handle> with no refcount limit.

On Mar 27, 2007, at 9:49 PM, Peter Dimov wrote:

There appears to be no need for a mutex in this prototype, unless I've missed something.

Even if the actual implementation uses no n2178::handle, the above sketch might still be useful as an illustration of the semantics and thread safety that I have in mind. The omission of joinable() and cancel_requested() is intentional, the omission of the rest of the members is for brevity, although hardware_concurrency would probably be better off as a free function.

Thanks for the code. Correct me if I'm wrong, but this code keeps state around after join and detach, and so might be problematic in the vector<thread> use case shown earlier (where potentially many threads can sit idle waiting to be recycled). Indeed, this implementation appears to simply wrap const semantics in a non-const handle. The "mixed thread safe" semantics I find interesting and a little worrisome. Having some functionality thread safe and some not could be seen as confusing. Looking again at the daemon spawner example: std::thread launch; ... vector<function<void()>> queue; while (!queue.empty()) { launch = std::thread(queue.back()); queue.pop_back(); register(launch.get_id()); if (something) break; launch.detach(); } // work with undetached launch thread here Except now "launch" is a global which other threads can access. Where is it safe to race against this code? A thread-safe detach() appears to have bought us little here since one would have to race against the thread-unsafe move assign. And the thread-safe detach() didn't come for free, we had to promise not to get rid of resources in the vector<thread> use case. Now we've got vector<thread> performance (good), but to do so we've partially compromised mulit-parent simultaneous access (bad), and we're potentially keeping resources around much longer than wanted (bad). Your suggestion is a compromise between the two designs (sole vs shared ownership), but I'm not sure it is a good compromise. I know we've got use cases which cry out for multi-parent access (thread pool is likely to be one). But by far the most common use cases appear to not need multi-parent access (http://www.boost.org/doc/html/threads/rationale.html#threads.rationale.non-c... ). And thread pool (and other shared-ownership use cases) can be layered onto the sole-ownership std::thread without compromise. -Howard

On Mar 28, 2007, at 1:26 PM, Peter Dimov wrote:

In particular, we want, given a std::thread X, thread A (or threads A1...An) to be able to join it and thread B to be able to cancel it, unblocking A.

This can't be achieved by a simple user-level mutex protection because it will deadlock. Thread A will lock the mutex protecting X, join and block with the mutex locked. Thread B will attempt to acquire the mutex in order to invoke cancel and also block. Unless the target thread exits by itself, A and B will wait forever.

That's a good use case, thanks. Let's look at this with a diagram (ascii art warning): A B \ / join cancel \ / \/ X This is actually one of the diagrams I worry about. :-) In your use case B affecting A's worker thread X is a desired effect. However I worry that X gets canceled *accidently* out from under A when B assumes that X is *its* worker thread and doesn't know about A. In the sole-ownership model, one redesigns the logic so that thread ownership is a tree instead of a lattice. In this case, a very simple tree: B | cancel | A | join | X Now X is a *private* implementation detail of A. And since join is a cancellation point, when B cancels A, A reacts (unless cancellations are disabled or the join has already returned). Subsequently X.~thread() runs and cancels X. So what if you only want to cancel X and you want A to continue on? Ok, that could happen. But normally A called X for a reason, and if X has an abnormal termination, A is going to have to deal with it, or result in abnormal termination itself. But if A really wants to deal with this situation there are a few options: 1. Use shared_future instead of thread. I.e. this would set up the first (multi-owner) diagram above. And now when B cancels X, the join doesn't just silently return as if everything is ok, it throws a child_canceled exception. Now you know A has to deal with it! :-) 2. Use thread and set up the second (single-owner) diagram: void A() { try { X.join(); } catch (std::thread_canceled&) { B canceled me, recover } continue } 3. Build your own multi-owner access to X (this would be my least recommended choice). But it would involve putting X on the heap along with a mutex and condition, then building "join" out of the mutex and condition instead of using thread::join (which is exactly what shared_future will do too). -Howard