Sohail Somani wrote:

Aside: Why do you want to bring pthreads into the mix? I know the two threading APIS are Windows and pthreads but still... Couldn't you define a new API that would be a thin wrapper for pthreads?

Why? Users of the POSIX API wouldn't want to adjust their existing code for it. (As a side node, there seems to be significant resistance against adding shallow C++ wrappers around POSIX functions in the POSIX C++ Binding Study Group List) Users of non-POSIX APIs would have to adjust their code, anyway. Transition to "proper C++ code" (for some sense of that phrase) would require the C++ <thread> API, anyway. New C++ code could would likely use the C++ <thread> API, anyway. Regards, m Send instant messages to your online friends http://au.messenger.yahoo.com

Sohail Somani wrote:

...

-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Martin Wille

...

Why?

Users of the POSIX API wouldn't want to adjust their existing code for it. (As a side node, there seems to be significant resistance against adding shallow C++ wrappers around POSIX functions in the POSIX C++ Binding Study Group List)

Interesting. Why?

Yeah, I'd call it depressing. Jeff

On Mar 19, 2007, at 5:29 PM, Sohail Somani wrote:

I was pretty much hoping for boost threads + cancellation.

Fwiw, this is the approach of N2184: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2184.html The very biggest change between boost::thread and N2184 is the addition of cancellation. And, imho, this is a huge change. There are several more minor changes: * The N2184::thread is non-copyable like boost::thread, but movable (taking advantage of new C++ language and std::lib features). Movable N2184::thread maintains the one-to-one mapping between the std::thread and the OS thread which boost has (sole ownership semantics). It just adds the ability to move the thread between scopes (such as return from factory functions). I recently drew the analogy this way: boost::thread is like a scoped_ptr to the OS thread. N2184::thread is like an auto_ptr to the OS thread (the proposed unique_ptr would be a more accurate analogy). * N2184::thread separates out the concept of thread identity and thread handle. In boost these are both just boost::thread. The thread::id is copyable. The only thing you can do with it is equality compare it. * N2184::thread moves some of the static boost thread members (yield, sleep) to a namespace called this_thread. So the syntax for calling them is std::this_thread::yield() as opposed to boost::thread::yield(). * N2184::threads adds a "back door escape hatch" to the OS thread called native_handle(). You can get that from the std::thread and do non-portable things with it at the OS level (setting thread priority is the poster child). N2184 does not mention the rest of boost::threads (mutex, locks, condition, thread_once, TLS, etc.). All of that stuff is pretty much assumed modulo details - TLS will be in the language, not the lib. N2184 just concentrates on the thread class (which has been much more controversial). -Howard

Howard Hinnant <hinnant@twcny.rr.com> writes:

On Mar 19, 2007, at 5:29 PM, Sohail Somani wrote:

...

I was pretty much hoping for boost threads + cancellation.

Fwiw, this is the approach of N2184:

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2184.html

For which I have now uploaded my Windows (MSVC) implementation to my website: http://www.justsoftwaresolutions.co.uk/threading/index.html I had to use a library-based workaround for "move" in place of rvalue references, and I substituted thread::create rather than the templated constructor, as I prefer that: jss::thread t=jss::thread::create(f); jss::thread t2=jss::move(t); Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL

"Peter Dimov" <pdimov@mmltd.net> writes:

Anthony Williams wrote:

...

Howard Hinnant <hinnant@twcny.rr.com> writes:

...

On Mar 19, 2007, at 5:29 PM, Sohail Somani wrote:

...

I was pretty much hoping for boost threads + cancellation.

Fwiw, this is the approach of N2184:

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2184.html

For which I have now uploaded my Windows (MSVC) implementation to my website:

http://www.justsoftwaresolutions.co.uk/threading/index.html

I take this to mean that you aren't interested in adopting any parts on N2178 as part of Boost.Threads. :-/

No, I just wanted to get my implementation out there for user feedback. My implementation isn't quite N2184. I've got a templated thread::create function (like N2178) rather than a templated constructor, for starters. On rereading N2178, I just spotted that your call_once has the flag as the first parameter, and allows additional arguments to be passed to the once_function. I like that, and might change my library to match. My focus is primarily on the C++ interface. I am concerned about the cost for implementors of requiring pthreads as the C api, but am not opposed in principle. Certainly, I think a boost.pthread-win32 library would be of benefit. As for copyable handles, my implementation is deliberately not copyable, and it would be trivial to convert it to be a copyable handle rather than just movable. If the consensus of opinion of the committee is for copyable handles, I'm happy to go with that for boost. I find it strange to template the condition var on the mutex, and then (separately) template the wait function on the lock type. I've opted for templating the whole condvar on the lock type, as it seems more coherent to me. Thread and mutex attributes are not currently something I've put much thought into. Though I've made some comments about priority to the committee, I tend to use default attributes in most cases. Are there any crucial differences between N2178 and N2184 that I've missed? Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL

On Mar 20, 2007, at 12:31 PM, Peter Dimov wrote:

...

I find it strange to template the condition var on the mutex, and then (separately) template the wait function on the lock type. I've opted for templating the whole condvar on the lock type, as it seems more coherent to me.

I actually "stole" this idea from Howard IIRC. His early version was templated on the lock. A later iteration was templated on the mutex. Both work for me.

Actually I never templated the wait (well, except on Predicate). That is the boost design. Here's my only paper on the subject: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n2094.html#cv I only just today (as Anthony pointed it out) realized you had the two templates and was contemplating that twist... ;-) -Howard

Anthony Williams wrote:

I'd be willing to help with a Boost.Pthread layer. I would be concerned about layering Boost.Thread on Boost.Pthread, since that seems to me to be too many abstractions. However, if we can make Boost.Thread and Boost.Pthread share stuff under the covers, that would be neat.

Since the Boost.Thread (and N2184) underlying model is essentially equivalent to pthreads, one easy way to make Boost.Thread and Boost.Pthread share stuff is to make Boost.Thread use the pthread_* functions. :-) Perhaps you have in mind an intermediate common API? What would we gain from it?

Anthony Williams wrote:

For a non-pthreads platform, it might make sense to implement pthreads in terms of the C++ interface, which is implementes in terms of the native API, rather than implement the C++ interface in terms of pthreads, which is then implemented in terms of the native API.

Yes, this is possible, and there are no technical reasons to avoid this implementation approach. There are, how should I put that, ideological reasons to not target non-pthread platforms directly, though. At some point we need to draw a line and say: this is the thread abstraction we're coding against, and could vendors please get along with the program? This stance is obviously unsustainable if there are strong technical reasons against it. But I think that there aren't.

Lets take call_once, for example. PThreads only allows a real function with no parameters returning void as the once function. This is relatively easy to implement on top of a C++-style call_once that allows arbitrary function objects with parameters, whereas it is much harder to do the reverse.

There is a note in N2178 that shows how to implement call_once on top of pthread_once2_np. ;-)

Peter Dimov wrote:

Anthony Williams wrote:

...

For a non-pthreads platform, it might make sense to implement pthreads in terms of the C++ interface, which is implementes in terms of the native API, rather than implement the C++ interface in terms of pthreads, which is then implemented in terms of the native API.

Yes, this is possible, and there are no technical reasons to avoid this implementation approach. There are, how should I put that, ideological reasons to not target non-pthread platforms directly, though.

Wow, this caught my eye when lurking around. Are we promoting Windows-bashing for its own sake here ;-)

At some point we need to draw a line and say: this is the thread abstraction we're coding against, and could vendors please get along with the program?

So, what about sockets, asynchronous I/O etc? <with some twinkle in eye> For asynch I/O I'd promote to code against the NT abstraction, where the OS was built and designed from ground up with asynchronous I/O support in the kernel. Or, even better, the superior VMS I/O model. And then, vendors should implement SYS$QIO, SYS$WAIT, SYS$SYNCH, et al ... </with some twinkle in eye>

This stance is obviously unsustainable if there are strong technical reasons against it. But I think that there aren't.

I'd be surprised if there were no performance considerations at all. I realize that C programmers also need portable threading support, but requiring the C++ implementation to use a specific underlying API is not the way to go, IMHO.

...

Lets take call_once, for example. PThreads only allows a real function with no parameters returning void as the once function. This is relatively easy to implement on top of a C++-style call_once that allows arbitrary function objects with parameters, whereas it is much harder to do the reverse.

There is a note in N2178 that shows how to implement call_once on top of pthread_once2_np. ;-)

Almost everything is possible with lots of effort. I don't know about this particular case. / Johan

Alexander Terekhov <terekhov@web.de> writes:

Johan Nilsson wrote:

...

Peter Dimov wrote:

...

Anthony Williams wrote:

...

For a non-pthreads platform, it might make sense to implement pthreads in terms of the C++ interface, which is implementes in terms of the native API, rather than implement the C++ interface in terms of pthreads, which is then implemented in terms of the native API.

Yes, this is possible, and there are no technical reasons to avoid this implementation approach. There are, how should I put that, ideological reasons to not target non-pthread platforms directly, though.

Wow, this caught my eye when lurking around. Are we promoting Windows-bashing for its own sake here ;-)

http://www.infoworld.com/articles/pl/xml/02/07/29/020729plmsu.xml (REVIEWS: A nod toward Unix... PROS: + Bypasses Win32 API for performance + Superior performance to Win32 API + Price is unbeatable)

All a bit vague. It might well give better performance to code UNIX APIs directly in terms of the Windows native API rather than the Win32 API. It makes sense to me --- the fewer abstraction layers between user code and the underlying OS, the better. That's actually precisely my argument above. Note also that V3.0 is being reviewed, which doesn't include pthreads.

http://www.microsoft.com/technet/interopmigration/unix/sfu/pthreads0.mspx

Demonstrating that it is possible to code pthreads in terms of the native Windows API, especially if you're writing a whole UNIX-interop layer, including a full POSIX C library. Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL

Johan Nilsson wrote:

Anthony Williams wrote:

...

...

http://www.microsoft.com/technet/interopmigration/unix/sfu/pthreads0.mspx

Demonstrating that it is possible to code pthreads in terms of the native Windows API, especially if you're writing a whole UNIX-interop layer, including a full POSIX C library.

[... nodding my head in violent agreement ...]

http://darwinsource.opendarwin.org/Current/Libc-391.2.5/pthreads/pthread.c Demonstrating that it's possible to code pthreads in terms of the native Mach API. And NPTL demonstrates that it's possible to code pthreads in terms of the native Linux kernel API. This is the whole point of an abstraction. The only reason we aren't using it on Windows is because Microsoft doesn't supply it. The performance arguments are a post-hoc rationalization.

Alexander Terekhov <terekhov@web.de> writes:

Peter Dimov wrote:

...

This is the whole point of an abstraction. The only reason we aren't using it on Windows is because Microsoft doesn't supply it.

Uhmm.

http://www.microsoft.com/presspass/press/2004/Jan04/01-15ServicesforUNIX2004...

<quote>

Microsoft's Newest Version of Windows Services for UNIX Is Now Available Free of Charge

Latest Version of Microsoft's Award-Winning Windows UNIX Interoperability Product Offers Customers Enhanced Performance and New Functionality That Extends Investments In UNIX-Based Systems; SFU 3.5 Is Finalist in LinuxWorld Product Excellence Awards

</quote>

As I understand it, SFU is a complete subsystem. You either write your app for SFU, or you write it for Win32. Though it does mean you can run pthreads-based apps on Windows, it's not quite the same. Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL

Johan Nilsson wrote: [...]

it still is): http://tinyurl.com/25nbyq

Thanks for the link. I see a reference to MKS Toolkit. Perhaps Microsoft should simply acquire http://www.mkssoftware.com/ just like Softway Systems (maker of Interix). regards, alexander.

Anthony Williams wrote:

However, Microsoft do not supply such an interface, so someone has to write one. Writing one might be a good idea, but using it to implement the C++ API, when the C++ API could be better written using the win32 API directly seems a bad plan to me, especially if you could use your C++ API to implement pthreads.

It depends on what your plan is supposed to achieve. If you want to gradually transition from where we are now to a point where Microsoft does supply a pthread layer, it's not a bad plan at all. If you want something else, it might or might not be, depending on what exactly do you want. Anyway, I see that you are taking this as an argument, so I'll leave you to your opinion.

Yuval Ronen wrote:

Peter Dimov wrote:

...

Anthony Williams wrote:

...

However, Microsoft do not supply such an interface, so someone has to write one. Writing one might be a good idea, but using it to implement the C++ API, when the C++ API could be better written using the win32 API directly seems a bad plan to me, especially if you could use your C++ API to implement pthreads.

It depends on what your plan is supposed to achieve. If you want to gradually transition from where we are now to a point where Microsoft does supply a pthread layer, it's not a bad plan at all. If you want something else, it might or might not be, depending on what exactly do you want. Anyway, I see that you are taking this as an argument, so I'll leave you to your opinion.

Excuse me for being a bit daft here, but why is our plan to make Microsoft supply a Windows pthread layer? The reason we all want that today, is because there *is no* C/C++ threading standard. Once there is such a standard, we wouldn't care anymore about the underlying OS API. That's the whole purpose of this (or any) standardization effort, isn't it?

That was the exact purpose of the pthreads standard. Now people on POSIX-compliant OSes don't care about the underlying OS APIs (which in general have little to do with pthreads) because the vendor ships a pthread layer. Add to that the fact that all threading proposals to date agree that the pthreads threading abstraction is the way to go; they all mandate its set of primitives and semantics. So we do want Microsoft to offer the pthread threading model to C++ programmers, even if some of us don't realize it. The obvious question now is: do we want Microsoft to offer this threading model only to C++ programmers and not to C programmers? And what do we stand to gain from that? How is it in the best interest of the C++ community to demand that C code remains nonportable?

Yuval Ronen wrote:

Peter Dimov wrote:

...

That was the exact purpose of the pthreads standard. Now people on POSIX-compliant OSes don't care about the underlying OS APIs (which in general have little to do with pthreads) because the vendor ships a pthread layer.

Maybe it was the purpose of the POSIX standard, but it was never accepted by the C or C++ standard committees.

If the committees only accepted things that were already accepted, they would never accept anything, right? ;-)

Of course we want Microsoft to offer this threading model to C programmers, but it's the job of the C standard committee to define the exact syntax. This standard C syntax might as well be the exact POSIX syntax, no problem by me. But it doesn't really matter one way or the other, as long as the C syntax accepted is a good one.

Doesn't this answer your original question, which was "why do we want Microsoft to ship a pthread layer"? They would have to, either way. The only question left is whether we want two incompatible C APIs to the same underlying model, only one of which works on Windows, or just one. It seems to me that it is in our best interest to want the latter.

Yuval Ronen wrote:

We would have only one C API, and it will work on Windows. The allegedly second C API is hidden and non of us should care about it.

We must be talking about different things here, since I don't see how anyone could call pthreads alleged or hidden with a straight face.

Yuval Ronen wrote:

So I'll try to re-phrase. The point I tried to make is that the pthread philosophy would be accepted by the C/C++ standard, but the pthread exact syntax doesn't have to. The C/C++ standard can adopt a different syntax.

Sure. We seem to agree that we want a C threading standard and a C++ threading standard, both based on the pthreads model, compatible with each other, supported by Microsoft and the rest of the world. So what is the best way to get there from where we are now? Do nothing and hope that this will eventually happen without our involvement... somehow? Or propose and back/support an integrated C/C++ API?

Peter Dimov wrote:

...

Yuval Ronen wrote:

...

So I'll try to re-phrase. The point I tried to make is that the pthread philosophy would be accepted by the C/C++ standard, but the pthread exact syntax doesn't have to. The C/C++ standard can adopt a different syntax.

Sure. We seem to agree that we want a C threading standard and a C++ threading standard, both based on the pthreads model, compatible with each other, supported by Microsoft and the rest of the world.

I guess the key word we are disagreeing about is "compatible" (which is what Emil's talking about). I assume you want threads created with the C++ API to be able to be manipulated by the C API. I don't see any major advantage in that.

Because C++ is compatible with C, one can come up with infinite number of examples of useful inteoperability between C libraries and C++ programs, as well as C++ libraries and C programs. I don't see how you could not see the advantage of that.

There's no precedence to such a thing. Do we want files opened with std::fstream to be read using fgetc()? Was this ever considered a factor when designing the C++ iostreams? Is there anyone missing this feature now? I've never heard anyone answering "yes" to any of those question (as much as that counts).

On the other hand, aren't most iostreams implementations based on the C standard library anyway? I can't answer the question whether it would make more sense if this was required by the C++ standard, however I'm pretty sure there are many benefits if such compatibility was official (which doesn't necessarily mean that there are no drawbacks.)

It's not that there's anything inherently wrong with such C<->C++ interoperability, as long as it doesn't cripple the C++ API.

My point exactly.

If it prevents creating the "ideal" C++ interface (which, IMHO, is something along the lines of Howard's non-copyable thread class), then I'd consider it a disadvantage.

If someone can show that to be the case, I'd agree. (Off-topic to this argument, but in response to your opinion, I am not sure I see the benefits of non-copyable thread handles. I am also concerned about the fact that in N2184 there is no thread handle for the current thread, which means that code that manipulates the current thread needs to be special case. To me, this looks a lot like C++ without a this pointer. Good design is all about selectively limiting unnecessary flexibility, but I am not sure that having copyable thread handles is unnecessary.) Emil Dotchevski

Emil Dotchevski wrote:

...

Peter Dimov wrote:

...

Yuval Ronen wrote:

...

So I'll try to re-phrase. The point I tried to make is that the pthread philosophy would be accepted by the C/C++ standard, but the pthread exact syntax doesn't have to. The C/C++ standard can adopt a different syntax. Sure. We seem to agree that we want a C threading standard and a C++ threading standard, both based on the pthreads model, compatible with each other, supported by Microsoft and the rest of the world. I guess the key word we are disagreeing about is "compatible" (which is what Emil's talking about). I assume you want threads created with the C++ API to be able to be manipulated by the C API. I don't see any major advantage in that.

Because C++ is compatible with C, one can come up with infinite number of examples of useful inteoperability between C libraries and C++ programs, as well as C++ libraries and C programs. I don't see how you could not see the advantage of that.

Have you ever needed such interoperability in practice? I haven't. That's not to say it's nonsense. I just think it's rare enough to be negligible.

...

There's no precedence to such a thing. Do we want files opened with std::fstream to be read using fgetc()? Was this ever considered a factor when designing the C++ iostreams? Is there anyone missing this feature now? I've never heard anyone answering "yes" to any of those question (as much as that counts).

On the other hand, aren't most iostreams implementations based on the C standard library anyway? I can't answer the question whether it would make more sense if this was required by the C++ standard, however I'm pretty sure there are many benefits if such compatibility was official (which doesn't necessarily mean that there are no drawbacks.)

Again, saying "I'm pretty sure there are many benefits if..." is hypothetical. I haven't met such benefits myself in real life.

...

It's not that there's anything inherently wrong with such C<->C++ interoperability, as long as it doesn't cripple the C++ API.

My point exactly.

So we agree ;-)

...

If it prevents creating the "ideal" C++ interface (which, IMHO, is something along the lines of Howard's non-copyable thread class), then I'd consider it a disadvantage.

If someone can show that to be the case, I'd agree.

I'd leave that for Howard (whose hands finally slipped out ;-) ) to answer. He'll be much better at this than me...

Emil Dotchevski wrote:

...

...

...

Of course we want Microsoft to offer this threading model to C programmers, but it's the job of the C standard committee to define the exact syntax. This standard C syntax might as well be the exact POSIX syntax, no problem by me. But it doesn't really matter one way or the other, as long as the C syntax accepted is a good one. Doesn't this answer your original question, which was "why do we want Microsoft to ship a pthread layer"? They would have to, either way. Microsoft might as well have to implement a pthread layer, but that's non of our business, as long as they comply with the C/C++ standard.

Except that it is of benefit to the C++ community if the C and C++ layer are compatible beyond just syntax. N2178 defines interoperability semantics that go beyond standard pthread.

...

...

The only question left is whether we want two incompatible C APIs to the same underlying model, only one of which works on Windows, or just one. It seems to me that it is in our best interest to want the latter. We would have only one C API, and it will work on Windows. The allegedly second C API is hidden and non of us should care about it.

I would agree if we were talking about 2 substantially different APIs. Sure, if you have pthreads and something that's somewhat different, why not. The point is that pthread, with a few extensions, is exactly what we need from this hidden API.

Maybe so, maybe not. It's simply irrelevant. A hidden API is, well, hidden.

But let's turn things over: why not use pthread instead of creating another API? I think it is clear that using pthreads has a few advantages. What is the drawback?

That's it's not good C++. We are designing a C++ interface here, and we want to take advantage of all the nice things C++ has to offer, as Sohail pointed out. N2184 is better at using all those nice things, IMO. I don't want to damage that for this compatibility. I just don't think it worth it.

Peter Dimov wrote:

Yuval Ronen wrote:

...

That's it's not good C++. We are designing a C++ interface here, and we want to take advantage of all the nice things C++ has to offer, as Sohail pointed out. N2184 is better at using all those nice things, IMO.

N2184 is closer to pthreads than the C++ portion of N2178 is, FWIW. But don't let that stop you.

I won't :-) Being close to pthreads model is a good thing. Being close to pthreads in other aspect depends on the aspect, and can be neither a good thing, not a bad thing. I don't know exactly what you mean. In general, I can say that apart from the good stuff in pthreads, all the rest is simply a non-issue - I'm neither promoting being close nor far away from pthreads. I'm promoting making the best C++ design possible.

Peter Dimov wrote: [...]

So it's not as simple as you think. Oh, N2178 has a pthread C API, therefore it must be evil.

You should really add something along the lines of wording regarding <iohw.h> and <hardware> (C++ performance TR stuff). Quoting "Appendix C" (http://www.research.att.com/~bs/performanceTR.pdf): <quote> The implementation of the basic <iohw.h> ... interface on top of the <hardware> interface is mainly straightforward. This section provides an example of how such an implementation can be achieved. The purpose of using C++ at the lowest level is to take advantage of compile-time evaluation of template code to yield object code specialized for specific hardware. For a good implementation of the basic templates that perform the lowest-level hardware access operations, this approach typically leads to code that maps directly to machine instructions as efficient as code produced by an expert programmer. Additionally, the type safety of the C++ interface minimizes debugging and errors. </quote> to N2178. ;-) regards, alexander. -- http://groups.google.com/group/comp.std.c++/msg/158390632f4667fb

Emil Dotchevski wrote:

...

...

But let's turn things over: why not use pthread instead of creating another API? I think it is clear that using pthreads has a few advantages. What is the drawback? That's it's not good C++. We are designing a C++ interface here, and we want to take advantage of all the nice things C++ has to offer, as Sohail pointed out.

If you are refering to design differences between N2178 and N2184, I don't think they have anything to do with pthreads. These differences express the different opinions of the authors.

I think these differences do have to do with pthreads. Very much to do with pthreads.

...

N2184 is better at using all those nice things, IMO. I don't want to damage that for this compatibility. I just don't think it worth it.

How does using pthreads as a lower level implementation for N2184 "damage" it?

I didn't mean that using pthreads as a lower level implementation for N2184 would damage anything. I'm sorry if it sounded like it. I meant that standardizing pthreads, as the motivation behind N2178, will damage it. Just standardizing pthreads as a lower level for N2184 won't damage anything, it's just not needed.

Peter Dimov wrote:

Johan Nilsson wrote:

...

Peter Dimov wrote:

...

Anthony Williams wrote:

...

For a non-pthreads platform, it might make sense to implement pthreads in terms of the C++ interface, which is implementes in terms of the native API, rather than implement the C++ interface in terms of pthreads, which is then implemented in terms of the native API.

Yes, this is possible, and there are no technical reasons to avoid this implementation approach. There are, how should I put that, ideological reasons to not target non-pthread platforms directly, though.

Wow, this caught my eye when lurking around. Are we promoting Windows-bashing for its own sake here ;-)

No, you misunderstood.

We are promoting ISO/IEC 9945 as a standard threading abstraction.

IIUC that is "The Single UNIX Specification", I assume you mean the "pthread.h" part of it?

We are (hypothetically) providing an implementation for Windows as a service to the users/developers and as a proof of viability. We are encouraging Microsoft to supply its own implementation because they can do it better than us with access to the kernel.

Even if they did, that'd have to be implemented as an update or add-on for current platforms. Requiring post-Vista versions of Windows for having threads available under C++ doesn't seem right to me. Also, what about unsupported OS's such as NT4? Microsoft will never release any officially supported updates to such platforms. Basically, what I'm saying is that any Win32 C/C++ thread implementation still has to provide its own threading support as it can't rely on pthreads being available. Or, sorry, was that what you actually meant, that the standard C++ library would "have to" include kernel modules for best performance under Windows? Am I still misunderstanding? I have a nagging feeling that I do ...

...

I'd be surprised if there were no performance considerations at all.

One would expect a C++ programmer to be more careful with that argument, given his constant exposure to C programmers who really would be surprised if using C++ doesn't make their code slower. ;-)

Touché. But, still - without any firm evidence I believe that "C++ -> pthreads layer -> Win32 threads" is slower than "C++ -> Win32 threads", which is what I based my assumption upon.

That aside, if there is a legitimate design-imposed source of inefficiency in N2178, I'll be glad to know about it so that it can be fixed.

I never said it was. To be honest though, I've only glanced at the proposal, so I'll give it another shot. / Johan

Peter Dimov wrote:

Johan Nilsson wrote:

...

Touché. But, still - without any firm evidence I believe that "C++ -> pthreads layer -> Win32 threads" is slower than "C++ -> Win32 threads", which is what I based my assumption upon.

Makes sense in the absence of any other information. But the key point here is that 'C++' is essentially the same as 'pthreads' with a C++ face - the primitives and the operations are basically the same. So if you refactor your C++ -> Win32 implementation into C++ -> stable nontemplated API -> Win32 and then observe that the stable nontemplated API is essentially the same as pthreads with several minor differences here and there... you get N2178.

I guess this post doesn't really belong here, but I just stumbled upon the following: http://msdn2.microsoft.com/en-us/library/ms686360.aspx Lots of new multi-threading functionality under Vista/Longhorn. / Johan

"Peter Dimov" <pdimov@gmail.com> skrev i meddelandet news:eu8l25$2sr$1@sea.gmane.org...

Johan Nilsson wrote:

...

I guess this post doesn't really belong here, but I just stumbled upon the following:

http://msdn2.microsoft.com/en-us/library/ms686360.aspx

Lots of new multi-threading functionality under Vista/Longhorn.

Yeah, the Win32 team has finally caught wind of that newfangled pthread model.

I can't help but noticing a slight hint of sarcasm here ;-) With "new", I meant new as in "new for Win32" (or, perhaps Win32/64). Regards / Johan

Peter Dimov wrote:

Johan Nilsson wrote:

...

"Peter Dimov" <pdimov@gmail.com> skrev i meddelandet news:eu8l25$2sr$1@sea.gmane.org...

...

Johan Nilsson wrote:

...

I guess this post doesn't really belong here, but I just stumbled upon the following:

http://msdn2.microsoft.com/en-us/library/ms686360.aspx

Lots of new multi-threading functionality under Vista/Longhorn.

Yeah, the Win32 team has finally caught wind of that newfangled pthread model.

I can't help but noticing a slight hint of sarcasm here ;-)

With "new", I meant new as in "new for Win32" (or, perhaps Win32/64).

My apologies. The sarcasm was not aimed at you at all. I didn't make the connection between my 'newfangled' and your 'new' when posting.

No problem. / Johan

"Peter Dimov" <pdimov@mmltd.net> writes:

Anthony Williams wrote:

...

For a non-pthreads platform, it might make sense to implement pthreads in terms of the C++ interface, which is implementes in terms of the native API, rather than implement the C++ interface in terms of pthreads, which is then implemented in terms of the native API.

Yes, this is possible, and there are no technical reasons to avoid this implementation approach. There are, how should I put that, ideological reasons to not target non-pthread platforms directly, though. At some point we need to draw a line and say: this is the thread abstraction we're coding against, and could vendors please get along with the program?

Well, if we *do* include pthreads AND a C++ API as part of the C++ Standard, then that's quite a good hint. It doesn't mean that you have to implement one in terms of the other, though.

This stance is obviously unsustainable if there are strong technical reasons against it. But I think that there aren't.

Is performance not a legitimate reason? What about avoiding unnecessary complications?

...

Lets take call_once, for example. PThreads only allows a real function with no parameters returning void as the once function. This is relatively easy to implement on top of a C++-style call_once that allows arbitrary function objects with parameters, whereas it is much harder to do the reverse.

There is a note in N2178 that shows how to implement call_once on top of pthread_once2_np. ;-)

I know that it's possible, just "much harder": pretty much anything is more complicated than typedef once_flag pthread_once_t; int pthread_once(pthread_once_t* once_control,void(*init_routine)()) { std::call_once(*once_control,init_routine); return 0; } Since on Windows the code for call_once is pretty much the same whether or not init_routine takes arguments, it seems pointless to go round the houses setting thread-specific data just for the benefit of writing call_once in terms of pthread_once rather than vice versa. Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL

Alexander Terekhov <terekhov@web.de> writes:

Anthony Williams wrote: [...]

...

typedef once_flag pthread_once_t; int pthread_once(pthread_once_t* once_control,void(*init_routine)()) { std::call_once(*once_control,init_routine); return 0; }

Right, the only problem is that folks like Drepper of Red Hat would rather commit suicide than accept the notion of pthread.h implementable in terms of <thread>. See the comedy at posix-c++-sg.

I'm not subscribed to posix-c++-sg, and the archives aren't available publicly. The list info page says it's a closed list: are random people (i.e. me) likely to get approval to subscribe? Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL

Anthony Williams wrote:

I'm not subscribed to posix-c++-sg, and the archives aren't available publicly. The list info page says it's a closed list: are random people (i.e. me) likely to get approval to subscribe?

I was approved the other day, and I'm at least as random as you.

"Peter Dimov" <pdimov@mmltd.net> writes:

Alexander Terekhov wrote:

...

Anthony Williams wrote: [...]

...

typedef once_flag pthread_once_t; int pthread_once(pthread_once_t* once_control,void(*init_routine)()) { std::call_once(*once_control,init_routine); return 0; }

Right, the only problem is that folks like Drepper of Red Hat would rather commit suicide than accept the notion of pthread.h implementable in terms of <thread>. See the comedy at posix-c++-sg.

This is not a good idea for maintenance reasons, because it still leaves us with two separate implementations, one for POSIX, one for pthreads-over-C++ (which is currently a synonym for Windows.)

That's a non-starter of an argument. The alternative is C++-over-POSIX and C++-over-POSIX-over-xxx for non-pthreads platforms, which is still two implementations. Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL

Anthony Williams wrote:

"Peter Dimov" <pdimov@mmltd.net> writes:

...

There is a note in N2178 that shows how to implement call_once on top of pthread_once2_np. ;-)

I know that it's possible, just "much harder": ...

You haven't even looked at it, have you. :-/ [Note: a typical implementation of call_once is shown below: template< class F > void __call_once_helper( void * pv ) { F * pf = static_cast< F * >( pv ); (*pf)(); } template< class F > void call_once( pthread_once_t & once_control, F f ) { pthread_once2_np( &once_control, &__call_once_helper<F>, &f ); } --end note.]

Since on Windows the code for call_once is pretty much the same whether or not init_routine takes arguments, it seems pointless to go round the houses setting thread-specific data just for the benefit of writing call_once in terms of pthread_once rather than vice versa.

That is why there is pthread_once2_np in N2178. The actual call_once code is pretty much the same on POSIX as well, and it is equally pointless to use TLS there. So I tried to fix both.

Anthony Williams wrote:

OK. But that doesn't really address my initial point: implementing call_once in terms of the POSIX pthread_once is a bit pointless if you need to implement pthread_once separately anyway. If you don't have pthread_once, I would write call_once (or, equivalently pthread_once2_np), and implement pthread_once in terms of that.

This is tied to another issue, whether the implementation should be header-only. If one is committed to produce a header-only implementation, it makes sense to reuse your current call_once in the implementation of pthread_once (and possibly pthread_once2_np). The N2178 pthread.h aims to support the scenario where one is to deliver a separately compiled pthread-win32.lib/dll (as part of the Platform SDK, say ;-) ). This is why I've tried to make sure that the C++ templated layer is implementable without loss of performance or generality in terms of a non-templated API (which happens to be a slightly extended pthreads layer). With careful definitions of the pthread_* types, it's possible to achieve binary compatibility as well, that is, link the program against a newer version of pthread-win32.lib/dll and have it work without recompilation. This is not possible with header-only templates or inline functions, because they are burned into the application code at compile time.

Anthony Williams wrote:

If you define a C API, and people stick to it, then it doesn't matter what you use internally: you can ship a .LIB file or .DLL with just that API exposed, and Bob's your uncle. If you want to use a templated API, you just can. You can even provide the necessary specializations of the templates as part of the library, and declare them as explicit specializations in the headers.

Case A, call_once implemented in the header. You ship an improved call_once.hpp. User program needs recompilation to take advantage of the improvement. Case B, call_once a thin wrapper over a C API, as in N2178. You ship an improved .lib/.dll. User program doesn't need recompilation (lib) or relink (dll).

----Original Message---- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Yuval Ronen Sent: 23 March 2007 11:43 To: boost@lists.boost.org Subject: Re: [boost] Boost.Threads, N2178, N2184, et al

Peter Dimov wrote:

...

Case A, call_once implemented in the header. You ship an improved call_once.hpp. User program needs recompilation to take advantage of the improvement.

Case B, call_once a thin wrapper over a C API, as in N2178. You ship an improved .lib/.dll. User program doesn't need recompilation (lib) or relink (dll).

That strikes me as an ADVANTAGE of a header only implementation. As an application author, I don't want somebody changing a library I'm using under my feet. (My favourite example is: Library documents that it returns a collection in a random order. Actual implementation returns sorted data. Application starts using std::lower_bound (which is a bug in the application, but works). Application ships. Library is improved and returns collection in unsorted order. Application crashes.)

Has anyone ever encountered a scenario where a new version of a library is published, and applications using it consider re-compiling a major (or even minor) drawback?

Well, I've often encountered the case that I consider changing to a new library has associated disadvantages (I need to go round retesting everything). But the recompilation is certainly not an issue.

And if so, why does Boost push so hard for a header-only implementation of its libraries? Isn't it a contradiction? There is a difference between Boost (which is hard to install - though getting easier), and a C++ implementation (which usually is pretty straightforward to install on it's target platforms).

-- Martin Bonner Project Leader PI SHURLOK LTD Telephone: +44 1223 441434 / 203894 (direct) Fax: +44 1223 203999 Email: martin.bonner@pi-shurlok.com www.pi-shurlok.com

Not a direct response to Anthony who already knows most of that, just a general explanation of the choices behind N2178: Anthony Williams wrote:

Sorry, I misread what you wrote above, and looked at the /implementation/ of pthread_once2_np. I agree that it's not that hard to write call_once in terms of pthread_once2_np, but that isn't what POSIX provides.

The idea behind the _POSIX_CXX09_EXTENSIONS part of N2178 is precisely to allow a pthread implementor (in the Windows case that's us) to avoid these sources of inefficiency. The join2 functions serve as a similar example. On Windows, they are trivially implementable with no overhead using WaitForSingleObject on the thread HANDLE. On non-extended POSIX, the situation is more complicated. The join2 functions illustrate another point where I believe N2178 is superior to N2184: they expose the superior Windows join model at the C++ level instead of the more limited pthread_join. It's trivial to make an n2184::thread provide the same semantics under Windows, but nobody would be able to take advantage of them portably.

Yuval Ronen wrote:

Peter Dimov wrote:

...

The join2 functions illustrate another point where I believe N2178 is superior to N2184: they expose the superior Windows join model at the C++ level instead of the more limited pthread_join. It's trivial to make an n2184::thread provide the same semantics under Windows, but nobody would be able to take advantage of them portably.

The Windows superior join model? Can you please explain what do you mean by that?

You can "join" a thread more than once, from several threads, and you can do try/timed joins.

Glenn Schrader wrote:

Peter Dimov wrote:

...

Yuval Ronen wrote:

...

The Windows superior join model? Can you please explain what do you mean by that?

You can "join" a thread more than once, from several threads, and you can do try/timed joins.

This implies that information about terminated threads "hangs around" for some amount of time in case more threads which to "join" it. Do you know what the Windows policy is regarding when the tid (or the Windows equivalent) finally becomes invalid?

There is a separate CloseHandle call, after which the handle is no longer valid.

"Peter Dimov" <pdimov@mmltd.net> writes:

Anthony Williams wrote:

...

"Peter Dimov" <pdimov@mmltd.net> writes:

...

I don't see call_once in jss_thread.zip, by the way; maybe you forgot to put it into the archive?

Oops. Thanks for spotting that. I've added it to the archive, and updated it to take multiple arguments in passing.

Some comments on that:

Thanks for taking the time to look at this.

template<typename Function> void call_once(once_flag& flag,Function f) { // Try for a quick win: if the proceedure has already been called // just skip through: long const function_complete_flag_value=0xc15730e2;

if(::jss::detail::interlocked_read(&flag)!=function_complete_flag_value) { char mutex_name[::jss::detail::once_mutex_name_length]; void* const mutex_handle(::jss::detail::create_once_mutex(mutex_name,&flag)); BOOST_ASSERT(mutex_handle); detail::win32::handle_holder const closer(mutex_handle); detail::win32_mutex_scoped_lock const lock(mutex_handle);

if(::jss::detail::interlocked_read(&flag)!=function_complete_flag_value) { f(); JSS_INTERLOCKED_EXCHANGE(&flag,function_complete_flag_value); } } }

The first load needs to be a load_acquire; the second can be ordinary since it's done under a lock. The store needs to be store_release.

I didn't want to think about acquire/release semantics when I wrote that, so I just went for "ordered" ops. Agreed that the second read can be ordinary. Actually I think the store can be ordinary too since it's also done under a lock, and the unlock has (or should have, anyway) release semantics. I agree that the first read needs to be load_acquire, though: without the acquire, there's no synchronization in the case that the flag has been set, and there's nothing to "release".

An interlocked_read is stronger ('ordered') and more expensive than needed on a hardware level, but is 'relaxed' on a compiler level under MSVC 7.1 (the optimizer moves code around it). It's 'ordered' for the compiler as well under 8.0; the intrinsics have been changed to be compiler barriers as well. InterlockedExchange is similar.

Have you got a reference for that? I would be interested to read about the details; MSDN is sketchy.

A load_acquire can be implemented as a volatile read under 8.0, and a volatile read followed by _ReadWriteBarrier under 7.1.

Why don't you need the barrier on 8.0? You need something there in order to prevent the CPU from doing out-of-order reads (and stores), even if the compiler won't reorder things. In fact, looking at the assembly code generated, I believe you need more than a _ReadWriteBarrier in both cases, as it seems to be purely a compiler barrier, and not a CPU barrier. On x86, I think a load_acquire needs to either be a simple load followed by an MFENCE, or a fully ordered RMW operation. The compiler Interlocked intrinics will generate the latter, but I don't know how to do the former short of writing inline assembly. Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL

Anthony Williams wrote:

"Peter Dimov" <pdimov@mmltd.net> writes:

...

An interlocked_read is stronger ('ordered') and more expensive than needed on a hardware level, but is 'relaxed' on a compiler level under MSVC 7.1 (the optimizer moves code around it). It's 'ordered' for the compiler as well under 8.0; the intrinsics have been changed to be compiler barriers as well. InterlockedExchange is similar.

Have you got a reference for that? I would be interested to read about the details; MSDN is sketchy.

The documentation for the intrinsics now states that they act as a compiler barrier for 8.0. http://msdn2.microsoft.com/en-us/library/1s26w950.aspx The documentation that shipped with VC 7.1 did not, and in fact I have observed the optimizer moving code across an interlocked intrinsic when I developed the prototype of N2195.

...

A load_acquire can be implemented as a volatile read under 8.0, and a volatile read followed by _ReadWriteBarrier under 7.1.

Why don't you need the barrier on 8.0? You need something there in order to prevent the CPU from doing out-of-order reads (and stores), even if the compiler won't reorder things. In fact, looking at the assembly code generated, I believe you need more than a _ReadWriteBarrier in both cases, as it seems to be purely a compiler barrier, and not a CPU barrier.

On x86 all loads already have acquire semantics by default, and all stores have release semantics. MSVC 8.0 extends a volatile load/store to have acquire/release semantics (both hardware and compiler) on every platform, including IA64. http://msdn2.microsoft.com/en-us/library/12a04hfd.aspx

Anthony Williams wrote:

"Peter Dimov" <pdimov@gmail.com> writes:

...

On x86 all loads already have acquire semantics by default, and all stores have release semantics.

Not according to the intel specs. 25366818.pdf (IA32 software developers manual volume 3A), section 7.7.2:

"1. Reads can be carried out speculatively and in any order."

This means that loads cannot have acquire semantics as they can be reordered with respect to other reads. You need a fence or other serializing instruction to stop this reordering.

This only happens on P4, Xeon and P6 processors --- Pentium and 486 CPUs don't do out-of-order reads.

Physical reads are carried out speculatively and out of order, but they still have acquire semantics from the program's point of view. The CPU has a unit that watches for load-acquire violations and discards the speculatively read value if such a violation is detected. This is a recurring topic in c.p.t and comp.arch, I believe, mostly because the Intel and AMD specs aren't clear. Alexander likes to cite the IA64 specs where it is stated that ld.acq and st.rel produce the equivalent of the x86 memory model.

Anthony Williams wrote:

"Peter Dimov" <pdimov@gmail.com> writes:

...

On x86 all loads already have acquire semantics by default, and all stores have release semantics.

On Itanium, sure. <quote source=Intel Itanium Architecture Software Developer's Manual> 6.3.4 Memory Ordering Interactions IA-32 instructions are mapped into the Itanium memory ordering model as follows: - All IA-32 stores have release semantics - All IA-32 loads have acquire semantics - All IA-32 read-modify-write or lock instructions have release and acquire semantics (fully fenced). </quote>

Not according to the intel specs. 25366818.pdf (IA32 software developers manual volume 3A), section 7.7.2:

The thing is that x86 native doesn't have officially defined memory model (Itanium mapping may well be stronger than x86 native). Note that what you quote below was written for testers with scopes on "system bus".

"1. Reads can be carried out speculatively and in any order."

However... http://www.well.com/~aleks/CompOsPlan9/0005.html <quote author=an architect at Intel> The PPro does speculative and out-of-order loads. However, it has a mechanism called the "memory order buffer" to ensure that the above memory ordering model is not violated. Load and store instructions do not get retired until the processor can prove there are no memory ordering violations in the actual order of execution that was used. Stores do not get sent to memory until they are ready to be retired. If the processor detects a memory ordering violation, it discards all unretired operations (including the offending memory operation) and restarts execution at the oldest unretired instruction. </quote> Consider also: Kourosh Gharachorloo, Anoop Gupta, and John Hennessy. Two techniques to enhance the performance of memory consistency models. In Proceedings of the 1991 International Conference on Parallel Processing (Vol. I Architecture), pages 1-355-364, August 1991. <quote> The speculative-load buffer provides the detection mechanism by signaling when the speculated result is incorrect. The buffer works as follows. Loads that are retired from the reservation station are put into the buffer in addition to being issued to the memory system. There are four fields per entry (as shown in Figure 4): load address, acq, done, and store tag. The load address field holds the physical address for the load. The acq field is set if the load is considered an acquire access. For SC, all loads are treated as acquires. The done field is set when the load is performed. If the consistency constraints require the load to be delayed for a previous store, the store tag uniquely identifies that store. A null store tag specifies that the load depends on no previous stores. When a store completes, its corresponding tag in the speculative-load buffer is nullified if present. Entries are retired in a FIFO manner. Two conditions need to be satisfied before an entry at the head of the buffer is retired. First, the store tag field should equal null. Second, the done field should be set if the acq field is set. Therefore, for SC, an entry remains in the buffer until all previous load and store accesses complete and the load access it refers to completes. Appendix A describes how an atomic read-modify-write can be incorporated in the above implementation. We now describe the detection mechanism. The following coherence transactions are monitored by the speculativeload buffer: invalidations (or ownership requests), updates, and replacements.3 The load addresses in the buffer are associatively checked for a match with the address of such transactions.4 Multiple matches are possible. We assume the match closest to the head of the buffer is reported. A match in the buffer for an address that is being invalidated or updated signals the possibility of an incorrect speculation. A match for an address that is being replaced signifies that future coherence transactions for that address will not be sent to the processor. In either case, the speculated value for the load is assumed to be incorrect. Guaranteeing the constraints for release consistency can be done in a similar way to SC. The conventional way to provide RC is to delay a release access until its previous accesses complete and to delay accesses following an acquire until the acquire completes. Let us first consider delays for stores. The mechanism that provides precise interrupts by holding back store accesses in the store buffer is sufficient for guaranteeing that stores are delayed for the previous acquire. Although the mechanism described is stricter than what RC requires, the conservative implementation is required for providing precise interrupts. The same mechanism also guarantees that a release (which is simply a special store access) is delayed for previous load accesses. To guarantee a release is also delayed for previous store accesses, the store buffer delays the issue of the release operation until all previously issued stores are complete. In contrast to SC, however, ordinary stores are issued in a pipelined manner. </quote> and, also somewhat related: http://www.cs.wisc.edu/~cain/pubs/micro01_correct_vp.pdf regards, alexander.

Howard Hinnant wrote:

Fwiw, this is the approach of N2184:

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2184.html

The very biggest change between boost::thread and N2184 is the addition of cancellation. And, imho, this is a huge change.

There are several more minor changes:

* The N2184::thread is non-copyable like boost::thread, but movable (taking advantage of new C++ language and std::lib features). Movable N2184::thread maintains the one-to-one mapping between the std::thread and the OS thread which boost has (sole ownership semantics). It just adds the ability to move the thread between scopes (such as return from factory functions). I recently drew the analogy this way:

boost::thread is like a scoped_ptr to the OS thread. N2184::thread is like an auto_ptr to the OS thread (the proposed unique_ptr would be a more accurate analogy).

Makes a lot of sense. Makes me wonder why the mutexes in N2094 are not movable also.

* N2184::thread separates out the concept of thread identity and thread handle. In boost these are both just boost::thread. The thread::id is copyable. The only thing you can do with it is equality compare it.

I think we might want it to be OutStreamable also. Mainly for debugging and logging purposes.

Howard Hinnant wrote:

On Mar 21, 2007, at 5:20 PM, Yuval Ronen wrote:

...

Makes a lot of sense. Makes me wonder why the mutexes in N2094 are not movable also.

I actually thought about that. My inclination is to model pthreads (which also appears to have been the boost inclination). And pthread_mutex_t does not appear to be a copyable item.

Copyable - no, but is it movable?

And I'm wanting to be able to implement like this:

class mutex { pthread_mutex_t mut_; public: ... };

as opposed to:

class mutex { pthread_mutex_t* mut_; // on heap public: ... };

Perfectly understood.

It appears that a pthread implementation is allowed to point into a pthread_mutex_t and thus copy/moving it becomes problematic.

So according to this, it seems that pthread_mutex_t isn't moveable. That's a problem.

To alleviate this, N2094 does make the locks movable. The locks simply have a pointer to the mutex. The pointer controls (owns) the locked status of the mutex, not the lifetime of it though.

Yes, of course.

The usual idiom is to have the mutex at namespace scope, or as a class data member, and have the lock referring to that mutex "on the stack". So for the usual idioms, non-copyable, non-movable mutexes do not seem overly problematic.

I'm not sure it's not problematic. If the mutex is class data member (a very common scenario - perhaps the most common one), then it makes that class non-movable. Non-copyable - I understand, but non-movable - that's a major limitation, IMO.

In contrast, pthread_t is clearly copyable.

pthread_t pthread_self();

However the semantics of pthread_t when looked at through a C++ lens is: sole ownership.

* A pthread_t initialized from pthread_create, must be joined or detached exactly once.

No argument between us here...

Howard Hinnant wrote:

On Mar 22, 2007, at 1:14 PM, Yuval Ronen wrote:

...

...

The usual idiom is to have the mutex at namespace scope, or as a class data member, and have the lock referring to that mutex "on the stack". So for the usual idioms, non-copyable, non-movable mutexes do not seem overly problematic. I'm not sure it's not problematic. If the mutex is class data member (a very common scenario - perhaps the most common one), then it makes that class non-movable. Non-copyable - I understand, but non-movable - that's a major limitation, IMO.

I think we're ok here, both for copying and moving. Here's a prototype class A which owns an int on the heap. And because I think this is a cool use of the read/write locks from N2094 I'm going to use them to implement the copy assignment. :-)

I'm typing this code in untested, so please forgive me any careless mistakes.

class A { int* the_data_; std::sharable_mutex mut_;

typedef std::sharable_lock<std::sharable_mutex> read_lock; typedef std::exclusive_lock<std::sharable_mutex> write_lock; public: explicit A(int data) : the_data_(new int(data)) {} ~A() {delete the_data_;}

// copy semantics A(const A& a) : the_data_(0) { read_lock lk(a.mut_); if (a.the_data_ != 0) the_data_ = new int(*a.the_data_); } A& operator=(const A& a) { if (this != &a) { read_lock lock_a(a.mut_, std::defer_lock); write_lock lock_this(mut_, std::defer_lock); std::lock(lock_this, lock_a); // prevents deadlock int* new_data = a.the_data_ ? new int(*a.the_data_) : (int*)0; delete the_data_; the_data_ = new_data; } return *this; }

// move semantics A(A&& a) : the_data_(a.the_data_) { a.the_data_ = 0; // no locking required, a is an rvalue! :-) } A& operator=(A&& a) { write_lock lock_this(mut_); delete the_data_; the_data_ = a.the_data_; a.the_data_ = 0; return *this; } };

I.e. you just move or copy the protected data and not the mutex.

The main thing I was thinking about is the move constructor. Instead of stealing the mutex from the rvalue, you construct a new mutex, and let the old one be destructed. This might be not optimal. But on the other hand, it might not be that bad after all, I don't know...

Howard Hinnant wrote:

On Mar 22, 2007, at 3:20 PM, Yuval Ronen wrote:

...

The main thing I was thinking about is the move constructor. Instead of stealing the mutex from the rvalue, you construct a new mutex, and let the old one be destructed. This might be not optimal. But on the other hand, it might not be that bad after all, I don't know...

<nod> If it did turn up a red flag in the profiling of A, then perhaps A could be changed to:

class A { int* the_data_; std::unique_ptr<std::sharable_mutex> mut_; ...

With an invariant that if the_data_ is non-null then so is mut_, otherwise both are null.

This would probably (I haven't fully analyzed it) be no more inefficient than if we did this:

class mutex { pthread_mutex_t* mut_; // on heap ...

And it enables other clients of mutex to not have to pay for a heap allocation internal to std::mutex if their app didn't need it.

I agree.

Hmm... and if your std::mutex ctor can throw an exception (in general it can), the A design with a plain mutex has a throwing move constructor which can be bad if you want to vector<A>. The A design with the mutex* side steps that problem I believe.

So the general recommendation for a class that holds a mutex and should be movable is to dynamically allocate the mutex. Pity, but maybe inevitable. <fantasy> Unless POSIX would change to allow movable pthread_mutex_t... </fantasy>

On Mar 21, 2007, at 6:01 PM, Emil Dotchevski wrote:

...

* The N2184::thread is non-copyable like boost::thread, but movable (taking advantage of new C++ language and std::lib features). Movable N2184::thread maintains the one-to-one mapping between the std::thread and the OS thread which boost has (sole ownership semantics). It just adds the ability to move the thread between scopes (such as return from factory functions). I recently drew the analogy this way:

boost::thread is like a scoped_ptr to the OS thread. N2184::thread is like an auto_ptr to the OS thread (the proposed unique_ptr would be a more accurate analogy).

This assumes that it is not desirable for the same N2184::thread object to be shared between multiple owners. The rationale given for the non- copyable semantics of boost::thread states that the only time you need copyable handle is in what they call "Use case 6: Creation of a thread whose ownership is shared between multiple objects", which is like saying that you only need shared handles when you need shared handles (http://www.boost.org/doc/html/threads/rationale.html#threads.rationale.non-c... doc/html/threads/rationale.html#threads.rationale.non- copyable.shared.)

I suppose similar argument can be made for movable semantics for N2184::thread, but again, what about this use case? Does N2184 argue that this use case does not exist in practice?

For this use case, N2184 argues that the use case should be supported with a higher-level library which sits non-intrusively on top of std::thread. My preferred name for this is std::shared_future (though std::tr2::shared_future is a more likely name). shared_future would be copyable, multi-joinable, and would deliver both normal and exceptional results from an asynchronous job when the shared_future is joined with. The advantage of putting this functionality in a separate class is that it costs space and time. And for clients that do not need this functionality it seems like a shame to force them to pay for it. So the big picture is to provide a series thread management objects: from low level to high level, with the lowest level being relatively small, cheap and fast, but lacking features. And the highest level adding all the niceties along with the subsequent cost. And it is hoped and anticipated that the highest level objects will end up being the most commonly used. std::thread is targeted at the very lowest level in this plan -- a solid foundation upon which much cooler stuff can be built: not only by std::lib vendors, but by anybody - portably. Peter Dimov and Ion Gaztanaga have both implemented shared ownership thread manager objects (i.e. future) on top of boost::thread (non- intrusively). I'm extremely grateful for their ground breaking proof of concept work showing that we don't need a copyable low level to support a copyable higher level. And this statement is not meant to imply that either Peter or Ion support or approve of the non-copyable (but move-only) lower level, so I feel a little guilty about stating it. But nevertheless I am grateful. And fwiw, N2184::thread delivers a std::thread which I believe has the more common motivations for a copyable std::thread: You can return it from factory functions and you can put into std::containers. These actions will only require movability, not copyability, in C++0X. -Howard

On Mar 21, 2007, at 7:30 PM, Emil Dotchevski wrote:

...

For this use case, N2184 argues that the use case should be supported

...

with a higher-level library which sits non-intrusively on top of std::thread. My preferred name for this is std::shared_future (though std::tr2::shared_future is a more likely name). shared_future would be copyable, multi-joinable, and would deliver both normal and exceptional results from an asynchronous job when the shared_future is joined with.

The advantage of putting this functionality in a separate class is that it costs space and time.

Assuming that the copyable semantics are commonly useful, is it easier to build them on top of N2184 than on top of, say pthreads (which has been ported to Windows too and is therefore competing with N2184 as the base of such higher level interface)?

Yes, I believe shared_future is more easily implemented in terms of N2184 than pthreads. Exception safety alone is a significant win. If a parent thread which owns a pthread_t on the stack experiences an exception (such as cancellation) propagating through the stack frame, it must explicitly deal with the pthread_t. N2184 automates this concern, first canceling the child thread, and then detaching it. A raw pthread_t would not only ignore the cancellation of the parent thread, it would also leak the pthread_t, never joining with it, nor detaching it. Indeed, N2184 is meant to be little more than an RAII wrapper around pthread semantics.

Speaking of which, on certain operating systems, pthreads probably is a good base for implementing the N2184 interface, but that way we're going from copyable handles to non-copyable handles. And now you're saying I can build copyable handles interface on top of N2184? I could be missing something important, but this doesn't make sense to me.

...

Peter Dimov and Ion Gaztanaga have both implemented shared ownership thread manager objects (i.e. future) on top of boost::thread (non- intrusively). I'm extremely grateful for their ground breaking proof of concept work showing that we don't need a copyable low level to support a copyable higher level.

Proof that it is possible is one thing, but is it trivial?

The current technique is to layer on top of a std::thread a std::mutex and std::condition that synchronize access using a function adaptor. It is difficult for me to characterize this as trivial or not. I am now used to it and find it relatively easy. But I also remember having trouble coming up with this solution. So I fully support std::supported encapsulations of this effort. On the other hand if you force the std::lib to put mutexes and condition variables inside std::thread (or whatever the lowest level is called), then you remove the possibility that some clients who don't need this overhead don't have to pay for it. Analogy: Do you want a mutex inside of ostream::sentry or not?

...

And fwiw, N2184::thread delivers a std::thread which I believe has the more common motivations for a copyable std::thread: You can return it from factory functions and you can put into std::containers. These actions will only require movability, not copyability, in C++0X.

You can do all that with shared handles too, even in current STL containers. I wasn't comparing N2184 to boost::thread, I was comparing it to N2178.

Sorry. I meant to say that a unique-ownership std::thread can provide this functionality (return from factory functions, place in containers) cheaper than a shared-ownership std::thread can, at least on pthread-based platforms. -Howard

headl004@bama.ua.edu wrote:

Are the projects due on August 20th or ond the evaluation deadline?

August 20th http://code.google.com/support/bin/answer.py?answer=60325&topic=10729 Evaluation deadline is for mentors to say if the student gets paid... Jeff

headl004@bama.ua.edu wrote:

...

Evaluation deadline is for mentors to say if the student gets paid...

Haha, okay thanks. Just wanted to make sure the amount of time allowed to work on the project.

Ah, I wasn't joking. If the mentor evaluation is negative the student doesn't get the last payment. Jeff

headl004@bama.ua.edu wrote:

Are the projects due on August 20th or ond the evaluation deadline?

Please don't change the topic in the middle of an ongoing discussion. Start a new thread instead of replying to an unrelated message. Regards, m Send instant messages to your online friends http://au.messenger.yahoo.com

Sohail Somani wrote:

A fine reason, but why would you want to lump it in with <thread>? Or alternatively, require <thread> to depend on <pthread.h>? Looks like there should be another proposal to add <pthread.h> to the standard...

I've tried to address some of these questions in N2178, sections III and IV. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2178.html#principle... Basically, it's easy for the committee to adopt just the pthread.h portion of N2178 if they like, but I've presented both APIs in the document to emphasize that they form a coherent whole, are intended to play well together (in mixed C/C++ code), and are merely different views to the same threading core.

Sohail Somani wrote:

Aside: Why do you want to bring pthreads into the mix?

Couple of reasons. 1. Some C++ committee members would like to see a C/C++ API instead of pure C++. This is important for mixed C/C++ code, but there's also the fact that the C committee is also looking at threading and will likely evaluate whatever is adopted by the C++ standard for eventual inclusion into C. 2. We need some platform abstraction layer. The usual approach has been to target Windows and Pthreads. I believe that writing a Pthread implementation for Windows once is a better option; this allows the higher level C++ API to only target a single abstract platform (arguably designed by the best in the field) and is (in my opinion) more maintainable in the long run.

Aside: Why do you want to bring pthreads into the mix? I know the two threading APIS are Windows and pthreads but still... Couldn't you define a new API that would be a thin wrapper for pthreads?

What would be the benefit of coming up with a third API? Would there be any differences between that API and pthreads, and if not, why not use pthreads directly? Besides, N2178 isn't simply based on pthreads (in that case you'd have a point, why talk about pthreads if it's simply an implementation detail). It defines a model for interoperability between C++ code (using <thread>) and C code (using pthreads). Emil Dotchevski

Peter Dimov wrote: [... TLS ...] I'm somewhat missing thread_specific_ptr<>. And corresponding _POSIX_CXX09_EXTENSIONS pthread_resetspecific_np(), pthread_releasespecific_np(), and pthread_disposespecific_np(). Stage 2? ;-) regards, alexander.

Peter Dimov wrote:

Alexander Terekhov wrote:

...

Peter Dimov wrote:

[... TLS ...]

I'm somewhat missing thread_specific_ptr<>. And corresponding _POSIX_CXX09_EXTENSIONS pthread_resetspecific_np(), pthread_releasespecific_np(), and pthread_disposespecific_np().

The current plan is to standardize something like __thread or __declspec(thread) for TLS. I hope. :-)

I know. But __thread proposal doesn't address dynamic context. "objects which may have static storage duration may have thread storage duration instead. These objects include namespace-scope variables, function-local static variables, and class static member variables ... __thread specifier can be applied only to the names of objects of block scope that also specify static or to the names of objects of namespace scope." regards, alexander.

on Mon Mar 19 2007, "Peter Dimov" <pdimov-AT-mmltd.net> wrote:

I could've gone with Howard's exclusive_lock, but it's a bit too lengthy and hard to type for repeated use.

unique_lock (c.f. unique_ptr)? -- Dave Abrahams Boost Consulting www.boost-consulting.com

Peter Dimov wrote:

Yuval Ronen wrote:

...

Peter Dimov wrote:

...

Hello everyone,

I'd like to start working on a Boost implementation of my threading proposal N2178:

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2178.html Just some naming comments, if I may:

It seems to me the word "lock" is used with different meanings. For example, there's a "rwlock" and the scoped wrappers for it are "read_lock" and "write_lock". Very confusing, IMO. I think the word "lock" should either be used for the synchronization classes, or for the scoped wrappers, but not both.

I've consistently used the POSIX names. I agree that it's unfortunate that they used mutex and rwlock for the two primitives.

Indeed unfortunate... But maybe there's no real need to follow the POSIX name so rigidly for the C++ classes?

...

The word "scoped" appears only for mutex's scoped_lock, although read_lock and write_lock are also scoped. My opinion is that this word should either be part of the names of all the scoped wrappers or none of them, in a uniform manner.

I tried the "none of them" approach, calling scoped_lock just lock, but it turned out to not work well in practice since it's better to reserve 'lock' for the variable name. I could've gone with Howard's exclusive_lock, but it's a bit too lengthy and hard to type for repeated use.

So maybe we should go for the "all of them" approach? scoped_mutex, scoped_rlock, scoped_wlock?