On 23 Aug 2015 at 18:42, gentryx@gmx.de wrote:

...

Niall, Ahmed, it will be very difficult to review this library as it has dependencies on libraries that are not adopted by Boost. The documentation is full of references to both libraries APBind and Monad.

If they are internals, no mention should be done on the documentation.

AFAIK Monad is visible to the user as AFIO functions return them.

Correct. I summarise everything you need to know about monad<T> on the first page of the tutorial https://boostgsoc13.github.io/boost.afio/doc/html/afio/quickstart/work shop/naive.html. They can be treated as if an always-ready shared_future<T>. Looking over the references in the docs to APIBind, I agree I over-mention APIBind, and I have logged https://github.com/BoostGSoC13/boost.afio/issues/87 which is to remove all mentions of APIBind in the docs. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 23 Aug 2015 at 23:57, Vicente J. Botet Escriba wrote:

I believe that I need something more about monad<T> than what I have found in the documentation.

If there is anything missing, please do say in detail.

BTW, the link to Boost.Monad [1] is broken in https://boostgsoc13.github.io/boost.afio/doc/html/afio/overview.html.

I profoundly apologise. In preparation for this review I relocated the Boost.Monad documentation onto GitHub's web service at https://ned14.github.io/boost.monad/ as it's more reliably there than ci.nedprod.com. I obviously forgot to update the link in the AFIO docs, despite having specifically moved the docs for this purpose. It's fixed now on develop branch. Thanks for the spot.

Just a question, does the user need to use some APBind interfaces or is this just an internal implementation detail (Sorry, I don't know nothing about APBind)?

If you want AFIO to use Boost.Thread, you define BOOST_AFIO_USE_BOOST_THREAD=1 before including. Else it uses the STL thread. The same goes for BOOST_AFIO_USE_BOOST_FILESYSTEM and ASIO_STANDALONE. If you want to always use latest AFIO, you bind afio like this: namespace afio = boost::afio; If you want to pin a specific ABI which won't break in the future, you bind afio like this: namespace afio = BOOST_AFIO_V2_NAMESPACE; That's probably all most people care about. More advanced stuff can be seen at https://boostgsoc13.github.io/boost.afio/doc/html/afio/FAQ/multi_abi.h tml.

It will be easier to know the scope of the review and what is left for the near future. Is the monad part of the review?

I am happy to accept bug reports on monad<T>. But I don't think the AFIO review is reviewing monad<T>, nor do you need to. I mean, if you like the design of boost::shared_future<T>, then by definition you like the design of monad<T>. I would have said monad<T> is therefore uncontroversial, as it's already in Boost.Thread and is very well understood (especially by you Vicente!). Just everywhere where you see monad<T>, replace it in your mind with boost::shared_future<T>. It has the same API and same behaviour. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 24 Aug 2015 at 0:46, Vicente J. Botet Escriba wrote:

...

...

I believe that I need something more about monad<T> than what I have found in the documentation. If there is anything missing, please do say in detail.

Is this link part of the review? I don't believe, so could you point me where the monad class is documented in AFIO? If it is not documented, why it is used so broadly.

It is used in the tutorial because as we progress through each stage of increasing complexity, you can see that the data_store class member functions lookup() and write() have either a monad<T> or a shared_future<T> return type depending on whether the API is synchronous (monad<T>) or asynchronous (shared_future<T>). I was trying to keep the interface class for each stage as identical as possible, so monad<T> was a natural fit to make the STL iostreams example look exactly like the AFIO asynchronous example. I suppose what you're saying here is that the tutorial should not be using monad<T> because monad<T> is not documented in AFIO. I can easily rewrite the tutorial code to not use monad<T>, it's no problem.

...

If you want to pin a specific ABI which won't break in the future, you bind afio like this:

namespace afio = BOOST_AFIO_V2_NAMESPACE;

Where can I find the description of this macro on the reference documentation?

https://boostgsoc13.github.io/boost.afio/doc/html/afio/quickstart/workshop/n...

We have two options: * we wait to review AFIO once Monad is reviewed in Boost * You include in AFIO, whatever you consider is need from Monad and only then we review AFIO.

I can remove monad<T> from all the tutorial code. Or I can include documentation of monad<T> in AFIO. Which would you prefer? Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 24.08.2015 04:29, Niall Douglas wrote:

On 24 Aug 2015 at 0:46, Vicente J. Botet Escriba wrote:

...

...

...

I believe that I need something more about monad<T> than what I have found in the documentation. If there is anything missing, please do say in detail.

Is this link part of the review? I don't believe, so could you point me where the monad class is documented in AFIO? If it is not documented, why it is used so broadly.

It is used in the tutorial because as we progress through each stage of increasing complexity, you can see that the data_store class member functions lookup() and write() have either a monad<T> or a shared_future<T> return type depending on whether the API is synchronous (monad<T>) or asynchronous (shared_future<T>).

I was trying to keep the interface class for each stage as identical as possible, so monad<T> was a natural fit to make the STL iostreams example look exactly like the AFIO asynchronous example. I suppose what you're saying here is that the tutorial should not be using monad<T> because monad<T> is not documented in AFIO.

I can easily rewrite the tutorial code to not use monad<T>, it's no problem.

...

...

If you want to pin a specific ABI which won't break in the future, you bind afio like this:

namespace afio = BOOST_AFIO_V2_NAMESPACE;

Where can I find the description of this macro on the reference documentation?

https://boostgsoc13.github.io/boost.afio/doc/html/afio/quickstart/workshop/n...

...

We have two options: * we wait to review AFIO once Monad is reviewed in Boost * You include in AFIO, whatever you consider is need from Monad and only then we review AFIO.

I can remove monad<T> from all the tutorial code. Or I can include documentation of monad<T> in AFIO. Which would you prefer?

Niall, I think you're missing the point. Is monad<T> part of the library interface? I.e. is the user allowed to use it to work with the library? If the answer is yes then it's a public part of the library and should be documented and reviewed. It doesn't matter if it is possible to code around it. If the answer is no then why is it there in the first place?

On 24 Aug 2015 at 12:27, Andrey Semashev wrote:

...

I can remove monad<T> from all the tutorial code. Or I can include documentation of monad<T> in AFIO. Which would you prefer?

Niall, I think you're missing the point. Is monad<T> part of the library interface? I.e. is the user allowed to use it to work with the library? If the answer is yes then it's a public part of the library and should be documented and reviewed. It doesn't matter if it is possible to code around it. If the answer is no then why is it there in the first place?

In hindsight the decision to use monad<T> in the tutorial examples was a mistake. I was trying to present a series of progressing steps which elegantly hanged together with respect to one another and which got the user used to the concept of invariance of API interface with respect to asynchronicity, but now I see that choice was confusing and nothing to do with AFIO. It would be a shame if AFIO were rejected due to that mistake in the tutorial, but it can't be undone for this review. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Andrey Semashev wrote:

On 24.08.2015 16:22, Niall Douglas wrote:

...

It would be a shame if AFIO were rejected due to that mistake in the tutorial, but it can't be undone for this review.

The problem is not the documentation mistake. The problem is that Boost.AFIO depends on another library that has not been reviewed or accepted. AFAIU, without that pre-requisite Boost.AFIO cannot be accepted. If my understanding is correct, this review should be cancelled (with the library being rejected as a result) and a review of Boost.Monad should be scheduled. That's my opinion.

I have to say that I find this choice of naming baffling. Why would you name a concrete type 'monad'? Makes no sense to me. Would not something like afio::result been better? In this way this is clearly a part of AFIO and no separate review would be necessary. If one insists on calling a type boost::monad, this is a serious claim over quite a territory and I agree with Andrey that this calls for a review of Boost.Monad.

On 24 August 2015 at 17:43, Niall Douglas <s_sourceforge@nedprod.com> wrote:

On 24 Aug 2015 at 19:21, Peter Dimov wrote:

...

I have to say that I find this choice of naming baffling. Why would you name a concrete type 'monad'? Makes no sense to me.

That was discussed on this very list in great depth and length. The choice of monad<T> was made to ensure people looked up the documentation for what it meant instead of making any assumptions. I think that aim worked out beautifully.

i'm sorry i missed the other thread, i've only been following boost-dev for a couple of months so sorry if i'm repeating what's already been said but this doesn't feel version user friendly. and although i get the intention if you're going to do that maybe you should make up a new word entirely. that way nobody knows what it means as opposed to this where some people may mistakenly think it means what monad is supposed to mean where here it doesn't quite (i think!).

...

Would not something like afio::result been better? In this way this is clearly a part of AFIO and no separate review would be necessary.

In Monad in addition to a monad<T> there is a result<T> and an option<T> in increasing order of lightweightness. They all specialise basic_monad<>.

...

If one insists on calling a type boost::monad, this is a serious claim over quite a territory and I agree with Andrey that this calls for a review of Boost.Monad.

It's coming in 2016. I am rewriting the AFIO engine first using Monad throughout, as that will debug Monad and put production ready finish on Monad. Then I'll have to write tutorials and docs and explain my design choices as I'm sure few will agree with them.

For reference, it's boost::monad::monad<T>, not boost::monad<T>. And Monad specifically disclaims trying to be a full fat Haskell type monad in C++, it is a lightweight C++-centric monad with almost zero overhead. In my own code to date I have been finding it *amazingly* useful, my productivity has jumped and my debugging and unit test writing time cut significantly. I genuinely suspect I will never write C++ code without using Monad or something very similar ever again.

i think the main issue i have over this is (unless totally mistaken) that boost::monad::monad<T> is not true to the definition of a monad. it is, like mentioned earlier, more akin to an expected/optional monad. so taking the word monad and the namespace boost::monad seems to be a bit scary when in the future they could be used for a totally generic full fat haskell type monad in c++ as you say. perhaps boost::afio::monad?

On 24 August 2015 at 18:21, Robert Ramey <ramey@rrsd.com> wrote:

On 8/24/15 9:59 AM, Sam Kellett wrote:

...

so taking the word monad and the namespace boost::monad seems to be a bit scary when in the future they could be used for a totally generic full fat haskell type monad in c++ as you say.

I have personal experience with this problem. As I mentioned before, when I made the boost serialization library. I included things like state_saver, singleton, etc in the boost namespace. No one objected through two reviews - but I eventually caught hell for this and ended up moving to the namespace boost::serialization. (boost was a rougher environment in those days). Seems to be an implicit but long running consensus that only those components which have passed formal review should be inserted directly into the boost namespace. I would agree with this idea.

I raised a few concerns about the "monad" "library" when it was first discussed on the list and never really bought into it as a boost library (as constituted) and don't think it should be included now. Though I'm skeptical of Niall's monad, I've got not complaint if Nail want's to make boost::afio::monad and later try to get it "promoted" to boost::monad. Doing this would demote Niall's monad to the status of implementation detail or private API and hence wouldn't irrevocably occupy any coveted territory. It would also make the review of AFIO much easier and more likely to pass. It would also be an easy change for Niall to (promise to) make.

that seems to be a nicer way to go about it. i've played with the idea in my head for the past half an hour and i think i would quite like boost::monad to be maybe a few customisation points: bind, return, fail, even >> and >>= (although i would imagine associativity and precedence would get in the way of the last two) that your own monadic-like classes could specialise and then use as first class monads. this is maybe getting a bit off topic now so i'll stop.

On 24 August 2015 at 21:18, Niall Douglas <s_sourceforge@nedprod.com> wrote:

On 24 Aug 2015 at 18:59, Sam Kellett wrote:

...

i've played with the idea in my head for the past half an hour and i think i would quite like boost::monad to be maybe a few customisation points: bind, return, fail, even >> and >>= (although i would imagine associativity and precedence would get in the way of the last two) that your own monadic-like classes could specialise and then use as first class monads. this is maybe getting a bit off topic now so i'll stop.

Monad has all those items you mention. See https://ned14.github.io/boost.monad/singletonboost_1_1monad_1_1basic__ monad.html under the section "Functional programming extensions (optional)".

You also get to customise them to your heart's content via the policy class infrastructure. After all, basic_future<> which subclasses basic_monad<> implements binds as continuations.

that does look good, don't get me wrong. but would i be able to use it to, say, provide monadic options to std::list? or io ala haskell? because if not i would wonder why not simply because it's called 'monad'. what if i wanted to use the >> operator on a boost::optional, could i do that with your monad type? i'm sure i'm being dense, but i couldn't find an example of the policy infrastructure so i couldn't answer those questions myself.

On 08/24/2015 10:21 AM, Robert Ramey wrote:

On 8/24/15 9:59 AM, Sam Kellett wrote:

...

so taking the word monad and the namespace boost::monad seems to be a bit scary when in the future they could be used for a totally generic full fat haskell type monad in c++ as you say.

<snip>

I raised a few concerns about the "monad" "library" when it was first discussed on the list and never really bought into it as a boost library (as constituted) and don't think it should be included now. Though I'm skeptical of Niall's monad, I've got not complaint if Nail want's to make boost::afio::monad and later try to get it "promoted" to boost::monad. Doing this would demote Niall's monad to the status of implementation detail or private API and hence wouldn't irrevocably occupy any coveted territory. It would also make the review of AFIO much easier and more likely to pass. It would also be an easy change for Niall to (promise to) make.

Hi Niall - There have been several suggestions (implicit and explicit) to move this type into the boost::afio namespace, but I haven' seen a response from you. Have I just missed it? I'm afraid that AFIO isn't able to be reviewed because there are so many questions about other someday-to-be libraries in the boost namespace. What are your thoughts? Thanks - michael -- Michael Caisse ciere consulting ciere.com

On 28 Aug 2015 at 22:30, Michael Caisse wrote:

There have been several suggestions (implicit and explicit) to move this type into the boost::afio namespace, but I haven' seen a response from you. Have I just missed it?

It's more that the suggestion is irrelevant with respect to the library. The code base as presented for review already exclusively uses boost::afio::monad<T>. It is only the just-added workshop tutorial which uses monad directly which made people think monad is not an internal library, and which I now realise was a mistake due to bad optics.

I'm afraid that AFIO isn't able to be reviewed because there are so many questions about other someday-to-be libraries in the boost namespace.

What are your thoughts?

I think many if not most other reviewers have not found the namespace layout nor dependent libraries an obstacle to making good reviews. I'll put it another way: the discussion to date has covered pretty much every thing I expected would be found of interest during a Boost review. This is why I felt AFIO was ready for review - the library had progressed to a point where I alone could go no further without a review. Even if rejected, these last nine days or so have been highly valuable. They have confirmed that my understanding of the tradeoffs in the design matches that of the community's. A big shock was how bad my documentation is, and how I really do need to go a lot slower in the tutorial than I have. My CppCon presentation should be much the better for this review and I am incorporating the name changes and feedback into the presentation materials. Renaming monad to outcome is literally a five minute job, as is rewriting how monad is bound into to code examples to `using BOOST_AFIO_V2_NAMESPACE::outcome`. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 29 Aug 2015 at 7:57, Robert Ramey wrote:

...

...

There have been several suggestions (implicit and explicit) to move this type into the boost::afio namespace, but I haven' seen a response from you. Have I just missed it?

It's more that the suggestion is irrelevant with respect to the library.

It is irrelevant to the operation of the library. But it's certainly no irrelevant with respect to the acceptance of the library. This is exactly the point I was making.

For sure. But please be aware that the use of Monad is more presentational than anything. I foolishly made Monad look like not an internal library in the tutorial, and I have been paying for it since. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 29 Aug 2015 at 9:21, Michael Caisse wrote:

...

...

There have been several suggestions (implicit and explicit) to move this type into the boost::afio namespace, but I haven' seen a response from you. Have I just missed it?

It's more that the suggestion is irrelevant with respect to the library. The code base as presented for review already exclusively uses boost::afio::monad<T>. It is only the just-added workshop tutorial which uses monad directly which made people think monad is not an internal library, and which I now realise was a mistake due to bad optics.

I think you may have missed parts of reviews. A few influential reviewers [1] have indicated that having components such as Monad and APBind is a non-starter for a review or an immediate "no" vote (Vicente [2] and Robert are just two on the user list... others on the dev list).

You have put a great deal of effort into this library and I would like to see the chances of acceptance during the review not hindered by non-essential choices. The bottom line is that implementations in the boost::afio namespace will have different scrutiny than automatic promotion to the boost namespace. Start with these concepts in the boost::afio namespace and then at a later date, request a review to promote them to the boost namespace as full-fledged libraries.

I appreciate all of what you just said. But please listen to what I just said: AFIO's use of these two dependent libraries is entirely internal in the implementation and header code. Where I came unstuck is that the documentation treats those two libraries as if there are not internal implementation libraries, and the documentation both refers to them by name e.g. Boost.Monad, Boost.APIBind and the code examples use them directly instead of via their boost::afio::* mapping. This makes them appear far more important than they are to the end users of the presented library.

I don't think it is worth risking the review outcome on trying to get multiple libraries into the boost namespace.

I think that most reviewers, once they actually looked at the code, realised that the use of Boost.Monad and Boost.APIBind is a purely presentational issue caused by a bad choice in the documentation. As I have said on many occasions now, I realise that was a mistake, but as I cannot change what has been presented for review I am stuck with it. For the record, I estimate that all mention of APIBind and Monad can be completely eliminated from the AFIO documentation and externally facing API in about two days of work. They are totally incidental to AFIO the library presented here for review. This is why I said I would be upset with reviewers who refuse point blank to review this library purely on a documentation mistake if they then later on find severe design flaws in the library after I have invested an additional 500 hours in the lightweight futures refactor based on the presented API and design (which is my current estimate, and I am usually within 10% of my estimates when estimating on my own code). I think anyone would react similarly. However after the many threads of discussion this week regarding the design tradeoffs in the AFIO API and design, I am confident that in any future review (if needed) that nothing not discovered in this review will come up. In this regard this review has been highly successful, and very valuable, irrespective of eventual outcome. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 24 Aug 2015 at 17:59, Sam Kellett wrote:

...

...

I have to say that I find this choice of naming baffling. Why would you name a concrete type 'monad'? Makes no sense to me.

That was discussed on this very list in great depth and length. The choice of monad<T> was made to ensure people looked up the documentation for what it meant instead of making any assumptions. I think that aim worked out beautifully.

i'm sorry i missed the other thread, i've only been following boost-dev for a couple of months so sorry if i'm repeating what's already been said but this doesn't feel version user friendly. and although i get the intention if you're going to do that maybe you should make up a new word entirely. that way nobody knows what it means as opposed to this where some people may mistakenly think it means what monad is supposed to mean where here it doesn't quite (i think!).

Naming of fundamental C++ primitive types is always guaranteed to lead to never ending debate here (and at WG21 for that matter). You can check the history of this list for the discussion which has plenty of names suggested. I chose monad<T>, result<T> and option<T> out of the many, many options suggested. Most people will disagree with those choices, and no matter what names I choose it would still be the case that most people will disagree with anything I chose. Such is naming C++ fundamental objects.

i think the main issue i have over this is (unless totally mistaken) that boost::monad::monad<T> is not true to the definition of a monad.

This is incorrect. It does have the full power of a monad if you use exception throws as an erased type transport. Which is of course insane and has awful performance, so you shouldn't do that. You're right in the ordinary sane use case it's not a Haskell monad. It's a C++ monad designed for C++ programmers writing in C++.

it is, like mentioned earlier, more akin to an expected/optional monad. so taking the word monad and the namespace boost::monad seems to be a bit scary when in the future they could be used for a totally generic full fat haskell type monad in c++ as you say.

Personally speaking I would doubt there will ever be a full fat haskell type monad in common C++ programming. The cost benefit isn't there for C++ 14 or C++ 1z as it is currently defined. There has been a big trend towards simpler, less compile time heavy library primitives in recent years, and until I am proven wrong that rules out full fat Haskell type monads in C++. Maybe some future C++ language edition with much improved tooling might make it work, but that's a 2020s thing minimum. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 25 Aug 2015 at 0:02, Giovanni Piero Deretta wrote:

...

It's a C++ monad designed for C++ programmers writing in C++.

Niall, you are missing the point: the reason that your monad can't be the true definition of a Monad is that no type can.

No, that's exactly why I chose that name for the library. I absolutely get the point.

That there is no such a thing as Monad type in Haskell at all; there is a type *class* named Monad. In Haskell there are many concrete types that are instances of the Monad type class (list, Maybe, IO). Other languages are similar: I believe you are familiar with rust, in that language Monad would be a trait.

Similarly, in C++ you can have multiple types[1] (option, future, etc...) that model the Monad concept (Haskell type classes loosely map to C++ concepts), but calling any of them (or a template type) a Monad, makes at much sense as having a class named RegularType.

Again, exactly why I chose it: it makes no sense.

In C++, you could argue that 'monad' would be a decent name for type erased wrapper over any type that model the Monad concept, but I do not believe that your monad type is that.

Now, afio::monad, while not great, at least suggests that we are talking about the Monad of Asynchronous File IO [2].

C++ is not a functional programming language and never will be, and I even think nor should it be. I certainly don't get anything like as worked up about naming as people are getting over this. Who cares if it doesn't pollute namespace, is short to type and isn't going to get confused with something else? That's why value<T> or holder<T> or return_<T> all got ruled out. They will confuse. monad<T> is nonsense name, but it stands out nicely in code and let's the eye quickly check that your logic is correct. As monad<T>'s main use case for me so far at least is exception safety management, it NEEDS to stand out from surrounding code as a return type from a function. But let me clear, if anyone can suggest a similarly short, instantly recognisable, uniquely standout name for monad<T>, I'm all ears. For example, I have toyed with donkey<T> instead of monad<T>. Also a nonsense name, but it does the donkey work of exception safety for you (with monadic programming extensions) and it has all the features of standing out from surrounding code, being quick to type and so on. Would people prefer Boost.Monad => Boost.Donkey, and monad<T> => donkey<T>? The reason I didn't choose donkey<T> is because I thought people wouldn't take a Boost.Donkey library seriously, and that would be a shame as I'm finding it an enormous productivity aid. It also doesn't fit to have a lightweight fast Concurrency TS implementation in a library called Boost.Donkey. But I also don't think the naming war on this library can be won. If Boost.Donkey will stop the war and make people vote for acceptance, I'll take Boost.Donkey over Boost.Monad any day. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Andrey Semashev Sent: 25 August 2015 15:12 To: boost@lists.boost.org Subject: Re: [boost] Monad (was: Re: [Boost-users] [afio] Formal review of Boost.AFIO)

On Tue, Aug 25, 2015 at 3:44 PM, Niall Douglas <s_sourceforge@nedprod.com> wrote:

...

On 25 Aug 2015 at 0:02, Giovanni Piero Deretta wrote:

...

...

It's a C++ monad designed for C++ programmers writing in C++.

Niall, you are missing the point: the reason that your monad can't be the true definition of a Monad is that no type can.

No, that's exactly why I chose that name for the library. I absolutely get the point.

...

That there is no such a thing as Monad type in Haskell at all; there is a type *class* named Monad. In Haskell there are many concrete types that are instances of the Monad type class (list, Maybe, IO). Other languages are similar: I believe you are familiar with rust, in that language Monad would be a trait.

Similarly, in C++ you can have multiple types[1] (option, future, etc...) that model the Monad concept (Haskell type classes loosely map to C++ concepts), but calling any of them (or a template type) a Monad, makes at much sense as having a class named RegularType.

Again, exactly why I chose it: it makes no sense.

...

In C++, you could argue that 'monad' would be a decent name for type erased wrapper over any type that model the Monad concept, but I do not believe that your monad type is that.

Now, afio::monad, while not great, at least suggests that we are talking about the Monad of Asynchronous File IO [2].

C++ is not a functional programming language and never will be, and I even think nor should it be.

I certainly don't get anything like as worked up about naming as people are getting over this. Who cares if it doesn't pollute namespace, is short to type and isn't going to get confused with something else?

That's why value<T> or holder<T> or return_<T> all got ruled out. They will confuse. monad<T> is nonsense name, but it stands out nicely in code and let's the eye quickly check that your logic is correct.

As monad<T>'s main use case for me so far at least is exception safety management, it NEEDS to stand out from surrounding code as a return type from a function.

But let me clear, if anyone can suggest a similarly short, instantly recognisable, uniquely standout name for monad<T>, I'm all ears.

For example, I have toyed with donkey<T> instead of monad<T>. Also a nonsense name, but it does the donkey work of exception safety for you (with monadic programming extensions) and it has all the features of standing out from surrounding code, being quick to type and so on.

Would people prefer Boost.Monad => Boost.Donkey, and monad<T> => donkey<T>?

The reason I didn't choose donkey<T> is because I thought people wouldn't take a Boost.Donkey library seriously, and that would be a shame as I'm finding it an enormous productivity aid. It also doesn't fit to have a lightweight fast Concurrency TS implementation in a library called Boost.Donkey.

But I also don't think the naming war on this library can be won. If Boost.Donkey will stop the war and make people vote for acceptance, I'll take Boost.Donkey over Boost.Monad any day.

I don't know Haskell or what monad means there, so to me monad<> does not stand out, apart from that I don't know what it means. That's probably because there are lots of things that I don't know or understand, so there's nothing special about it. :)

However, reading your argument I don't see the rationale behind choosing the name which overloads what apparently is a well understood term in Haskell. If the meaning of your monad<> differs from that well established understanding then this will certainly lead to confusion and endless questions and notes like 'that C++ monad is not the monad you know'. To me this indicates that the naming choice is poor.

So my point is name it donkey, if you feel this name reflects the component purpose well (I doubt it). Pick another name that is not overloaded yet or has close semantics. Only keep monad if it doesn't cause confusion.

Absolutely right. Anything but monad! It *does* cause confusion. Think again - and donkey won't do either ;-) Paul --- Paul A. Bristow Prizet Farmhouse Kendal UK LA8 8AB +44 (0) 1539 561830

On 25 Aug 2015 3:48 pm, "Niall Douglas" <s_sourceforge@nedprod.com> wrote:

On 25 Aug 2015 at 17:11, Andrey Semashev wrote:

...

So my point is name it donkey, if you feel this name reflects the component purpose well (I doubt it). Pick another name that is not overloaded yet or has close semantics. Only keep monad if it doesn't cause confusion.

monad<T> still provides all the essential monadic programming operations, and it behaves mostly like a fixed function monad.

"Socrates is a human, All humans are mortal. Therefore all humans are Socrates". -- gpd

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Niall Douglas Sent: 25 August 2015 13:44 To: boost@lists.boost.org Subject: Re: [boost] Monad (was: Re: [Boost-users] [afio] Formal review of Boost.AFIO)

On 25 Aug 2015 at 0:02, Giovanni Piero Deretta wrote:

...

...

It's a C++ monad designed for C++ programmers writing in C++.

Niall, you are missing the point: the reason that your monad can't be the true definition of a Monad is that no type can.

No, that's exactly why I chose that name for the library. I absolutely get the point.

...

That there is no such a thing as Monad type in Haskell at all; there is a type *class* named Monad. In Haskell there are many concrete types that are instances of the Monad type class (list, Maybe, IO). Other languages are similar: I believe you are familiar with rust, in that language Monad would be a trait.

Similarly, in C++ you can have multiple types[1] (option, future, etc...) that model the Monad concept (Haskell type classes loosely map to C++ concepts), but calling any of them (or a template type) a Monad, makes at much sense as having a class named RegularType.

Again, exactly why I chose it: it makes no sense.

...

In C++, you could argue that 'monad' would be a decent name for type erased wrapper over any type that model the Monad concept, but I do not believe that your monad type is that.

Now, afio::monad, while not great, at least suggests that we are talking about the Monad of Asynchronous File IO [2].

C++ is not a functional programming language and never will be, and I even think nor should it be.

I certainly don't get anything like as worked up about naming as people are getting over this. Who cares if it doesn't pollute namespace, is short to type and isn't going to get confused with something else?

That's why value<T> or holder<T> or return_<T> all got ruled out. They will confuse. monad<T> is nonsense name, but it stands out nicely in code and let's the eye quickly check that your logic is correct.

As monad<T>'s main use case for me so far at least is exception safety management, it NEEDS to stand out from surrounding code as a return type from a function.

But let me clear, if anyone can suggest a similarly short, instantly recognisable, uniquely standout name for monad<T>, I'm all ears.

OK - here's a concrete suggestion. If I've understood correctly from this Predefined basic_monad implementations: <dl> <dt>`monad<R>`</dt> <dd>Can hold a fixed variant list of empty, a type `R`, a lightweight `std::error_code` or a heavier `std::exception_ptr` at a space cost of `max(24, sizeof(R)+8)`. This corresponds to `tribool::unknown`, `tribool::true_`, `tribool::false_` and `tribool::false_` respectively.</dd> <dt>`result<R>`</dt> <dd>Can hold a fixed variant list of empty, a type `R` or a lightweight `std::error_code` at a space cost of `max(24, sizeof(R)+8)`. This corresponds to `tribool::unknown`, `tribool::true_` and `tribool::false_` respectively. This specialisation looks deliberately like Rust's `Result<T>`.</dd> <dt>`option<R>`</dt> <dd>Can hold a fixed variant list of empty or a type `R` at a space cost of `sizeof(value_storage<R>)` which is usually `sizeof(R)+8`, but may be smaller if `value_storage<R>` is specialised e.g. `sizeof(option<char>)` is just two bytes, and `sizeof(option<bool>)` is just one byte (see note about `option<bool>` below). This corresponds to `tribool::unknown` and `tribool::true_` respectively. This specialisation looks deliberately like Rust's `Option<T>`.</dd> </dl> these are all *outcomes* - of varying types of varying and unspecified Thingy. So rather than Thingy or Donkey, how about: Outcome<R> Paul --- Paul A. Bristow Prizet Farmhouse Kendal UK LA8 8AB +44 (0) 1539 561830

On 25 August 2015 at 19:50, Niall Douglas <s_sourceforge@nedprod.com> wrote:

On 25 Aug 2015 at 17:49, Paul A. Bristow wrote:

...

...

But let me clear, if anyone can suggest a similarly short, instantly recognisable, uniquely standout name for monad<T>, I'm all ears.

OK - here's a concrete suggestion. [snip] these are all *outcomes* - of varying types of varying and unspecified Thingy.

So rather than Thingy or Donkey, how about:

Outcome<R>

I like Boost.Outcome and boost::outcome::basic_outcome<Policy>.

Do these make sense however:

outcome<T>: Can be empty/T/error_code/exception_ptr.

result<T>: Can be empty/T/error_code.

option<T>: Can be empty/T.

If people like this, I can change Boost.Monad to Boost.Outcome very easily.

Niall

A potential user POV: 1. I agree with Giovanni Piero Deretta and Andrey Semashev, to me (even without having done any concrete Haskell but after having learned what is a monad in the abstract/mathematical sense), I see naming a _type_ "monad" as very obscure as it does not tell me what it does, just part of how it works. It's exactly like if std::vector was named std::container as already mentioned and that's exactly what I was thinking when reading the previous discussions on the subject. I didn't say anything because when talking about monads it's very easy to endup with a confusing discussion. 2. I disagree with the naming argument you gave so far, mainly because it seems that only you believe in it (but I might be wrong) and your arguments don't rely on logic, observation or popularity (but my pov is less important than specialists around here, and naming is hard etc.) 3. I have no problem with a _library_ that would be named Boost.Monad if it provides different types using this useful ...err..."pattern", like optional, future and the thing you are trying to name at the moment. It's with a type named monad that it is very confusing. 4. 'outcome' seems reasonable, it's a little vague but there was no more specific propositons so far and it seems to suggest the way it is used. It feels like a more general expected. There might be a more apropriate name or a neologism to be made though. Hope it helps. Joël Lamotte

On 8/26/2015 7:57 AM, Glen Fernandes wrote:

Paul wrote:

...

these are all *outcomes* - of varying types of varying and unspecified Thingy.

Niall wrote:

...

I like Boost.Outcome and boost::outcome::basic_outcome<Policy>. Do these make sense however: outcome<T>: Can be empty/T/error_code/exception_ptr. result<T>: Can be empty/T/error_code. option<T>: Can be empty/T.

Isn't "Outcome" no less generic and unspecific as "Result"? All of Boost.Monad, Boost.Outcome, or Boost.Result, seem like terrible names to me.

We already name specific result types for the kind of result they are (I'm glad boost::optional is not boost::result). Why not name monad types for the specific kind of monad they are?

Unless this "Monad" library isn't about offering a concrete type called "boost::monad" but instead is just a library to enable someone write their own monad types.

A link to this thread should appear next to bikeshedding in the dictionary. At this point I'd change the name to Outcome just in the hopes that all the people that have made their opinions known about the name of this utility could move their focus to the library under review. For my 2cents if I can live with names like Qi Hana Spirit Wave and std::vector I think I can manage with monad as is.

On 27 Aug 2015 3:46 am, "Niall Douglas" <s_sourceforge@nedprod.com> wrote:

On 26 Aug 2015 at 5:57, Glen Fernandes wrote:

...

Paul wrote:

...

these are all *outcomes* - of varying types of varying and unspecified Thingy.

Niall wrote:

...

I like Boost.Outcome and boost::outcome::basic_outcome<Policy>. Do these make sense however: outcome<T>: Can be empty/T/error_code/exception_ptr. result<T>: Can be empty/T/error_code. option<T>: Can be empty/T.

Isn't "Outcome" no less generic and unspecific as "Result"? All of Boost.Monad, Boost.Outcome, or Boost.Result, seem like terrible names

to me.

Choice between the pan and the fire.

...

We already name specific result types for the kind of result they are

...

glad boost::optional is not boost::result). Why not name monad types for the specific kind of monad they are?

Unless this "Monad" library isn't about offering a concrete type called "boost::monad" but instead is just a library to enable someone write

(I'm their

...

own monad types.

That's exactly what it does. It's a framework for creating bespoke future and monad types.

A Boost.Monad library that would provide a monad_adaptor a la Boost.Iterator could indeed be useful, but that doesn't seem to be the case here.

You define your policy implementations classes and feed them to basic_monad<Policy> or basic_future<Policy> as needed. The library itself provides three monad specialisations and six future specialisations by stamping out the nine policies for each of those.

Unfortunately the documentation for the policy is literally an empty line. I have looked at the detail/monad_policy.hpp file, but beyond customising the storage, I can't see any hook to customise the behaviour. Could you show us a custom policy implementation for a list monad? -- gpd

On 28 Aug 2015 2:14 am, "Niall Douglas" <s_sourceforge@nedprod.com> wrote:

On 27 Aug 2015 at 7:43, Giovanni Piero Deretta wrote:

...

...

You define your policy implementations classes and feed them to basic_monad<Policy> or basic_future<Policy> as needed. The library itself provides three monad specialisations and six future specialisations by stamping out the nine policies for each of those.

Unfortunately the documentation for the policy is literally an empty

line.

...

I have looked at the detail/monad_policy.hpp file, but beyond customising the storage, I can't see any hook to customise the behaviour.

Could you show us a custom policy implementation for a list monad?

As I mentioned earlier, it's not a full monadic programming solution by any stretch. Think of it as a value transporter with monadic extensions, very similar to the Rust Monad.

... You just claimed that: "It's a framework for creating bespoke future and monad types." Also what's the rust monad you are referring to? -- gpd

On 29 Aug 2015 3:07 am, "Niall Douglas" <s_sourceforge@nedprod.com> wrote:

On 28 Aug 2015 at 12:02, Giovanni Piero Deretta wrote:

...

...

...

Could you show us a custom policy implementation for a list monad?

As I mentioned earlier, it's not a full monadic programming solution by any stretch. Think of it as a value transporter with monadic extensions, very similar to the Rust Monad.

... You just claimed that:

"It's a framework for creating bespoke future and monad types."

I'm not seeing what the problem is here. It is as I described.

How can it be both a "framework for creating bespoke monads" and at the same time " not a full monadic solution". I either can create a monad - any monad - with it or not.

...

Also what's the rust monad you are referring to?

Search google for the Rust programming language and Option<T> and Result<T>.

Those aren't in any way rust monad. They are specific implementations of a monad concept, which incidentally can't be encoded yet as a rust Trait as Rust doesn't have yet a type system powerful enough. -- gpd

On 27 Aug 2015 at 11:40, Thomas Heller wrote:

...

The idea is that no one ever need implement the Concurrency TS themselves ever again. Just pick up a copy of Boost.Monad/Outcome. Write your policy class for your custom variant. Profit.

Why not just call what it is? Future or FutureFactory or LightweightFuture?

I'm happy with all three of those names too. But I suspect anything which contains the word "Future" will bring even more attacks by people who have not looked at the design nor the code and just want any efforts by me killed off period. Hence I have avoided those names to date. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Niall Douglas wrote:

The idea is that no one ever need implement the Concurrency TS themselves ever again. Just pick up a copy of Boost.Monad/Outcome. Write your policy class for your custom variant. Profit.

Is anyone besides Boost or standard library vendors actually interested in doing this? Standard library implementors are not going to take a dependency on a Boost library. For example: Are there many (or any) Filesystem TS implementations that people care about outside of standard library implementations and Boost? I just can't imagine someone, say, proposing Boost.Path that provides a boost::basic_path that can be used to implement std::experimental::filesystem::path and expecting anyone to care about it. Anyone implementing the TS will just implement std::experimental::filesystem::path themselves and not take a dependency on boost::basic_path to do it. What makes your basic_future so different to the hypothetical basic_path? Glen -- View this message in context: http://boost.2283326.n4.nabble.com/afio-Formal-review-of-Boost-AFIO-tp467911... Sent from the Boost - Dev mailing list archive at Nabble.com.

On 27 Aug 2015 at 5:05, Glen Fernandes wrote:

Niall Douglas wrote:

...

The idea is that no one ever need implement the Concurrency TS themselves ever again. Just pick up a copy of Boost.Monad/Outcome. Write your policy class for your custom variant. Profit.

Is anyone besides Boost or standard library vendors actually interested in doing this? Standard library implementors are not going to take a dependency on a Boost library.

For example: Are there many (or any) Filesystem TS implementations that people care about outside of standard library implementations and Boost?

I just can't imagine someone, say, proposing Boost.Path that provides a boost::basic_path that can be used to implement std::experimental::filesystem::path and expecting anyone to care about it. Anyone implementing the TS will just implement std::experimental::filesystem::path themselves and not take a dependency on boost::basic_path to do it.

What makes your basic_future so different to the hypothetical basic_path?

A basic_path is probably not going to attract much need for customisation without losing interoperability with other basic_path customisations. basic_future lets you implement any custom future type of your heart's desire without losing wait composure with other custom basic_futures. Also people do have a real need for custom futures. A big bone Chris picks with the Concurrency model in the Concurrency TS is its reliance on futures which he views (rightly) as being too chunky for low latency socket i/o. My lightweight futures don't address all of Chris' concerns, but they are a whole ton load better for low latency work than standard futures. If someone wanted even lower latency than my default implementation and wanted to give up standards conformance, they certainly can do that. My default implementation has to match the TS, and there is a definite overhead to that. I also suspect they'll be a real boon for C++ 1z coroutines given how that is designed, but testing that is many months away yet. If they are a real boon, I don't doubt Gor will be very interested. Finally, you may not be aware that my lightweight futures are dependent on a system future implementation. They do not replace system nor Boost futures, they act as a layer implemented on top dropping down onto the system futures when unavoidable. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Le 25/08/15 14:44, Niall Douglas a écrit :

On 25 Aug 2015 at 0:02, Giovanni Piero Deretta wrote:

...

...

It's a C++ monad designed for C++ programmers writing in C++. Niall, you are missing the point: the reason that your monad can't be the true definition of a Monad is that no type can. No, that's exactly why I chose that name for the library. I absolutely get the point. Clearly you are missing the point. You class monad could be seen as a model of a Monad (but don't as it doesn't defines a generic unit/return_/make function) as others can. I would say that your monad class is something close to an instantiation of a variadic expected

expected<T, E1, ... En> After some request I'm thinking in extending expected<T,E> to variadic templates. expected<T, E1, ... En> would be something like variant<T, unexpected<E1>, ... unexpected<En>> Your class would be something like expected<T, empty, error_code, exception_ptr> with some specific interfaces. Am I missing something? P.S. Note that you can already have expected<T, variant<empty, error_code, exception_ptr>> The variadic version would be be more efficient and with a more friendly syntax.

...

That there is no such a thing as Monad type in Haskell at all; there is a type *class* named Monad. In Haskell there are many concrete types that are instances of the Monad type class (list, Maybe, IO). Other languages are similar: I believe you are familiar with rust, in that language Monad would be a trait.

Similarly, in C++ you can have multiple types[1] (option, future, etc...) that model the Monad concept (Haskell type classes loosely map to C++ concepts), but calling any of them (or a template type) a Monad, makes at much sense as having a class named RegularType. Again, exactly why I chose it: it makes no sense.

...

In C++, you could argue that 'monad' would be a decent name for type erased wrapper over any type that model the Monad concept, but I do not believe that your monad type is that.

Now, afio::monad, while not great, at least suggests that we are talking about the Monad of Asynchronous File IO [2]. C++ is not a functional programming language and never will be, and I even think nor should it be.

Hmm, I prefer to don't replay here, as I don't know what you understand by a functional programming and I can not divine what C++ will be in the future. I can not understand why do you think it shouldn't be one but I respect your opinion however.

I certainly don't get anything like as worked up about naming as people are getting over this. Who cares if it doesn't pollute namespace, is short to type and isn't going to get confused with something else?

because naming is important.

That's why value<T> or holder<T> or return_<T> all got ruled out. They will confuse. monad<T> is nonsense name, but it stands out nicely in code and let's the eye quickly check that your logic is correct.

It seems that not too much people agree with your point of view. Maybe it is time you consider what other are saying.

As monad<T>'s main use case for me so far at least is exception safety management, it NEEDS to stand out from surrounding code as a return type from a function.

But let me clear, if anyone can suggest a similarly short, instantly recognisable, uniquely standout name for monad<T>, I'm all ears.

If you don't like expected<T>, I will suggest you to name it value_or_error<T>

For example, I have toyed with donkey<T> instead of monad<T>. Also a nonsense name, but it does the donkey work of exception safety for you (with monadic programming extensions) and it has all the features of standing out from surrounding code, being quick to type and so on.

Would people prefer Boost.Monad => Boost.Donkey, and monad<T> => donkey<T>?

The reason I didn't choose donkey<T> is because I thought people wouldn't take a Boost.Donkey library seriously, and that would be a shame as I'm finding it an enormous productivity aid. It also doesn't fit to have a lightweight fast Concurrency TS implementation in a library called Boost.Donkey.

But I also don't think the naming war on this library can be won. If Boost.Donkey will stop the war and make people vote for acceptance, I'll take Boost.Donkey over Boost.Monad any day.

As you can see choosing the good names has its importance. With a good name we would be reviewing the concrete features already. Best, Vicente

On 25 Aug 2015 at 0:41, Sam Kellett wrote:

...

Personally speaking I would doubt there will ever be a full fat haskell type monad in common C++ programming. The cost benefit isn't there for C++ 14 or C++ 1z as it is currently defined. There has been a big trend towards simpler, less compile time heavy library primitives in recent years, and until I am proven wrong that rules out full fat Haskell type monads in C++.

Maybe some future C++ language edition with much improved tooling might make it work, but that's a 2020s thing minimum.

i'm not sure i see where the complexity comes from.

Anyone can implement a monadic C++ programming library, and there are quite a few excellent implementations on github. The complexity comes in getting a large code base which makes extensive use of monads to compile quickly and ideally not bloat the binary and be easy to understand in the debugger. I wanted just enough monad for my needs, no more. Call it monad-lite if you want. Monad compiles very quickly indeed in debug, adds virtually no overhead to either runtime or binary size and is very easy to understand in the debugger. Those tick the boxes I was looking for. I think you have different boxes you want to tick. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Le 24/08/15 03:29, Niall Douglas a écrit :
> On 24 Aug 2015 at 0:46, Vicente J. Botet Escriba wrote:
>
>>>> I believe that I need something more about monad<T> than what I have
>>>> found in the documentation.
>>> If there is anything missing, please do say in detail.
>>>
>> Is this link part of the review? I don't believe, so could you point me
>> where the monad class is documented in AFIO? If it is not documented,
>> why it is used so broadly.
> It is used in the tutorial because as we progress through each stage
> of increasing complexity, you can see that the data_store class
> member functions lookup() and write() have either a monad<T> or a
> shared_future<T> return type depending on whether the API is
> synchronous (monad<T>) or asynchronous (shared_future<T>).
>
> I was trying to keep the interface class for each stage as identical
> as possible, so monad<T> was a natural fit to make the STL iostreams
> example look exactly like the AFIO asynchronous example. I suppose
> what you're saying here is that the tutorial should not be using
> monad<T> because monad<T> is not documented in AFIO.
The intention was not clear to me. I believe that this concern belongs 
more to a Monad library.
> I can easily rewrite the tutorial code to not use monad<T>, it's no
> problem.
>
>>> If you want to pin a specific ABI which won't break in the future,
>>> you bind afio like this:
>>>
>>> namespace afio = BOOST_AFIO_V2_NAMESPACE;
>> Where can I find the description of this macro on the reference
>> documentation?
> https://boostgsoc13.github.io/boost.afio/doc/html/afio/quickstart/workshop/naive.html
I asked for the reference documentation, not the tutorial.
>> We have two options:
>> * we wait to review AFIO once Monad is reviewed in Boost
>> * You include in AFIO, whatever you consider is need from Monad and only
>> then we review AFIO.
> I can remove monad<T> from all the tutorial code. Or I can include
> documentation of monad<T> in AFIO. Which would you prefer?
>
>
I believe that your Monad library would have a lot of issues, so for 
your interest I believe it is better to remove it from AFIO.

Vicente

On 24-Aug-15 1:17 AM, Niall Douglas wrote:

...

It will be easier to know the scope of the review and what is left for the near future. Is the monad part of the review?

I am happy to accept bug reports on monad<T>. But I don't think the AFIO review is reviewing monad<T>, nor do you need to.

I mean, if you like the design of boost::shared_future<T>, then by definition you like the design of monad<T>. I would have said monad<T> is therefore uncontroversial, as it's already in Boost.Thread and is very well understood (especially by you Vicente!).

Personally, I have concerns about the naming. E.g. if one reads http://bartoszmilewski.com/2011/01/09/monads-for-the-curious-programmer-part... one understands that monad is a fairly general concept, with several implementation in Haskell, and the "Monad" class has multiple implementations: https://hackage.haskell.org/package/base-4.8.1.0/docs/Control-Monad.html The monad<T> you are proposing seems related to Haskell 'Maybe', but not very much to anything else. I would suggest that a different name be used. It would certainly be unfortunately if a controversial name is necessary part of using an accepted library. - Volodya

On 24 Aug 2015 at 7:37, Andreas Schäfer wrote:

...

Correct. I summarise everything you need to know about monad<T> on the first page of the tutorial https://boostgsoc13.github.io/boost.afio/doc/html/afio/quickstart/work shop/naive.html.

Which links to [1], a link that yields a 404. Via [2] and [3] I found [4]. Is that the current documentation of Monad?

That's all the formal documentation I've written to date yes.

In "Complexity guarantees" some of the minimum complexities exceed their maximum counterparts (e.g. "51 opcodes <= Value transport <= 32 opcodes"). What's that supposed to mean? What is the rationale behind citing minimum complexities? And why do you measure opcodes instead of, say, time?

Monad is designed to be as absolutely as lightweight as possible, and is per-commit CI tested to ensure it generates no more than X opcodes per operation. Traditional complexities such as O(N) make no sense for Monad. No operation it does isn't constant time. You're really testing how friendly Monad is to the compiler optimiser (which is very friendly).

Since you mentioned monads were basically identical: why don't you just use std::future?

AFIO has never been able to return a plain std::future because of the lack of standardised wait composure in current C++ i.e. when_all(futures ...). It returns an afio::future<> which internally keeps a unique integer which looks up stored continuations in an unordered_map. This let AFIO implement continuations on top of std::future. Monad contains a full Concurrency TS implementation, and works everywhere in any configuration and currently with far lower overheads than any existing STL implementation by a big margin (about 50%). That means AFIO can drop the internal unordered_map, and become a pure Concurrency TS implementation with no workarounds and more importantly, no central locking at all. That will eliminate the last remaining synchronisation in AFIO, and it will be a pure wait free solution. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 25 Aug 2015 at 8:26, Andreas Schäfer wrote:

...

Monad is designed to be as absolutely as lightweight as possible, and is per-commit CI tested to ensure it generates no more than X opcodes per operation.

Traditional complexities such as O(N) make no sense for Monad. No operation it does isn't constant time. You're really testing how friendly Monad is to the compiler optimiser (which is very friendly).

OK, but how is the number of opcodes relevant in any practical setting? I for one expect the compiler to generate fast code. And if that means that one horribly slow instruction gets replaces by 10 fast ones, then so be it. I'd suggest to set up performance tests which measure time or throughput for sets of typical workloads.

I have those too, of course. Have a look at https://raw.githubusercontent.com/ned14/boost.monad/master/Readme.md, bottom of the document.

I'm still unsure what "51 opcodes <= Value transport <= 32 opcodes" is supposed to mean.

Min/max code bloat for a monad<int> or future<int>.

...

...

Since you mentioned monads were basically identical: why don't you just use std::future?

AFIO has never been able to return a plain std::future because of the lack of standardised wait composure in current C++ i.e. when_all(futures ...). It returns an afio::future<> which internally keeps a unique integer which looks up stored continuations in an unordered_map. This let AFIO implement continuations on top of std::future.

OK, this explains why you're using afio::future instead of std::future, but not why afio::future relies on afio::monad instead of std::future. AFAICS basic_monad doesn't add much over future's API, except for get_or() and friends. But those functions aren't being used anyway, right?

Monad's basic_future<Policy> is base for all future-ish and future-like future types. You can make whole families of customised future types with any semantics you like, so long as they are future-ish. One such custom future type is afio::future. Monad's basic_future<Policy> is a refinement of basic_monad<Policy> i.e. inherits directly. Anything implementing basic_future<Policy> therefore also implements basic_monad<Policy>. i.e. all lightweight futures are also lightweight monads. Just asynchronous ones. One big win is universal wait composure. You can feed any combination of custom future type into the same when_all() or when_any() function, and they'll compose.

...

Monad contains a full Concurrency TS implementation, and works everywhere in any configuration and currently with far lower overheads than any existing STL implementation by a big margin (about 50%).

That's a big claim. Do you have performance tests to back it up? Which which implementation did you compare your results?

See the benchmarks posted earlier. I compared to libstdc++ 5.1 and Dinkumware VS2015. There are known bugs and todo parts in Monad typical of a library only started in June, not least that future<void>.get() is not returning a void. I know of no serious bugs though, and AFIO is working well with Monad so far.

...

That means AFIO can drop the internal unordered_map, and become a pure Concurrency TS implementation with no workarounds and more importantly, no central locking at all. That will eliminate the last remaining synchronisation in AFIO, and it will be a pure wait free solution.

So, is AFIO composable with other parallel code? Could I use it along with other threads?

Absolutely! The whole point is generic reusability, and why I'm about to embark on such a large internal refactor to more closely align with the current Concurrency and Coroutines TSs. Code you write using AFIO right now will fit hand and glove with the C++ of the next five years - you are future-proofed with the API before review today. Regarding thread safety of AFIO objects, everything should be thread safe and can be concurrently used by multiple threads. You may not get desirable outcomes of course e.g. closing a handle during a write on that handle from another thread will likely cause data loss for obvious reasons. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 25 Aug 2015 at 16:32, Andreas Schäfer wrote:

...

I have those too, of course. Have a look at https://raw.githubusercontent.com/ned14/boost.monad/master/Readme.md, bottom of the document.

Thanks. Is the benchmark code for this available online so I can see what was actually measured?

Of course: https://github.com/ned14/boost.monad/blob/master/test/unittests.cpp

...

...

I'm still unsure what "51 opcodes <= Value transport <= 32 opcodes" is supposed to mean.

Min/max code bloat for a monad<int> or future<int>.

So the minimum is 51 and the maximum is 32?

Correct. That non-sensical outcome is due to definition. The minimum code bloat _should_ result from a known-memory outcome where the compiler knows for a fact memory has some value, and therefore can prune branching down to no code output at all. The maximum code bloat is where memory has unknown state, and therefore the compiler must generate all possible branches for that unknown state. Compiler optimisers are not as deterministic as perhaps they should be, so you get this weird outcome where the compiler generates more bloat for the known case than the unknown case. Which is a compiler optimiser bug, strictly speaking.

...

i.e. all lightweight futures are also lightweight monads. Just asynchronous ones.

So this architecture makes it easy to define different "future-ish" types. Got it. I've seen code blocks like this...

...

#define BOOST_MONAD_FUTURE_NAME_POSTFIX #define BOOST_MONAD_FUTURE_POLICY_ERROR_TYPE stl11::error_code #define BOOST_MONAD_FUTURE_POLICY_EXCEPTION_TYPE std::exception_ptr #include "detail/future_policy.ipp" #define BOOST_MONAD_FUTURE_NAME_POSTFIX _result #define BOOST_MONAD_FUTURE_POLICY_ERROR_TYPE std::error_code #include "detail/future_policy.ipp" #define BOOST_MONAD_FUTURE_NAME_POSTFIX _option #include "detail/future_policy.ipp"

...which, if I'm not mistaken define future, future_result and future_option. Same for monad, monad_result and monad_option. Of these only future and monad seem to be used in boost.monad, and only future is actually in use by boost.afio.

Yep, that's code which stamps out many policy class implementations using the preprocessor by reincluding the same base policy implementation with macros reconfiguring it. I got bored maintaining many separate policy classes, so I automated a single solution.

This leads me to the question: why this complicated architecture, if you only really rely on future?

Monad is designed as a useful library on its own. AFIO only uses part of it.

...

...

...

Monad contains a full Concurrency TS implementation, and works everywhere in any configuration and currently with far lower overheads than any existing STL implementation by a big margin (about 50%).

That's a big claim. Do you have performance tests to back it up? Which which implementation did you compare your results?

See the benchmarks posted earlier. I compared to libstdc++ 5.1 and Dinkumware VS2015.

Do you happen to have a link handy? I must have missed that post.

End of the Readme file! Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 2015-08-25 10:47, Niall Douglas wrote:

On 25 Aug 2015 at 16:32, Andreas Schäfer wrote:

...

...

...

I'm still unsure what "51 opcodes <= Value transport <= 32 opcodes" is supposed to mean.

Min/max code bloat for a monad<int> or future<int>.

So the minimum is 51 and the maximum is 32?

Correct. That non-sensical outcome is due to definition. The minimum code bloat _should_ result from a known-memory outcome where the compiler knows for a fact memory has some value, and therefore can prune branching down to no code output at all. The maximum code bloat is where memory has unknown state, and therefore the compiler must generate all possible branches for that unknown state.

Compiler optimisers are not as deterministic as perhaps they should be, so you get this weird outcome where the compiler generates more bloat for the known case than the unknown case. Which is a compiler optimiser bug, strictly speaking.

In this case perhaps you should document these differently. Label the two values for what they are, rather than using misleading inequalities; something like: Code bloat when memory known: 51 Code bloat when memory unknown: 32 (and each case should ideally link to the source code you're using to measure this bloat so someone who needs to understand the details can do so, or at least link back to the definitions). John

On 24 Aug 2015 at 0:46, Klaim - Joël Lamotte wrote:

Small side question: some time last year there was a discussion about filesystem/directory watcher libraries, with mention about one that would be part (or an example implementation) of boost AFIO. Looking at github, the branches that seem to be related (dir_monitor) are "staled".

Paul was working on that back in 2013/2014. It's his branch.

Am I correct that the directory monitoring "feature" is not part of this library review (I can't find it in the documentation)?

Correct. He never finished it. It's surprisingly hard to do correctly - most implementations out there are broken in subtle ways (mainly in handling change in large directories without racing on the delta calculation, and doing that with a good complexity and without using tons of RAM). I have some watch side code here for forthcoming async byte range locking, but it is not well tested, plus it only monitors single file entries (i.e. the lock file). I don't expect to finish that code until late 2016. In the meantime, simply poll async_enumerate(). It's a very fast call with good race guarantees on Windows at least, and I took care to make sure the afio::directory_entry structure is as tightly packed as possible. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 24 Aug 2015 at 11:22, Andreas Schäfer wrote:

I'm trying to run a benchmark from the documentation (see attached afio.cpp). Compilation works fine (what was the reason for leaving out the includes?), but the program then deadlocks in line 280:

finished_waiter.wait();

This is how I compiled, linked:

time g++ -std=c++11 afio.cpp -Iboost.afio/include/ -Iboost.afio -lboost_system -lpthread -lboost_filesystem -ldl -o afio && echo go && ./afio foo

I'm surely missing something obvious. Could you help me?

Looks like the find regex in files example. No problem, I'll get into it later today and get back to you. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

...

- The author clearly states that performance is not one of his goals (http://thread.gmane.org/gmane.comp.lib.boost.devel/261914/focus=262478). Yet, the documentation, especially the introduction, puts a great emphasize on that, while the presented benchmarks do not show any of those promises or compare apples with oranges.

I place correctness and lock free file system programming before absolute maximum performance yes.

Sure, that's something one should stride for, but that's not what the documentation conveys.

The comparing of apples to oranges is a fair point.

...

- Portability, the author claims to present a portable solution to the ACID problem however goes into great length in explaining the differences between various OSes and FSes from which I concluded that users of the library still have to implement special code paths for some systems. This comes especially obvious when looking at example code provided in the documentation.

There is a lot of variability in conformance and behaviour in filing systems and operating systems - even versions of the Linux kernel.

I could simply remove all mention of platform specific workarounds in the code examples and the docs. But that would not be useful to end users - they *want* to know the platform specific quirks and workarounds.

There is always a difference between entry level, basic documentation of the API you are presenting and going all in by tuning every possible knob there is. As I mentioned in the "usefulness" section, lots of people wanting an async file I/O API probably don't care about race freedom or platform specific quirks you ironed out. They just want something to get the job done.

...

- "Readiness": The docs list tons and tons of features which aren't really there yet. One can get the impression that it is only half done. Especially concerning is that running the unit tests might actually break your hard drive (https://boostgsoc13.github.io/boost.afio/doc/html/afio/quickstart/workshop/e...) which is not very assuring in the context of the promised feature set.

It is unfortunate that you interpreted my well defined future roadmap for AFIO as meaning "it isn't finished". I could simply remove all mention of the roadmap completely, and then AFIO would appear to be "finished".

That's not the point. The documentation is full of features that aren't there yet, for example: - Introduction: "Exclusive lock files (manually operated support already there, async support coming in v1.5)." "File change monitoring (coming in v1.5)." "File byte range advisory locking (coming in v1.5)." - In the cheat sheet, the complete last paragraph. - 2. World's simplest named blob store in AFIO (synchronous) "This is the first use of asynchronous i/o in this tutorial. AFIO provides a custom future<T> type extending the lightweight monadic futures framework in Boost.Monad, so you get all the C++ 1z Concurrency TS extensions, C++ 1z coroutines support and Boost.Thread future extensions in the AFIO custom future<>. There are also many additional extensions beyond what Boost.Thread or the Concurrency TS provides, including foreign future composable waits." Having a roadmap is all nice and dandy, but repeating it all over the place just creates the impression that your hasn't been completed yet.

But it would be a lie. I personally highly value a well planned documented future evolution of a library. It helps me to decide if locking myself into a library is a good idea. That's why the roadmap is so publicly well defined - the end user knows exactly where the library is at, and where it is expected to go next.

Right, as said, having a roadmap is nothing bad, but after reading through your documentation, the impression of just having a "almost done" library is created. I as a user wouldn't feel comfortable in using it as of yet.

...

It looks like it would have been better to let the dust settle after the final additions to the documentation and maybe even properly implement the praised backend before aiming for review. I think it would also be best to only include the latest version of the library for inclusion to boost, is the community really supposed to also review the apparently inferior V1 of the library?

Those are ABI versions. If you build and link against v1, I give you the promise your end user code will build and link against v1 for the lifetime of v1. For any break in ABI, I'll iterate to v2 (as has already happened), or v3 and so on.

Sure, but as far as I understand, v2 is something like the first official release, so why do I want to bother with v1?

...

...

2. What is your evaluation of the design?

The design of the documented, user facing API isn't what I expected from a modern C++ API. Following are some points, in no particular ordering, that struck me most.

- afio::future<T = void>:

As far as I can see, there is not a single function documented that returns anything than some equivalent to "shared_future<handle_ptr>".

The synchronous single shot functions return an afio::handle_ptr or some type appropriate for that function (e.g. enumerate() returns a vector of directory_entry).

The asynchronous single shot functions return an afio::future<T> where T is usually void. afio::future<T>.get_handle() returns an afio::handle_ptr, while afio::future<T>.get() returns whatever T is (which might be void).

The batch functions return vectors of afio::future<T>.

Ahh i missed the enumerate (btw, the sync enumerate example only shows the usage of the async version...) and stat functions (and maybe some others). The question remains: Why do they also need to return a handle?

...

The usage of afio::future<void> (explicitly added the default parameter) is, IMHO, very confusing as soon as you pass them as dependencies to another operation. The void template parameter always looks like "just a signal" instead most of the time, the actual *handle* associated with that future is the dependency, and from time to time even needed as an argument to the async operation.

I am unaware of any occasion where an async operation consumes a handle_ptr. If there are any, I would expect that to be a bug needing to be fixed.

Consuming as in taking it as a parameter, not consuming as in being moved into a function. All your "preconditions" are essentially handles to futures, right? So why not express it in the type? Or file open functions return a handle (or a future to handle), right? The return type is future<void> though. This is not nice. All in all, *everything* that returns an afio::future<T> (where T might be void) has two return values, that's very irritating. In addition your afio::future violates the "single responsibility principle" it sometimes serves as return value transport mechanism, sometimes to just be able to (out of convenience I guess) be able to access the underlying handle and sometimes both. What does "valid()" or "valid(true)" mean then?

From the documentation it's also not clear at all what semantics are attached to afio::future. It looks like it is something between a unique future and a shared future but couldn't decide between the two.

You may not be aware that afio::future<void> implicitly converts from an afio::handle_ptr. It's equivalent to make_ready_future(handle). This is effectively free of cost under lightweight futures, hence it being used.

Pleas don't do that! Please add a make_ready_future functionality, those non explicit constructors make my head hurt and are a potential source of errors, as convenient as they may seem at first sight. Just by having those explicit factory doesn't make anything less efficient.

...

In addition, "shared_future<handle_ptr>" has two indirections, one to the shared state of the future and one to the actual handle.

You have spotted one of those unusual design choices I mentioned earlier.

It turns out that shared_ptr<shared_ptr<handle>> (which is effectively what this is) is much faster than shared_ptr<handle>. This is because the outer shared_ptr is local to the operation in question and therefore its use count is not contended across threads, whereas the inner shared_ptr manages lifetime for the handle and is global across operations and its use count gets very much contended across threads.

Every shared_ptr comes with a cost be it contended or not ...

I agree the choice is not obvious, and indeed this was not the design in very early AFIO. I came to it through being puzzled about poor benchmark performance.

Is there a better alternative to this? To the next question ...

Why does it need to be shared_ptr<handle> in the first place? Why not just handle? A shared_ptr always implies shared ownership, when for example a boost::afio::file function returns, with whom is the ownership shared?

...

Moreover, I disagree that afio::future<T> should have shared future semantics by default. I am a proponent of the std::future/std::shared_future design and believe this should be followed by all libraries dealing with some sort of futures. I think most of the design decisions are based on shared_future only semantics, which is probably the reason why I have problems with them.

Did you notice that afio::future<T> has non-shared semantics for T? That's why it's called afio::future<>, not afio::shared_future<>.

As stated earlier, I have no idea which semantics exactly a afio::future<T> has ...

You are correct that the implicit handle_ptr available from afio::future<>.get_handle() always has shared future semantics. That's because handle_ptr is a shared_ptr, so you are already referring to shared state, and therefore shared future semantics would seem to fit best.

Should afio::future<>.get_handle() always have shared future semantics? This is a very profound question, and the entire of the AFIO design hangs around that design choice. It was not an easy decision back in 2012, and it is still not an easy decision in 2015 even after hundreds of hours working on AFIO.

I share completely your unease with the design choice, and I have the same problems you do. I can only tell you that on balance, I think that shared future semantics for the handle_ptr only make for a much easier implementation, and I chose ease of maintenance over strictness.

I don't think that this design choice impacts end users. No end user code is going to get broken or misdesigned (in my opinion) by handle_ptr always having shared future semantics. It is, in my opinion, a quality of internal implementation decision.

It is misdesigned in the way that it implies shared ownership from the start on, that is something that should be left to the user.

I think the true controversial design choice is the concept of a future<> type transporting more than one thing each with differing semantics. Especially when future<T> is allowed and expected to type slice into future<void> when entering the AFIO DLL boundary.

These controversial design choices are exactly why I wanted a Boost peer review now. If someone can think of a better viable solution which allows a stable DLL ABI, I am more than happy for AFIO to be rejected now and I reimplement a fundamentally better design in a new reborn AFIO.

If handle and future<T> is ABI safe, then future<handle> is ABI safe. Besides, a implicit conversion from future<T> to future<void> is perfectly acceptable, but should not and doesn't need to include type slicing.

...

- Extensive use of shared_ptr:

Some functions return some shared_ptr (specifically handle and dispatcher), therefore probably always relying on some sort of memory allocation. This seems to be due to the fact that only opaque abstract base classes are returned from those API functions. Why not return unique_ptr or even by value of the distinct type?

The answer is lifetime and stability of DLL ABI. AFIO v1.0 didn't make as much use of shared_ptr, and it had race issues and a bad balance of locking as a result. I rewrote the core engine in v1.2 to use shared_ptr to manage lifetime and therefore remove all but two locks per operation because the shared_ptr can be relied upon to keep lifetime without locking. Performance was immensely improved.

Can a lighter weight AFIO be implemented? Yes. Should it be? No [1]. The overhead of a shared_ptr is immaterial to a 1m CPU cycle i/o operation. And shared_ptr makes the code much easier to maintain, and much harder to accidentally introduce races during a modification.

That's actually a very strange statement. shared_ptr implies shared ownership, aka some shared state. Sharing state requires synchronization to protect it from races. And yes, if you modify this shared from two threads without synchronization, you accidentally introduced a race.

I haven't introduced a race (according to the thread sanitiser) into AFIO in well over a year thanks to pervasive use of shared_ptr.

That doesn't make a lot of sense ... Of course, if your state is not shared, you don't introduce races, but if you don't have a shared state, why use shared_ptr?

[1]: This isn't an opinion - I tested this empirically. I spent two days hacking out the shared_ptr to see what the effect would be on performance of the benchmark code. It improved by 0.5%. Not worth it.

You always emphasize that you care more about correctness than performance, exposing the proper semantics in a library API is also some form of correctness and leads to a certain quality of the design. Just using shared_ptr everywhere is convenient, sure, is it the right thing to do? I don't think so.

...

...

3. What is your evaluation of the implementation?

I refrained from actually looking at the implementation. This is actually a pretty severe point. The meat of the implementation is split into 4 headers only, with two headers (afio.hpp and detail/impl/afio.ipp) having the majority code with over 6000 lines of codes each! Despite the actual content, I believe this is a maintenance nightmare. I don't want to review such large blobs of text.

It is true I only orthonongally split files when I see a point because I like as few files as possible, as you can tell. I also like really long lines.

Most Boost libraries keep a header per type. I dislike that - I prefer a header per orthogonal reusable collection of stuff. On that basis there is some splitting of afio.hpp which is probably a good idea. I've recorded that as https://github.com/BoostGSoC13/boost.afio/issues/90.

...

...

4. What is your evaluation of the documentation?

- I have no idea what the table in https://boostgsoc13.github.io/boost.afio/doc/html/afio/overview.html is trying to tell me

Can you suggest what would be better?

If I would know what information you wanted to convey: maybe. After some email exchanges it appears that you mean the number of overloads, but again, what does that tell me? You want to give me the functions your library exposes and probably the capabilities of those functions? Express it that way!

...

- The design introduction contains lots of text that seems like unrelated anecdotes - The reference section doesn't cross link between the different related API functions/classes that are needed for one specific class/function.

Can you explain what you mean exactly? Do you mean links on each reference page?

I mean cross reference between things I might be interested in while looking at a specific reference section. For example "file (batch)" https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/functions/f... it takes a vector of path_req, what the heck was path_req again?

I had thought the categorisation of functions by functionality implemented made it easy to find what you needed quickly? How would you improve the reference organisation?

The organisation is fine. It, IMHO, just misses some cross referencing.

...

On the incomplete part: - The workshop tutorial is not finished. - Some references are not completely documented, for example:

https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/classes/dir...

For example st_dev, st_ino, st_type etc.

...

https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/classes/dir...

what does operator()(const directory_entry & p) so? what does it return?

...

https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/classes/dis...

read and write.

I'm confused. What is not completely documented?

Do you mean that trivial APIs do not have their own full reference page?

See above.

...

On the examples do not match reference documentation part: - https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/functions/m...

https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/functions/m...

https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/functions/m... The example mentioned in the above entries do not mention the make_io_req function at all.

You are absolutely right. I apologise for the mistake (it was a typo). I have logged these documentation errors at https://github.com/BoostGSoC13/boost.afio/issues/91.

...

https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/functions/d... Why does this example not focus on the presented function?

I felt that a single example showing the batch use and the free function use shown comparatively alongside other functions you would likely be using was more useful. As I mention below, it looks like it is instead confusing and I have logged an issue for fixing it.

...

On the examples containing errors part: - https://boostgsoc13.github.io/boost.afio/doc/html/afio/quickstart/workshop/n... " // Boost.Monad futures are also monads, so this implies a make_ready_future() return std::current_exception(); " The return type is shared_future<data_store::ostream>. What does this have to do with Boost.Monad and any of its behavior? Which shared_future is meant here that has a non-explicit ctor that takes a std::exception_ptr?

You may not have realised that all shared_future<T> mentioned by the documentation is boost::monad::lightweight_futures::shared_future<T>. Not std::shared_future<T>.

boost::monad::lightweight_futures::shared_future<T> will implicitly convert from a std::exception_ptr into an already ready future. That's what the comment was explaining, as that is not normal shared_future<T> behaviour.

Ok, this is overly confusing ... really, if you want to implement a "Concurrency TS compatible" future, please do so in its full extent, everything else is just confusing. Especially the non-explicit single argument constructors, please get rid of them.

...

- For example (there are many more):

https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/functions/f... documents dispatcher::file taking a vector of path_req, in the example, the vector is constructed, but only the front is passed to the function. This seems very odd.

The batch API is exampled further below under the section commented "Batch". I think you were looking at the top part.

I'm getting from this that a unified code example for both batch and single shot functions on both batch and single shot pages is confusing. Logged at https://github.com/BoostGSoC13/boost.afio/issues/92.

The biggest problem is that the unified example 1) contains too much other, unrelated information and 2) doesn't show the usage of the function on that page. The batch function is not in the example. Furthermore, why the heck do you create a vector to just get its first element? This is kind of inefficient and confusing, this appears all over the place (not only in the documentation but also in the implementation).

...

Other documentation flaws: - https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/functions/p... https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/macros_flag... looks odd, there are no parameters or anything there, so why does this page contain a section for it?

Sadly out of my control. It's a bug in the doxygen_xml2qbk tool.

...

https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/macros_flag... the synopsis is very hard to read.

Can you explain how it is hard to read? It tells you what each flag does. How could it be improved?

It can be improved by having one enumeration value per line.

...

(same goes for: https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/macros_flag... and https://boostgsoc13.github.io/boost.afio/doc/html/afio/reference/macros_flag...) - Why document undocumented features?

Unfortunately doxygen_xml2qbk seems to insist on outputting documentation for undocumented members, or maybe I don't understand the tool or maybe I've misconfigured it. So I can have a blank documentation entry like for afio::file_flags::os_lockable which seems inferior to saying "Don't use this".

...

https://boostgsoc13.github.io/boost.afio/doc/html/afio/quickstart/mini_progr... - Can you please strip example code from commented sections of code that obviously don't add anything to the example (or "#if 0"ed sections or "legacy" behavior)?

Logged to https://github.com/BoostGSoC13/boost.afio/issues/93.

...

- It would have been nice to see all examples (at least in a non advanced section) to not rely on platform dependent macro definitions.

If you are referring to https://boostgsoc13.github.io/boost.afio/doc/html/afio/quickstart/mini _programs/filesystem_races.html, one is trapped by substantial differences in Windows filesystem semantics. I still think it more important to real life end users of AFIO to demonstrate and show those platform specific differences than pretend they don't exist.

...

In addition, I personally dislike the "Try AFIO now in online web compiler" button. Why does it have to appear on *every* single page (there is also a bug that it doesn't appear in some of the reference doc sections)? Besides, it doesn't really match the style of the rest of the documentation. Which brings me to the "Comments" part, doesn't match the style either, which makes it look totally out of place especially since it comes from some third party provider, and I am not sure I can trust that. It looks especially odd when javascript is disabled. Isn't discussing a library and its documentation on a mailing list not enough?

Regarding the online web compiler button, I can remove it from every page, and place it just on one or two pages?

I'd personally put it onto one page. Just make a new section "Try AFIO now" or so...

Regarding the comments, I had thought there was consensus here that per-page commenting was a good idea, so that's why they are there.

I didn't take part of that discussion. I just dislike this disqus thingy, it doesn't fit the style of the rest of the page and is totally ill suited for discussing and commenting on documentation for a C++ library, in order to place code, you have to surround it with a <pre><code class="C++">your code here</pre></code> thingy.

Thanks hugely for the review Thomas. I found it very valuable.

Niall

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

-- Thomas Heller Friedrich-Alexander-Universität Erlangen-Nürnberg Department Informatik - Lehrstuhl Rechnerarchitektur Martensstr. 3 91058 Erlangen Tel.: 09131/85-27018 Fax: 09131/85-27912 Email: thomas.heller@cs.fau.de

2015-08-25 21:22 GMT+08:00 Niall Douglas <s_sourceforge@nedprod.com>:

2. The reason why it's shared_ptr<handle> not handle * is precisely because of how to manage lifetime over unknown continuations. If the user writes:

future<> oh=async_file("niall.txt"); future<> reads[100]; for(size_t n=0; n<100; n++) reads[n]=async_read(oh, buffers[n], 10, n*4096);

So this schedules niall.txt to be opened, and then adds 100 continuations onto that file open each of which schedules the read of 10 bytes from a 4Kb multiple. All those 100 continuations occur in *parallel*, and could complete in *any* order.

So how do you keep the handle to the niall.txt file around until the last continuation using that file has completed which is at some, unknown point in the future? The natural fit is shared_ptr. It's exactly what it was designed for.

Does this make sense?

I think it's an design issue, I wonder if it's possible to design the API around the ASIO model, e.g. ``` afio::file file(io); // move-only await file.async_open("niall.txt"); future<size_t> reads[100]; for(size_t n=0; n<100; n++) reads[n] = file.async_read(buffers[n], 10, n*4096); ``` Drop the dependency based API, since the forthcoming coroutine should solve it for you elegantly. This way, you don't need shared semantic for future<>, just like how we used to do with ASIO.

:

On 25 Aug 2015 at 8:22, Thomas Heller wrote:

...

...

...

As far as I can see, there is not a single function documented

Am 25.08.2015 3:24 nachm. schrieb "Niall Douglas" <s_sourceforge@nedprod.com that

...

...

...

returns anything than some equivalent to "shared_future<handle_ptr>".

The synchronous single shot functions return an afio::handle_ptr or some type appropriate for that function (e.g. enumerate() returns a vector of directory_entry).

The asynchronous single shot functions return an afio::future<T> where T is usually void. afio::future<T>.get_handle() returns an afio::handle_ptr, while afio::future<T>.get() returns whatever T is (which might be void).

The batch functions return vectors of afio::future<T>.

Ahh i missed the enumerate (btw, the sync enumerate example only shows the usage of the async version...) and stat functions (and maybe some others). The question remains: Why do they also need to return a handle? [snip] Why does it need to be shared_ptr<handle> in the first place? Why not just handle? A shared_ptr always implies shared ownership, when for example a boost::afio::file function returns, with whom is the ownership shared? [snip] It is misdesigned in the way that it implies shared ownership from the start on, that is something that should be left to the user. [snip] That's actually a very strange statement. shared_ptr implies shared ownership, aka some shared state. Sharing state requires synchronization to protect it from races. And yes, if you modify this shared from two threads without synchronization, you accidentally introduced a race. [snip] That doesn't make a lot of sense ... Of course, if your state is not shared, you don't introduce races, but if you don't have a shared state, why use shared_ptr?

I think Thomas you have missed something very fundamental about the AFIO design. I hope you don't mind me not replying to the rest of it (which was useful) and I'll just dig into the above as others may be where you are.

1. An afio::future<> is *always* a future handle to a file. You asked why all functions always return a future handle to a file instead of just the functions which open a file. I'm getting the feeling you didn't understand that:

future<vector<directory_entry>> async_enumerate(future<> precondition)

... expands, quite literally as you'll see in afio.hpp, into:

return precondition.then(detail::async_enumerate());

... which is a Concurrency TS continuation. detail::async_enumerate might then be effectively implemented as:

struct async_enumerate { future<vector<directory_entry>> operator()(const future<> &precondition) const { return await do_async_directory_enumeration(precondition.get_handle()); } };

In other words, when the future precondition becomes ready, it will execute the continuation which calls the call operator on the instance of detail::async_enumerate. That call operator will schedule (via await) the directory enumeration of the file handle just become ready in the future precondition.

Erm no, await doesn't schedule the directory enumeration, it just continues the control flow after the future returned from do_async_directory_enumeration returns. In fact, await would have been better suited to wait for the precondition.

This is why afio::future<> *always* is a future to a handle to a file. Does this make sense?

No, not really, that doesn't answer my question at all.

2. The reason why it's shared_ptr<handle> not handle * is precisely because of how to manage lifetime over unknown continuations. If the user writes:

future<> oh=async_file("niall.txt"); future<> reads[100]; for(size_t n=0; n<100; n++) reads[n]=async_read(oh, buffers[n], 10, n*4096);

So this schedules niall.txt to be opened, and then adds 100 continuations onto that file open each of which schedules the read of 10 bytes from a 4Kb multiple. All those 100 continuations occur in *parallel*, and could complete in *any* order.

So how do you keep the handle to the niall.txt file around until the last continuation using that file has completed which is at some, unknown point in the future? The natural fit is shared_ptr. It's exactly what it was designed for.

Does this make sense?

No, not at all, sorry. The file handle should be kept alive by the future holding on to it and probably also by the continuations attached by async_read. Anything else should be handled by the user. Furthermore, if I'm not completely mistaken, do all OSes internally reference count open file handles, why duplicate that in user space again?

3. You have stated that you dislike the idea of afio::future<T> having shared future semantics for the future handle. But can you see that it is not possible to schedule 100 parallel data read continuations onto a file open unless the handle has shared future semantics?

After all, if it had future semantics, you could only schedule exactly one read to each file open because then you would consume the future. It would not be possible to schedule parallel operations on any single preceding operation!

That's correct. However you miss an important point: shared_future<T>::shared_future(future<T>&&) Meaning, a rvalue reference to future<T> can be implicitly converted to a shared_future<T> (same is true for shared_ptr and unique_ptr, BTW). With that, no publicly facing api function should ever return a shared_future<T>, in my book as this doesn't make a lot of sense when a future is seen as a return value transport endpoint. It should be in the users responsibility how to deal with that. Some options: - just ".get()" the handle and continue with it as a named variable. This will save you the overhead of attaching continuations over and over again. - just ".then()" and continue as above - await it (my favorite actually) and continue as above - put it into a shared_future and continue as you outlined it. I understand, of course, that this might get clumsy in your current design, hence my criticism.

Perhaps I am not understanding your concerns correctly?

Looks like it.

Niall

-- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

_______________________________________________ Unsubscribe & other changes:

http://lists.boost.org/mailman/listinfo.cgi/boost

On 08/25/2015 07:45 PM, Niall Douglas wrote:

On 25 Aug 2015 at 17:44, Thomas Heller wrote:

...

No, not at all, sorry. The file handle should be kept alive by the future holding on to it and probably also by the continuations attached by async_read.

But that is *exactly* what is implemented. Anything which will use a file handle in the future increments its reference count. Once it's finished with the handle, it decrements its reference count. When the count reaches zero and we know it will never be used again, the handle is destroyed.

Important nitpick of your implementation: Not the shared state of the future but the value held in the future keeps it alive.

...

Anything else should be handled by the user. Furthermore, if I'm not completely mistaken, do all OSes internally reference count open file handles, why duplicate that in user space again?

I don't see how this bears any relevance. It doesn't ever bear relevance to code using AFIO. I could change the typedef for afio::handle_ptr in the future for some reason, and code would not need to care usually.

You're not, by any chance, saying that you think the AFIO design should be 1 afio::handle = 1 fd? And when you copy construct afio::handle, that calls dupfd()?

That is what I would expect from a handle copy constructor, yes. Note that there are also move operations and a user is still free to store handles in any "container" (including smart pointers or whatever).

...

...

3. You have stated that you dislike the idea of afio::future<T> having shared future semantics for the future handle. But can you see that it is not possible to schedule 100 parallel data read continuations onto a file open unless the handle has shared future semantics?

After all, if it had future semantics, you could only schedule exactly one read to each file open because then you would consume the future. It would not be possible to schedule parallel operations on any single preceding operation!

That's correct. However you miss an important point: shared_future<T>::shared_future(future<T>&&) Meaning, a rvalue reference to future<T> can be implicitly converted to a shared_future<T> (same is true for shared_ptr and unique_ptr, BTW). With that, no publicly facing api function should ever return a shared_future<T>, in my book as this doesn't make a lot of sense when a future is seen as a return value transport endpoint.

Ah I think I'm beginning to understand now. You're assuming a world where only std::future and std::shared_future exist and the relationship between them is well understood.

By which, I infer you're saying that one does not need anything more than std::future and std::shared_future in an asynchronous filesystem and file i/o library?

I didn't say there is no place for other things than std::future or std::shared_future. You should know that ... BUT if you claim conformance to the Concurrency TS ...

And therefore, the current approach taken by AFIO of custom futures is wrong?

... a design that clearly does not conform to anything defined in the standard and only the names coincide, then yes, I consider the design broken. In addition, I see no shortcomings in the design of std::future/std::shared_future that wouldn't fit your usecase.

Would this summary of your criticism be correct? If it is, I can address all those points, but it'll probably be Friday as tomorrow I am away and on Thursday we have no electricity all day, so no internet nor computers.

It's a very short summary of it, yes. Please keep in mind that all I said was under the assumption that anything "future" like in AFIO (or the depend NotMonad library) is just like a shared_future/future with some extensions (you repeated that multiple times on the list). If you have something different that's totally fine but needs other and special attention because fighting an uphill battle against an ISO standard is not easy. Especially after all the claims that have been made.

Niall

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Am 25.08.2015 11:48 nachm. schrieb "Niall Douglas" < s_sourceforge@nedprod.com>:

On 25 Aug 2015 at 21:10, Thomas Heller wrote:

...

...

You're not, by any chance, saying that you think the AFIO design should be 1 afio::handle = 1 fd? And when you copy construct afio::handle, that calls dupfd()?

That is what I would expect from a handle copy constructor, yes. Note that there are also move operations and a user is still free to store handles in any "container" (including smart pointers or whatever).

Just reminding you I won't be online after this until Friday ...

You may not have considered that:

1. On POSIX file descriptors held by a process is limited, especially on OS X which has a crazy low limit.

2. Metadata updates are not guaranteed to be immediately consistent across file descriptors. E.g. fstat() on one fd can return very different information from fstat() on another fd even to the same file in the same process. This particularly happens with NTFS and ZFS, and is a major pain in lock free file system concurrency. Making this go away most of the time by reusing the same fd for end users is a big win.

3. In scatter gather file i/o there is no concept of file position, and therefore there is no good reason to needlessly duplicate fds when one will do.

4. AFIO has to "spend" fd's on workarounds for annoying problems on networked filing systems. For example, every byte range locked file needs to create a shadow file which is locked rather than the real file. This makes the problem of running out of fd's even worse.

5. Because of the need to pin lifetimes of things to one another due to things like shadow lock files and delete-on-close semantics, you're going to need a reference counting system one way or another. You can either hide that internally, or expose it. I chose to expose it, as I suspect it is more useful to end user code who also need to manage lifetimes.

Finally I would also suspect that dupfd() is going to be a lot slower than a shared_ptr increment, though granted you'd only call it non-frequently hopefully (unless you put a handle in a C++ 98 container implementation).

I did of course not consider all points, however I am assuming that actual copies are very rare in the general case. My main beef with the design you chose is that it implies shared ownership where it seems to not be necessary.

...

...

Ah I think I'm beginning to understand now. You're assuming a world where only std::future and std::shared_future exist and the relationship between them is well understood.

By which, I infer you're saying that one does not need anything more than std::future and std::shared_future in an asynchronous filesystem and file i/o library?

I didn't say there is no place for other things than std::future or std::shared_future. You should know that ... BUT if you claim conformance to the Concurrency TS ...

I claim conformance to a suite of *extensions* to the Concurrency TS.

I hope you are confirming to your own set of extensions, would be very bad otherwise...

AFIO needs to be able to schedule multiple continuations on a future, that's the single biggest extension I need. The Concurrency TS doesn't allow that because the first continuation is expected to consume the future.

That's true for std::future but as already stated not so much for std::shared_future.

The ability to avoid exception_ptr is nice as well though. exception_ptr is expensive.

So it's expensive in the exceptional case, so what?

...

...

And therefore, the current approach taken by AFIO of custom futures is wrong?

... a design that clearly does not conform to anything defined in the standard and only the names coincide, then yes, I consider the design broken. In addition, I see no shortcomings in the design of std::future/std::shared_future that wouldn't fit your usecase.

Do bear in mind that up until v1.3, AFIO was using std::future and std::shared_future. I didn't go off and extend the Concurrency TS without two years of experience of what exactly needs extending in the TS and why.

My extensions are actually very conservative because I remain 100% compatible with the TS.

I actually lost track of what exactly your extensions are... Implicit conversions from T, exception_ptr and error_code? What else? Would you consider afio::future to be a complaint implementation? To what does it comply to?

If you look at what others are doing to future-promise e.g. Sean Parent, they are reworking the fundamental design.

Sure, but that design is not up for review and doesn't claim conformance.

...

...

Would this summary of your criticism be correct? If it is, I can address all those points, but it'll probably be Friday as tomorrow I am away and on Thursday we have no electricity all day, so no internet nor computers.

It's a very short summary of it, yes. Please keep in mind that all I said was under the assumption that anything "future" like in AFIO (or the depend NotMonad library) is just like a shared_future/future with some extensions (you repeated that multiple times on the list). If you have something different that's totally fine but needs other and special attention because fighting an uphill battle against an ISO standard is not easy. Especially after all the claims that have been

made.

That makes sense. And changes my perspective on what the documentation needs to focus on. Thanks.

Niall

-- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

_______________________________________________ Unsubscribe & other changes:

http://lists.boost.org/mailman/listinfo.cgi/boost

...

My main beef with the design you chose is that it implies shared ownership where it seems to not be necessary.

Let us assume that file descriptors must be reference counted in user space as they are a precious resource, and as I mentioned you need to internally tie lifetimes of internal workaround objects to things. So let's assume the need for std::shared_ptr<handle> has been agreed and move onto the meat on the bone which is the choice of shared future semantics over unique future semantics for the std::shared_ptr<handle> transported by afio::future.

Why isn't your 'handle' reference counted on its own, then? The user does not even have to know/see if that was the case. This also would get rid of this handle to an handle to and handle design which is so confusing and probably creates a lot of inefficiency. A simple boost::future<handle> as return values would cut the bill (note, I said boost::future as it already implements all of the Concurrency TS). That would also eliminate the whole monad/afio::future mess you're in. Those do not belong in AFIO.

First things first: shared future vs unique future isn't helpful terminology. For AFIO both have equal overhead. What we're really talking about is whether future.get() can be called exactly once (future) or many times (shared future). From now on, I will therefore talk in terms of consuming futures (future) vs non-consuming futures (shared future).

First, the problem lies not in whether the implementations are equivalent, but in the fact that shared_future and (unique_)future convey a design choice (FWIW, future and shared_future in HPX share their implementation as well, mostly). From this, the terminology is not only helpful, but absolutely essential for good design. Second, the possibility to call .get() more than once on a shared_future is a _consequence_ of the design, not the _rationale_ for it. Third, the terminology 'consuming' and 'non-consuming' future does not make any sense (to me). More importantly it is by no means Standard-terminology - thus reasoning in terms of those makes it much more difficult for us to talk about the same thing.

I've been trying to think of an API for asynchronous filesystem and file i/o which (a) uses consuming future instead of non-consuming future and (b) is invariant to the concurrency engine under the bonnet so end user code doesn't need to be made customised for the concurrency runtime.

If you're interested in an concurrency-safe API, then returning (unique_)future would be the proper choice. That conveys unique ownership (i.e. no concurrency problems are possible) and makes it explicit that if the user decides to assign it to a shared_future, then he/she is responsible for dealing with concurrency issues.

Ok, let's revisit the original pattern code I mentioned:

EXAMPLE A:

shared_future h=async_file("niall.txt"); // Perform these in any order the OS thinks best for(size_t n=0; n<100; n++) async_read(h, buffer[n], 1, n*4096);

This expands into these continuations:

EXAMPLE B:

shared_future h=async_file("niall.txt"); // Call these continuations when h becomes ready for(size_t n=0; n<100; n++) // Each of these initiates an async read, so queue depth = 100 h.then(detail::async_read(buffer[n], 1, n*4096));

Here we explicitly say we don't care about ordering for the async_reads with respect to one another and the OS can choose any ordering it thinks best, but we do insist that no async_reads occur before the async_file (which opens "niall.txt" for reading).

Ok, that's possible just fine if you return boost::future from async_file (shared_future has a non-implicit conversion constructor)

Let's imagine that with value-consuming semantics:

EXAMPLE C:

future h=async_file("niall.txt"); // Call these continuations when h becomes ready for(size_t n=0; n<100; n++) // Each of these initiates an async read, so queue depth = 100 h.then(detail::async_read(buffer[n], 1, n*4096));

On the second h.then() executed i.e. for when n==1, you should see a future_errc::no_state exception thrown because the first .then() is considered to have consumed the future state. This is because future.get() can be called exactly once.

I don't see a reason why anybody would do this. If you know (as a user) you need shared ownership you just make it explicit by assigning the boost::future to a boost::shared_future (as you showed in Example A).

So, to have it not throw an exception, you need to do this:

EXAMPLE D:

future h=async_file("niall.txt"); shared_future h2(h); // Call these continuations when h becomes ready for(size_t n=0; n<100; n++) // Each of these initiates an async read, so queue depth = 100 h2.then(detail::async_read(buffer[n], 1, n*4096));

Sure, but Example A is just fine, as said.

Now we consume the consuming future into a non-consuming future, and non consuming futures are not consumed by their continuations so you can add as many as you like (in this case 100).

This makes my head spin! What is consuming what? Or do you mean 'consummate'? Can we stick to Standard-terminology, please? We have unique ownership and shared ownership with regard to the value represented by the future - that's all I know.

Where I was coming from in the presented AFIO API design was that (a) you the programmer is always trying to give as much opportunity to parallelise (i.e. no ordering) as possible to the OS, hence the pattern of adding more than one continuation to a future is very common. That would throw an exception if the programmer forgot to first convert the consuming future into a non consuming future (b) I can't see any way of adding type safety to enforce single consumption of consuming futures such that Example C refuses to compile.

If the programmer 'forgot' to use a shared_future when shared ownership is needed then he/she gets what he/she deserves.

However this discussion makes me wonder if I'm missing a trick. The big value in non consuming futures is they force you to think about better design. I'd like to not lose that if possible. So, perhaps Example A could be rewritten as:

EXAMPLE E:

future h=async_file("niall.txt"); // Perform this after h future r=async_read(h, buffer0, 1, 0); // Perform these in any order the OS thinks best // after r for(size_t n=0; n<100; n++) parallel_async_read(r, buffer[n], 1, n*4096);

In other words, for the case where you issue multiple continuations onto a consuming future, you must use an API with a parallel_* prefix. Otherwise it will throw on the second continuation add.

There are probably a 100 ways to express explicit data dependencies in your code. Why not stick to the one specified in the Concurrency TS? Why invent something new - again? A general note. My main issue with the way you have designed things in AFIO is that you verbally strive for conformance, but then for no (or little) reason decide to change some minor semantics here and there (maybe nobody will notice) such that the result still appears to be conforming, but is nevertheless different in subtle ways. That makes it _very_ difficult to reason about your design as one never knows whether you really mean what you say. Before going off to design your own, you may consider first fully understanding the design and rationale of existing Standard-solutions. And only if you really can't make it work build something on your own. But never, really never, take a Standard-design and _change_ it (adding things is fine) without giving it a new name! Regards Hartmut --------------- http://boost-spirit.com http://stellar.cct.lsu.edu

On 27 Aug 2015 at 15:31, Vinícius dos Santos Oliveira wrote:

Anyway, maybe I can help here. I know Niall had played with Rust for sometime[1] (and end up eventually hating for the lack of an asynchronous foundation[2]). I too end up playing with Rust for some time and maybe that's why I understood Niall without trouble.

Hating is far too strong a word. Rust's standard i/o library can't do select() on sockets and instead forces you to spin up a dedicated kernel thread to sit around waiting for reads, and then provides no way whatsoever of poking we-are-about-to-shutdown at that dedicated thread. I got very frustrated working around that, as would anyone. We're not stuck in the 1970s for god's sake. Rust's standard i/o library is close to useless for real world portable programming, that's all. They know this, and this time next year they'll have a vastly improved standard i/o library.

Anyway, a value that was consumed is a value that cannot be used again. There are consuming operations that will consume the value. It's closely related to the move idea, where the moved value is left in an unspecified state. There are moving operations and there are consuming operations, but consuming operations work on values other than rvalues too.

Imagine a range that invalidates the original container. It *consumes* the container.

Does that explanation help?

Remember that Hartmut and Thomas are coming from a standards-ease perspective, and hence their disavowal of thinking in terms of anything not termed in standards-ease. You may not be aware Vinícius that Hartmut is a world leading expert in concurrency, and is one of the main minds behind the Concurrency TS. I in the meantime don't really care much about design purity - if I did I would not write in C++. I do care about impure design causing users to write dangerous code which will fail in unexpected ways, and to that end consuming futures are usually superior. However in AFIO's case the thing being transported with non-consuming semantics is a shared_ptr, and both that and the handle it reference counts is thread safe. This, as far as I can tell, eliminates any chance for dangerous code being written by end users and is why I allowed that design choice and it does save significant additional typing to be written by end users. But in the end, I'm no world expert on concurrency. I mostly design by instinct not logic. My design choices are famously mongrel as a result as you can see in afio::future<T> having dual future semantics, one for the handle, the other for the T. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Am 28.08.2015 4:59 vorm. schrieb "Niall Douglas" <s_sourceforge@nedprod.com

:

On 27 Aug 2015 at 15:31, Vinícius dos Santos Oliveira wrote:

Remember that Hartmut and Thomas are coming from a standards-ease perspective, and hence their disavowal of thinking in terms of anything not termed in standards-ease.

That's quite a statement! *YOU* are the one claiming standard conformance. That's the reason why *I* argue in terms of the standard. I absolutely have no problem with other design decisions if the rationales behind them are sane, but you can't have both, standard compliance and non standard compliance, decide yourself. And yes, I have significant experience in both standard complaint and non standard compliant futures. It took me quite a while to understand all decisions but after all I'm quite happy that we adapted our implementation to be aligned with the standard and now design library APIs and applications that result out of that ecosystem.

On 28/08/2015 00:58, Hartmut Kaiser wrote:

...

Third, the terminology 'consuming' and 'non-consuming' future does not make any sense (to me). More importantly it is by no means Standard- terminology - thus reasoning in terms of those makes it much more difficult for us to talk about the same thing.

future.get() moves the result of the promise from the internal state, thereby consuming it. Further calls to get() or then() fail.

shared_future.get() copies the result of the promise from the internal state, thereby not consuming it. Further calls to get() or then() succeed.

All of this is an irrelevant implementation detail. Nobody forces the future to share its state with a promise. It could very well be a packaged_task, for instance, or any other asynchronous provider.

I find it hard to imagine this being a difficult concept.

...

I don't see a reason why anybody would do this. If you know (as a user) you need shared ownership you just make it explicit by assigning the boost::future to a boost::shared_future (as you showed in Example A).

His point was that someone might forget and then it becomes a runtime exception, which can be painful to find if it's in a seldom-executed part of the code (perhaps error handling).

If somebody forgets, then an exception is the correct thing to do. Alternatively an assertion could be used in this case as well. BTW, an exception is not difficult to find, even gdb can break on a specific exception being thrown.

...

...

future h=async_file("niall.txt"); shared_future h2(h); // Call these continuations when h becomes ready for(size_t n=0; n<100; n++) // Each of these initiates an async read, so queue depth = 100 h2.then(detail::async_read(buffer[n], 1, n*4096));

Sure, but Example A is just fine, as said.

Note that it's not user code calling then() on the futures in most cases -- see the examples. Instead the future is passed as a "precondition" parameter to a wrapper API in the library, which is what actually registers the continuation.

I think Niall's point is that it's harder for the wrapper to know whether it's going to be called a single time or multiple times for a given precondition future, so it's safest if that is accepted only as a shared_future.

BUT that means that there is absolutely no benefit (and some drawbacks) to returning non-shared futures, as a result (there are only potentially some savings if they're never used as preconditions).

I suppose arguably the wrappers could be templated to accept either future or shared_future, which might mitigate that. But perhaps there are reasons why that's undesirable as well.

If an API enforces wrong behavior, then the API itself is wrong. Thus, if the wrappers require for the code to return a shared_future, then the wrappers are wrong in the first place. They force the user to deal with library internals which are irrelevant and misleading. Regards Hartmut --------------- http://boost-spirit.com http://stellar.cct.lsu.edu

...

...

On 28/08/2015 00:58, Hartmut Kaiser wrote:

...

Third, the terminology 'consuming' and 'non-consuming' future does not make any sense (to me). More importantly it is by no means Standard- terminology - thus reasoning in terms of those makes it much more difficult for us to talk about the same thing.

future.get() moves the result of the promise from the internal state, thereby consuming it. Further calls to get() or then() fail.

shared_future.get() copies the result of the promise from the internal state, thereby not consuming it. Further calls to get() or then() succeed.

All of this is an irrelevant implementation detail. Nobody forces the future to share its state with a promise. It could very well be a packaged_task, for instance, or any other asynchronous provider.

You're sidestepping.

No. I'm trying to get your attention.

Some mechanism results in the value given to the promise arriving at the future, and thereby allowing it to return the value from get() or invoke then() continuations. For the purposes of this discussion I frankly don't care how that implementation is done, or whether it arrives via some other source such as a packaged_task.

However, the consequence of the front-end interface (future and shared_future) is as I described above. One of them is one-use-only and the other allows multiple uses. This is a visible and important characteristic -- in fact it is probably the *most* visible and important characteristic from a usage perspective. And that is the perspective that Niall appears to be using as well.

I'm not arguing that. I'm arguing, that the fact that you can use the value more than once is not the rationale of the design but a consequence of it. The rationale is to differentiate between unique ownership and shared ownership with regard to the value provided by the future. Thus the question whether code asking for a value is making the future invalid (is 'consuming the future' as you put it) is a consequence and not the purpose of the design either. Thus talking about 'consuming' a future is making it more difficult to understand than needed.

Whether the future internally uses shared state or not is actually the inconsequential part from the perspective of users of the classes. (Except where it affects performance -- and even that is inconsequential to some users. Perhaps even most users, given the prevalence of people saying "just use shared_ptr everywhere" not that long ago.)

Which is as wrong as is using shared_future everywhere (as done in AFIO). Even worse, AFIO does both, btw.

...

If somebody forgets, then an exception is the correct thing to do. Alternatively an assertion could be used in this case as well. BTW, an exception is not difficult to find, even gdb can break on a specific exception being thrown.

But only if the code path actually executes. While in an ideal world the unit tests will have 100% coverage, we don't live in that ideal world.

If the code path does not execute the fact you forgot is inconsequential :-P But seriously, you're trying to argue in favor of a bad design decision (namely creating the wrappers which apparently require using shared_futures) by saying that if we don't use the shared_futures the user suffers. What about not having the wrappers in the first place and create a proper API instead which would allow for unique ownership wherever possible and leave the decision whether shared ownership is necessary to the user? Regards Hartmut --------------- http://boost-spirit.com http://stellar.cct.lsu.edu

...

I think Niall's point is that it's harder for the wrapper to know whether it's going to be called a single time or multiple times for a given precondition future, so it's safest if that is accepted only as a shared_future.

It's more that when shared_future can only EVER refer to a shared_ptr<afio::handle> all of which is thread safe, I see little risk to defaulting to shared_future. What does one lose?

You lose the possibility to express your intent. There is no conceptual need in having shared ownership. The need is imposed by the bad design of the API.

...

I suppose arguably the wrappers could be templated to accept either future or shared_future, which might mitigate that. But perhaps there are reasons why that's undesirable as well.

The wrappers can do either without a problem because in Boost.Monad/Outcome an extension allows consuming futures to accept any number of continuations. So from my perspective, it's six of one and half a dozen of the other from an implementation viewpoint. The real question is which makes for a better library use experience, and just because a design "should" be some way or other just because theory says so and you are wilfully ignorant of the problem domain being solved isn't useful to me.

Being willfully ignorant of rules of good design isn't useful to your users either. Regards Hartmut --------------- http://boost-spirit.com http://stellar.cct.lsu.edu

On 31 Aug 2015 at 2:27, Gruenke,Matt wrote:

...

You lose the possibility to express your intent. There is no conceptual need in having shared ownership.

I agree with this. Libraries shouldn't dictate memory management strategies to their users. If you must accept only shared_future<> as preconditions, that's understandable. But it doesn't mean you need to return them, when they're not (yet) shared by anything.

This semantic distinction can have real, practical implications. When I see code using shared_ptr<>, it's a sign to me that the object is truly shared. This has implications on how I use it, so I need to take the time to understand with whom it's shared and how they access it. If it's not *really* shared, then it's misleading and potentially wastes users' time.

I don't know why you and the HPC crowd are conflating shared_ptr with shared ownership. It an atomic safe reference counting implementation which provides no guarantees to its pointee. I find it personally unfortunate that C++ decided to call it shared_ptr because of the word "shared" in the name, because it really isn't what its name says. I might add that in AFIO I use the typedef handle_ptr. You don't think to think shared anything, just use afio::handle_ptr and stop worrying about shared_ptr. It's an implementation detail, and not important.

Another reason why it might be necessary to understand the ownership structure of shared objects is a consequence of the fact that ref counting != garbage collection. Since it's incumbent on the user to avoid circular references, this is also a cost, in development time.

I think you are making up speculative problems here. This has never been an issue in anything I've written using AFIO to date, and I would doubt it ever will. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 08/31/2015 04:52 PM, Niall Douglas wrote:

On 31 Aug 2015 at 2:27, Gruenke,Matt wrote:

...

...

You lose the possibility to express your intent. There is no conceptual need in having shared ownership.

I agree with this. Libraries shouldn't dictate memory management strategies to their users. If you must accept only shared_future<> as preconditions, that's understandable. But it doesn't mean you need to return them, when they're not (yet) shared by anything.

This semantic distinction can have real, practical implications. When I see code using shared_ptr<>, it's a sign to me that the object is truly shared. This has implications on how I use it, so I need to take the time to understand with whom it's shared and how they access it. If it's not *really* shared, then it's misleading and potentially wastes users' time.

I don't know why you and the HPC crowd are conflating shared_ptr with shared ownership. It an atomic safe reference counting implementation which provides no guarantees to its pointee.

I find it personally unfortunate that C++ decided to call it shared_ptr because of the word "shared" in the name, because it really isn't what its name says.

From ISO/IEC 14882:2014:

20.9.2 Shared-ownership pointers [...] 20.9.2.2 Class template shared_ptr The shared_ptr class template stores a pointer, usually obtained via new. shared_ptr implements semantics of shared ownership; the last remaining owner of the pointer is responsible for destroying the object, or otherwise releasing the resources associated with the stored pointer. A shared_ptr object is empty if it does not own a pointer. 'nuff said.

I might add that in AFIO I use the typedef handle_ptr. You don't think to think shared anything, just use afio::handle_ptr and stop worrying about shared_ptr. It's an implementation detail, and not important.

...

Another reason why it might be necessary to understand the ownership structure of shared objects is a consequence of the fact that ref counting != garbage collection. Since it's incumbent on the user to avoid circular references, this is also a cost, in development time.

I think you are making up speculative problems here. This has never been an issue in anything I've written using AFIO to date, and I would doubt it ever will.

Niall

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

-- Thomas Heller Friedrich-Alexander-Universität Erlangen-Nürnberg Department Informatik - Lehrstuhl Rechnerarchitektur Martensstr. 3 91058 Erlangen Tel.: 09131/85-27018 Fax: 09131/85-27912 Email: thomas.heller@cs.fau.de

On 08/31/2015 07:08 PM, Niall Douglas wrote:

On 31 Aug 2015 at 17:18, Thomas Heller wrote:

...

...

I don't know why you and the HPC crowd are conflating shared_ptr with shared ownership. It an atomic safe reference counting implementation which provides no guarantees to its pointee.

I find it personally unfortunate that C++ decided to call it shared_ptr because of the word "shared" in the name, because it really isn't what its name says.

From ISO/IEC 14882:2014:

20.9.2 Shared-ownership pointers [...] 20.9.2.2 Class template shared_ptr The shared_ptr class template stores a pointer, usually obtained via new. shared_ptr implements semantics of shared ownership; the last remaining owner of the pointer is responsible for destroying the object, or otherwise releasing the resources associated with the stored pointer. A shared_ptr object is empty if it does not own a pointer.

'nuff said.

An excellent example of how theory based code design can introduce a certain myopia in its adherants which leads to certain design rigidities.

The ISO standard tries to describe behaviour guarantees rather than implementation algorithms that must be used, and it does that above for shared_ptr.

Yes, it describes the behavior. What you describe is totally beyond me, please provide me with your goggles to cure me and the rest of the (C++) world from our myopia.

*IF* shared_ptr were implemented on ANY solution not using an atomic reference count, I could see a strong rationale for considering shared_ptr as the thingy which abstracts some of shared ownership.

I have no clue what that even means. Must be because I am German.

But the reality is that on every implementation I'm aware of, it's an atomically incremented and decremented reference count.

I *could* go roll my own reference count solution for AFIO. But std::shared_ptr is well tested, well supported, I would assume well optimised, and does everything I need.

So I grabbed the standard atomic reference count implementation which comes in the STL and bent it to my needs, and I get on with life.

Ok, please provide an example other than sharing ownership to a pointer/resource that requires atomic reference counting?

I would like to once again reiterate that AFIO exposes its use of shared_ptr as afio::handle_ptr. You, the AFIO end user, need not concern yourself with what kind of pointer handle_ptr is. You simply need to accept that it behaves like any smart pointer. I might change afio::handle_ptr to be something else at some future point, and your code need not concern itself about that.

This witch hunt over my use of shared_ptr I think is done now. I've proven its overhead is inconsequential in AFIO's use case. I think we can accept this design decision is not important and move on now.

So it's a witch hunt now? My post was merely to inform you about what others think about what semantics a shared_ptr implies. You don't have to like it, just be aware that without a common ground of definitions, there can be no profounded technical discussion. Call it a witch hunt if you like... From your "ivory tower"-ly yours, Thomas P.S.: I am casting a spell with over 9000!

Niall

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

...

...

You lose the possibility to express your intent. There is no conceptual need in having shared ownership.

I agree with this. Libraries shouldn't dictate memory management strategies to their users. If you must accept only shared_future<> as preconditions, that's understandable. But it doesn't mean you need to return them, when they're not (yet) shared by anything.

This semantic distinction can have real, practical implications. When I see code using shared_ptr<>, it's a sign to me that the object is truly shared. This has implications on how I use it, so I need to take the time to understand with whom it's shared and how they access it. If it's not *really* shared, then it's misleading and potentially wastes users' time.

I don't know why you and the HPC crowd are conflating shared_ptr with shared ownership. It an atomic safe reference counting implementation which provides no guarantees to its pointee.

I find it personally unfortunate that C++ decided to call it shared_ptr because of the word "shared" in the name, because it really isn't what its name says.

I believe that a lot of thought went into this decision. And it's called shared_ptr exactly because of that it represents shared ownership. I think you're again confused about what's the rationale of the design and what's the consequence of it. Shared ownership is the rationale, being an 'atomic safe reference counting implementation which provides no guarantees to its pointee' is a consequence.

I might add that in AFIO I use the typedef handle_ptr. You don't think to think shared anything, just use afio::handle_ptr and stop worrying about shared_ptr. It's an implementation detail, and not important.

What kind of ownership with regards to the handle does the afio::handle_ptr give your user? Unique ownership or shared ownership? Regards Hartmut --------------- http://boost-spirit.com http://stellar.cct.lsu.edu

On 31 Aug 2015 at 13:06, Agustín K-ballo Bergé wrote:

Let me add that an "atomic safe reference counting implementation" is just one of the possible implementations. Although I am not aware of any concrete implementation doing things differently, there are alternative possible implementations like chaining, and it is the reason for having both `use_count` and `unique`.

This is exactly why I supplied a afio::handle_ptr typedef. I don't expect to ever have to change it from a shared_ptr, but if it ever came up that some STL out there did things differently, I could swap in my own implementation very easily. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 28 Aug 2015 6:34 am, "Thomas Heller" <thom.heller@gmail.com> wrote:

On 08/27/2015 04:04 AM, Niall Douglas wrote: <snip> A lot of things about future vs. shared_future... </snip>

...

Thoughts?

Ok, after reading all the other messages in this and other threads I am

starting to understand what the real problem is (all this is IMHO, of course):

The biggest mistake in your design is that of the functions taking

"preconditions". This brought you all this mess. The problem is, those are actually not preconditions but actually arguments to the function, more specifically, your "precondition" is always (?) the handle (or a future to the handle so to speak) to the actual operation that is being performend for which you need to get the value. That all is a result of the non-existing single responsibility pattern in your code. After getting rid of that, you will see that most of the points you brought up against using std::future (or boost::future) will actually go away.

Here is my concrete suggestion: - Let your async functions return a future to their result! (Not a tuple to the handle and the result...) - Don't use futures as arguments to your async functions. Say what you want as input!

Let's take on your read example... and suppose we have the following two function signatures:

future<handle> open(std::string name); future<std::vector<char>> read(handle const & h, std::size_t bytes, std::size_t offset);

See how easy, clear and succinct those functions are? Almost self explaining without any further documentation.

Couldn't possibly agree more. I was going to focus on this issue on my review. Thanks for explaining it so well. -- gpd

On 28 Aug 2015 at 10:16, Thomas Heller wrote:

I took the effort to also reimplement your hello world example: https://gist.github.com/sithhell/f521ea0d818d168d6b35

Firstly thank you very much for your gist. This makes it much easier to understand what you are talking about. My very first thought was that you are using reference capturing lambdas in a concurrent use case which may be fine in internal implementation, but is a serious no-no to ever place upon a library's end users. People regularly get waits and wakeups wrong, and I have seen code bases wrecked by memory corruption and race conditions induced by reference capturing lambdas used in concurrent situations. For me this is an absolute red line which cannot be crossed. I will never, *ever* agree to a library API design which makes end users even feel tempted to use reference capturing semantics in a concurrency context. Period. They are banned as they are nuclear bombs in the hands of the typical C++ programmer. So let's replace all your reference captures with shared_ptr or value captures and thereby making them safe. I have forked your gist with these changes to: https://gist.github.com/ned14/3581d10eacb6a6dd34bf As I mentioned earlier, any of the AFIO async_* free functions just expand into a future.then(detail::async_*) continuation which is exactly what you've written. So let me collapse those for you: https://gist.github.com/ned14/392461b85e73add13e30 This is almost identical to my second preference API for AFIO, and the one Gavin Lambert suggested in another thread. As I mentioned in that thread with Gavin, I have no problem at all with this API design apart from the fact you need to type a lot of depends() and I suspect it will be a source of unenforced programmer error. I feel the AFIO API as submitted has a nice default behaviour which makes it easy to follow the logic being implemented. Any depends() which is a formal announcement that we are departing from the default stand out like a sore thumb which draws the eye to giving it special attention as there is a fork in the default logic. Under your API instead, sure you get unique futures and shared futures and that's all very nice and everything. But what does the end user gain from it? They see a lot more verbiage on the screen. They find it harder to follow the flow of the logic of the operations and therefore spot bugs or maintain the logic. I am seeing lots of losses and few gains here apart from meeting some supposed design principle about single responsibility which in my opinion is a useful rule of thumb for inexperienced programmers, but past that isn't particularly important. I however am probably being a little unfair here. I reduced your original to something AFIO like and something I previously gave deep consideration to adopting before rejecting it, and you were probably actually advocating that people manually write out all the .then() logic instead of using free functions which expand into exactly the same thing. If you really strongly feel that writing out the logic using .then() is very important, I can add that no problem. Right now the continuation thunk types live inside a detail namespace, but those can be moved to public with very little work. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Am 29.08.2015 10:15 nachm. schrieb "Thomas Heller" <thom.heller@gmail.com>:

On 08/29/2015 02:05 AM, Niall Douglas wrote:

...

On 28 Aug 2015 at 10:16, Thomas Heller wrote:

...

I took the effort to also reimplement your hello world example: https://gist.github.com/sithhell/f521ea0d818d168d6b35

Firstly thank you very much for your gist. This makes it much easier to understand what you are talking about.

My very first thought was that you are using reference capturing lambdas in a concurrent use case which may be fine in internal implementation, but is a serious no-no to ever place upon a library's end users. People regularly get waits and wakeups wrong, and I have seen code bases wrecked by memory corruption and race conditions induced by reference capturing lambdas used in concurrent situations.

All true. This contrived example is not really concurrent though, but

whatever. Please, stop treating your users as they wouldn't know what they do.

...

For me this is an absolute red line which cannot be crossed. I will never, *ever* agree to a library API design which makes end users even feel tempted to use reference capturing semantics in a concurrency context. Period. They are banned as they are nuclear bombs in the hands of the typical C++ programmer.

Sure, whatever, this is just derailing ... You do realize though that you

This is by giving up your own promoted "Don't pay for what you don't use"

demote your users library to idiots not knowing what they do? Despite promoting your library as something very niche only used by experts? principle. But let's move on...

...

So let's replace all your reference captures with shared_ptr or value captures and thereby making them safe. I have forked your gist with these changes to:

https://gist.github.com/ned14/3581d10eacb6a6dd34bf

As I mentioned earlier, any of the AFIO async_* free functions just expand into a future.then(detail::async_*) continuation which is exactly what you've written. So let me collapse those for you:

https://gist.github.com/ned14/392461b85e73add13e30

Where does error handling happen here? How would the user react to

errors? What does depends do (not really clear from the documentation)? How do I handle more than one "precondition"? How do I handle "preconditions" which have nothing to do with file I/O?

Which ultimately leads me to the question: Why not just let the user

standard wait composure mechanisms and let the functions have arguments that they really operate on?

All questions would have a simple answer then.

NB: I am still not sure what afio::future<>::then really implies, does it invalidate the future now? When will the continuation be executed, when the handler future is ready or when the "value" future is ready? What does that mean for my precondition?

Yet another problem I can't spin my head around... What is the result of "await af" where af is some afio::future<T>? The handle? Some T? How do I make a difference between a function really returning me a handle or one which just has the handle returned as a side effect?

...

This is almost identical to my second preference API for AFIO, and the one Gavin Lambert suggested in another thread.

As I mentioned in that thread with Gavin, I have no problem at all with this API design apart from the fact you need to type a lot of depends() and I suspect it will be a source of unenforced programmer error. I feel the AFIO API as submitted has a nice default behaviour which makes it easy to follow the logic being implemented. Any depends() which is a formal announcement that we are departing from the default stand out like a sore thumb which draws the eye to giving it special attention as there is a fork in the default logic.

That's exactly the problem: You have a default logic for something that

isn't really necessary, IMHO. The async operation isn't an operation on the future, but on the handle. And to repeat myself: That's my problem with that design decision.

...

Under your API instead, sure you get unique futures and shared futures and that's all very nice and everything. But what does the end user gain from it?

1. Clear expression of intent 2. Usage of standard utilities for wait composures 3. Transparent error handling 4. "Don't pay for something you don't use"

...

They see a lot more verbiage on the screen. They find it harder to

...

follow the flow of the logic of the operations and therefore spot bugs or maintain the logic.

Verbosity isn't always a bad thing. Instead of trying to guess what is

the best default for your users stop treating them as they would not know what they do.

...

I am seeing lots of losses and few gains here apart from meeting some

...

principle about single responsibility which in my opinion is a useful rule of thumb for inexperienced programmers, but past that isn't particularly important.

You do realize that this is highly contradicting to anything else you just said in your mail? You claim to have your library designed for the inexperienced user not knowing what they do, yet your design choice violates principles that are a nice "rule of thumb for inexperienced

supposed design programmers"?

Beside that, single responsibility should *always* be pursued. It makes your code testable, composable and easier to reason about.

...

I however am probably being a little unfair here. I reduced your original to something AFIO like and something I previously gave deep consideration to adopting before rejecting it, and you were probably actually advocating that people manually write out all the .then() logic instead of using free functions which expand into exactly the same thing.

Right, dependency handling is something inherhent to the users code. Anything you'll come up with inside of your library will be the wrong thing in some cases and then you'll have a problem.

...

If you really strongly feel that writing out the logic using .then() is very important, I can add that no problem. Right now the continuation thunk types live inside a detail namespace, but those can be moved to public with very little work.

Niall

On 29 Aug 2015 at 22:30, Thomas Heller wrote:

...

For me this is an absolute red line which cannot be crossed. I will never, *ever* agree to a library API design which makes end users even feel tempted to use reference capturing semantics in a concurrency context. Period. They are banned as they are nuclear bombs in the hands of the typical C++ programmer.

Sure, whatever, this is just derailing ... You do realize though that you demote your users library to idiots not knowing what they do? Despite promoting your library as something very niche only used by experts? This is by giving up your own promoted "Don't pay for what you don't use" principle. But let's move on...

No, this stuff is at the core of where we are diverging in approach and why the AFIO API is so displeasing to you. Where I am coming from is this: "If the cost of defaulting to guaranteed memory safety to the end user is less than 0.5% overhead, I will default to guaranteed memory safety." Which in the case of a shared_ptr relative to ANY filing system operation is easily true, hence the shared_ptr. I am also assuming that for power end users who really do care about this, it is trivial for them to extract from the shared_ptr an afio::handle&. The only difference here from my perspective is what is the default. I appreciate that from your perspective, it's a question of good design principles, and splashing shared_ptr all over the place is not considered good design. For the record, I *agree* where the overhead of a shared_ptr *could* be important - an *excellent* example of that case is std::future<T> which it is just plain stupid that those use memory allocation at all, and I have a non memory allocating implementation which proves it in Boost.Outcome. But for AFIO, where the cost of a shared_ptr will always be utterly irrelevant compared to the operation cost, this isn't an issue. I also have a second big reason I haven't mentioned until now for the enthusiasm for shared_ptr. In addition to needing to bind internal workaround objects to lifetimes of things, AFIO's dispatcher and handle will be bound into other languages such as .NET, C, Python etc. All these languages use reference counting as part of their garbage collection, and therefore if the lifetime of AFIO objects is also reference count managed it makes getting the interop in situations where say a Python interpreter is loaded into the local process much easier. I expect after the lightweight futures refactor of AFIO to add John Bandela's CppComponents support to AFIO. This lets you wrap AFIO objects into Microsoft COM objects, and from there rebinds into any other programming language is quite easy. This is a BIG reason I am so absolute regarding ABI stability. Microsoft COM imposes stern requirements on the ABI layer. A big reason behind Monad is solving C++ exception transport through Microsoft COM, hence my predilection for APIs which are noexcept returning monad<T> as those are 100% Microsoft COM compatible.

...

So let's replace all your reference captures with shared_ptr or value captures and thereby making them safe. I have forked your gist with these changes to:

https://gist.github.com/ned14/3581d10eacb6a6dd34bf

As I mentioned earlier, any of the AFIO async_* free functions just expand into a future.then(detail::async_*) continuation which is exactly what you've written. So let me collapse those for you:

https://gist.github.com/ned14/392461b85e73add13e30

Where does error handling happen here? How would the user react to errors? What does depends do (not really clear from the documentation)? How do I handle more than one "precondition"? How do I handle "preconditions" which have nothing to do with file I/O?

I literally collapsed your original code as-is with no additional changes, so the answers to all the above are identical to your original code. The depends(a, b) function is simply something like: a.then([b](future){ return b; }) Preconditions are nothing special. Anything you can schedule a continuation onto is a precondition.

Which ultimately leads me to the question: Why not just let the user standard wait composure mechanisms and let the functions have arguments that they really operate on? All questions would have a simple answer then.

In my collapsed form of your original code, all the futures are std::future same as yours, and hence the need for a depends() function to swap the item with continues the next operation with a handle on which to do the operation. And there is absolutely nothing stopping you from writing out the continuations by hand, intermixed to as much or as little degree the default expanded continuations. I'm thinking in a revised tutorial for AFIO this probably ought to be on the second page to show how the async_* functions are simply less-typing time savers. I'm thinking, thanks to this discussion of ours, that I'll move those default expansions out of their detail namespace. People can use expanded or contracted forms as they prefer.

NB: I am still not sure what afio::future<>::then really implies, does it invalidate the future now? When will the continuation be executed, when the handler future is ready or when the "value" future is ready? What does that mean for my precondition?

afio::future<T> simply combines, for convenience, a future handle with a future T. Both become ready simultaneously with identical errored or excepted states if that is the case. You can think of it as if std::future<std::pair<handle, T>> but without the problems of ABI stability.

...

This is almost identical to my second preference API for AFIO, and the one Gavin Lambert suggested in another thread.

As I mentioned in that thread with Gavin, I have no problem at all with this API design apart from the fact you need to type a lot of depends() and I suspect it will be a source of unenforced programmer error. I feel the AFIO API as submitted has a nice default behaviour which makes it easy to follow the logic being implemented. Any depends() which is a formal announcement that we are departing from the default stand out like a sore thumb which draws the eye to giving it special attention as there is a fork in the default logic.

That's exactly the problem: You have a default logic for something that isn't really necessary, IMHO. The async operation isn't an operation on the future, but on the handle. And to repeat myself: That's my problem with that design decision.

Would you be happy if AFIO provides you the option of programming AFIO exclusively using the expanded continuations form you posted in your gist? In other words, I would be handing over the decision to the end user. It would be entirely up to them which form, or mix of forms, they choose.

...

Under your API instead, sure you get unique futures and shared futures and that's all very nice and everything. But what does the end user gain from it?

1. Clear expression of intent 2. Usage of standard utilities for wait composures 3. Transparent error handling 4. "Don't pay for something you don't use"

I appreciate from your perspective that the AFIO API is not designed using 100% standard idiomatic practice according to your experience. However, if as I mentioned above, if I give you the end user the choice to program AFIO *exactly* as your gist proposed (apart from the shared_ptr<handle>), would you be happy with the AFIO design?

...

They see a lot more verbiage on the screen. They find it harder to follow the flow of the logic of the operations and therefore spot bugs or maintain the logic.

Verbosity isn't always a bad thing. Instead of trying to guess what is the best default for your users stop treating them as they would not know what they do.

Excellent library design is *always* about defaults. I default to safest use practice where percentage of runtime and cognitive overhead allows me. I always provide escape hatches for the power programmer to escape those defaults if they choose, but wherever I am able I'm not going to default to semantics which the average C++ programmer is going to mess up.

...

I am seeing lots of losses and few gains here apart from meeting some supposed design principle about single responsibility which in my opinion is a useful rule of thumb for inexperienced programmers, but past that isn't particularly important.

You do realize that this is highly contradicting to anything else you just said in your mail? You claim to have your library designed for the inexperienced user not knowing what they do, yet your design choice violates principles that are a nice "rule of thumb for inexperienced programmers"?

Even very experienced programmers regularly mess up memory safety when handed too many atomic bombs as their primitives, and debugging memory corruption is an enormous time waste. If you like programming exclusively using atomic bombs, C and assembler is the right place to be, not C++. Atomic bomb programming primitives are a necessary evil, and the great thing about C++ is it gives you the choice, and when a safer default introduces a big overhead (e.g. std::future over std::condition_variable) you can offer options to library end users. However when the overhead of using safer programming and design patterns is unimportant, you always default to the safer design pattern (providing an escape hatch for the power programmer where possible).

Beside that, single responsibility should *always* be pursued. It makes your code testable, composable and easier to reason about.

There are many ways of making your code testable, composable and easier to reason about. You can follow rules from a text book of course, and that will produce a certain balance of tradeoffs. Is following academic and compsci theory always going to produce a superior design to not following it? For most programmers, probably yes most of the time. However once you reach a certain level of experience, I personally believe the answer is no: academic and compsci theory introduces a hard-to-see rigidity of its own, and it has the particular failing of introducing a dogmatic myopia to its believers. My library design is almost exclusively gut instinct driven with no principles whatsoever. Its single biggest advantage is a lack of belief in any optimality at all, because the whole thing is completely subjective and I wake up on different days believing different things, and the design evolves accordingly haphazardly. Its biggest failings are explainability and coherency and you probably saw that in the documentation because I don't really know why I choose a design until forced to explain myself in a review like this. And thank you Thomas for enabling me to explain myself.

...

If you really strongly feel that writing out the logic using .then() is very important, I can add that no problem. Right now the continuation thunk types live inside a detail namespace, but those can be moved to public with very little work.

Can I get a yay or nay to the idea of giving the end user the choice to program AFIO using expanded continuations as per your gist? If you yay it, I think this long discussion will have been extremely valuable (and please do reconsider your vote if you agree). Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On 8/30/2015 1:01 PM, Niall Douglas wrote:

I appreciate that from your perspective, it's a question of good design principles, and splashing shared_ptr all over the place is not considered good design. For the record, I*agree* where the overhead of a shared_ptr*could* be important - an*excellent* example of that case is std::future<T> which it is just plain stupid that those use memory allocation at all, and I have a non memory allocating implementation which proves it in Boost.Outcome. But for AFIO, where the cost of a shared_ptr will always be utterly irrelevant compared to the operation cost, this isn't an issue.

Let's get this memory allocation concern out of the way. One just can't have a conforming implementation of `std::future` that does not allocate memory. Assume that you could, by embedding the storage for the result (value-or-exception) inside either of the `future/promise`: 1) Allocator support: `std::future::share` transfer ownership of the (unique) future into a shared future, and thus necessarily requires allocation [see below]. This allocation ought to be done with the allocator/pmr supplied to the `std::promise` constructor. You then have a few options: a) Keeping this allocator around so that `std::future::share` can use it, this is the standard conforming option. This means type-erasure in some way or another, which accounts to doing memory allocation when the size of the allocator is greater than the size of some hard-coded small buffer (properly aligned, etc). b) You can ditch allocator support, which is an option to the majority of the population but the standard, and resort to `std::allocator`. You now have a problem, because you have no control over the allocation process, and thus you cannot mitigate the cost of it by using pools, stack buffers, etc. However `std::shared_future` usage should be rare, so this might not be that big of a deal. c) You can try to change the standard so that it is `std::future::share` that takes an allocator, and guarantee no memory allocation anywhere else. This would be a reasonable approach under certain conditions. The reason `std::shared_future` cannot make use of embedded storage, thus necessarily requiring allocation, has to do with lifetime and thread-safety. `std::shared_future::get` returns a reference to the resulting value, which is guaranteed to be valid for as long as there is at least one instance of `std::shared_future` around. If embedded storage were to be used, it would imply moving the location of the resulting value when the instance holding it goes away. This can happen in a separate thread, as `std::shared_future` and `std::shared_future::get` are thread-safe. All in all it would lead to the following scenario: std::shared_future<T> s = get_a_shared_future_somehow(); T const& r = s.get(); std::cout << r; // potentially UB, potentially race 2) Type requirements: The standard places very few restrictions on which types can be used with asynchronous results, those are (besides Destructible) - `std::future<T>::get` requires `T` be move constructible, - `std::promise<T>::set_value` requires `T` be copy/move-constructible (some individuals are considering proposing `emplace_value`, which would lift this restriction), - `std::shared_future<T>::get` requires nothing. The use of embedded storage increases those restrictions: a) `T` has to be move-constructible, which is fine today as it is already required implicitly by `std::promise`, `std::packaged_task`, etc. I'm only mentioning this as there's interesting in dropping this requirement, to increase consistency with regard to emplace construction support. b) `T` has to be nothrow-move-constructible, as moving any of `std::promise`, `std::future`, `std::shared_future` is `noexcept`. c) If synchronization is required when moving the result from one embedded-storage to the other, `T` has to be trivially-move-constructible, as executing user code could potentially lead to a deadlock. This might be tractable by using atomics, the atomic experts would know (I am not one of them). This could also be addressed by transactional memory, but this would only further increase the restrictions on types that could be used (although I am not a TM expert either). So far we know for sure that a standard-conforming non-allocating `std::promise<T>/future<T>` pair can be implemented as long as: - `T` is trivially-move-constructible - The allocator used to construct the promise is `std::allocator`, or that it is a viable candidate for small buffer optimization. Such an implementation would use embedded storage under those partly-runtime conditions, which is quite a restricted population but still promising as it covers the basic `std::future<int>` scenario. But as it usually happens, it is a tradeoff, as such an implementation would have to incur synchronization overhead every time either of the `std::future/promise` is moved for the case where the `std::future` is retrieved before the value is ready, which in my experience comprises the majority of the use cases. But for completeness, let's analyze the possible scenarios. It always starts with a `std::promise`, which is the one responsible for creating the shared-state. Then either of these could happen: I) The shared-state is made ready by providing the value-or-exception to the `std::promise`. II) The `std::future` is retrieved from the `std::promise`. In the case where (I) happens before (II), no extra synchronization is needed, since the `std::promise` can simply transfer the result to the `std::future` during (II). Once the result has been provided, there is no further communication between `std::promise` and `std::future`. This represents the following scenario: std::promise<int> p; p.set_value(42); std::future<int> f = p.get_future(); which is nothing but a long-winded overhead-riddled way of saying: std::future<int> f = std::make_ready_future(42); In the case where (II) happens before (I), every time either one of the `std::future` or `std::promise` moves it has to notify the other one that it has gone to a different location, should it require to contact it. Again, this would happen for as long as the shared-state is not made ready, and represents the following scenario: std::promise<int> p; std::future<int> f = p.get_future(); std::thread t([p = std::move(p)] { p.set_value(42); }); which is the poster-child example of using `std::promise/future`. Finally, in Lenexa the SG1 decided to accept as a defect LWG2412, which allows for (I) and (II) to happen concurrently (previously undefined behavior). This appears to not have yet moved forward by LWG yet. It represents the following scenario: std::promise<int> p; std::thread t([&] { p.set_value(42); }); std::future<int> f = p.get_future(); which is in reality no different than the previous scenario, but which an embedded storage `std::promise` implementation needs to address with more synchronization. Why is this synchronization worth mention at all? Because it hurts concurrency. Unless you are in complete control of every piece of code that touches them and devise it so that no moves happen, you are going to see the effects of threads accessing memory of other threads with all what it implies. But today's `std::future` and `std::promise` are assumed to be cheaply movable (just a pointer swap). You could try to protect from it by making `std::future` and `std::promise` as long as a cache line, and even by simply using dynamic memory allocation for them together with an appropriate allocator specifically designed to aid whatever use case you could have where allocation time is a constraint. And finally, let's not forget that the Concurrency TS (or actually the futures continuation section of it) complicates matters even more. The addition of `.then` requires implementations to store an arbitrary Callable around until the future to which it was attached becomes ready. Arguably, this Callable has to be stored regardless of whether the future is already ready, but I'm checking the final wording and it appears that you can as-if run the continuation in the calling thread despite not being required (and at least discouraged in an initial phase). Similar to the earlier allocator case, this Callable can be whatever so it involves type-erasure in some way or another, which will require memory allocation whenever it doesn't fit within a dedicated small buffer object. To sum things up (based on my experience and that of others which I had a chance to discuss the subject), a non-allocating quasi-conformant `std::future/promise` implementation would cater only to a very limited set of types in highly constrained scenarios where synchronization overhead is not a concern. In real word scenarios, and specially those that rely heavily on futures due to the use of continuations, time is better spent by focusing in memory allocation schemes (the actual real concern after all) by using the standard mechanism devised to tend to exactly those needs: allocators. I'll be interested in hearing your findings during your work on the subject. And would you want me to have a look at your implementation and come up with ways to "break it" (which is what I do best), you have just to contact me. Regards, -- Agustín K-ballo Bergé.- http://talesofcpp.fusionfenix.com

On 8/30/2015 5:06 PM, Niall Douglas wrote:

On 30 Aug 2015 at 15:05, Agustín K-ballo Bergé wrote:

...

On 8/30/2015 1:01 PM, Niall Douglas wrote:

...

I appreciate that from your perspective, it's a question of good design principles, and splashing shared_ptr all over the place is not considered good design. For the record, I*agree* where the overhead of a shared_ptr*could* be important - an*excellent* example of that case is std::future<T> which it is just plain stupid that those use memory allocation at all, and I have a non memory allocating implementation which proves it in Boost.Outcome. But for AFIO, where the cost of a shared_ptr will always be utterly irrelevant compared to the operation cost, this isn't an issue.

Let's get this memory allocation concern out of the way. One just can't have a conforming implementation of `std::future` that does not allocate memory. Assume that you could, by embedding the storage for the result (value-or-exception) inside either of the `future/promise`:

I'll just limit my comments to your text to what my Boost.Outcome library does if that's okay. I should stress before I begin that I would not expect my non-allocating futures to be a total replacement for STL futures, but rather a complement to them (they are in fact dependent on STL futures because they use them internally) which might be useful as a future quality-of-implementation optimisation if and only if certain constraints are satisified.

As long as we agree that a non-allocating standard-conformant future is an oxymoron...

7. future.wait() very rarely blocks in your use scenario i.e. most if not nearly all the time the future is ready. If you are blocking, the cost of any thread sleep will always dwarf the cost of any future.

`future::wait` should not be a concern, you just spawn a `condition_variable` and attach a fake continuation to the future that will wake it up. The cv will be stored in the thread stack, which is guaranteed to stay around since we will be sleeping. This allows the fake continuation to simply be `reference_wrapper`, for which you only need `sizeof(void*)` embedded storage. Since the (unique) future cannot be used concurrently with `wait`, there can only ever be at most one fake continuation. The situation gets trickier for `wait_for/until`, where you need to remove the fake continuation on a timeout without racing.

These circumstances are common enough in low latency applications such as ASIO and using them is a big win in ASIO type applications over STL futures. These circumstances are not common in general purpose C++ code, and probably deliver little benefit except maybe a portable continuations implementation on an older STL.

Have you tried a custom allocator in this scenario? Together with a decent implementation of `std::future`, which granted does not exist AFAIK.

...

And finally, let's not forget that the Concurrency TS (or actually the futures continuation section of it) complicates matters even more. The addition of `.then` requires implementations to store an arbitrary Callable around until the future to which it was attached becomes ready. Arguably, this Callable has to be stored regardless of whether the future is already ready, but I'm checking the final wording and it appears that you can as-if run the continuation in the calling thread despite not being required (and at least discouraged in an initial phase).

I read the postconditions as meaning:

if(future.is_ready()) callable(future); else store_in_promise_for_later(callable);

... which is what I've implemented.

I *do* allocate memory for continuations, one malloc per continuation added.

I assume this is just rushed pseudocode, but anyways with my pedant hat on: - Callable means _INVOKE_, if you are not using _INVOKE_ then you do not have a callable. You might have a function object at best. - You must do the _DECAY_COPY_ dance, even for the immediate execution case. Not doing it means different types, different overloads, different semantics, so you could end up computing the wrong return type. - The argument to the continuation shall be an xvalue for `std::future`, and a const lvalue for `std::shared_future`. - You must do "implicit unwrapping", ideally not in a naive way which would involve two memory allocations instead of one. - If you were to execute immediately and were in an "implicit unwrapping" case, you'd have to be careful to handle the invalid source case, and to catch any exception and store it in the returned future instead of letting it leave the scope. Finally, about executing immediately, an earlier draft of the TS specified the semantics in terms of launch policies, which implied execution of the continuation happening "as-if" on a new thread. That would preclude running the continuation immediately, as it would be observable. The current wording says "is called on an unspecified thread of execution", which technically allows to call the continuation right here right now. That's not necessarily a good idea, and was initially explicitly discouraged, since you are basically executing sequentially which was clearly designed to execute concurrently. If that were to be what you want, you could always do it (as shown in examples in earlier drafts): if (f.is_ready()) { ... f.get() ... } else { f.then(...); } Regards, -- Agustín K-ballo Bergé.- http://talesofcpp.fusionfenix.com

On 9/1/2015 9:35 AM, Giovanni Piero Deretta wrote:

On 30 Aug 2015 10:04 pm, "Agustín K-ballo Bergé" <kaballo86@hotmail.com> wrote:

...

On 8/30/2015 5:06 PM, Niall Douglas wrote:

...

[...]

...

...

7. future.wait() very rarely blocks in your use scenario i.e. most if not nearly all the time the future is ready. If you are blocking, the cost of any thread sleep will always dwarf the cost of any future.

`future::wait` should not be a concern, you just spawn a `condition_variable` and attach a fake continuation to the future that will wake it up. The cv will be stored in the thread stack, which is guaranteed to stay around since we will be sleeping. This allows the fake continuation to simply be `reference_wrapper`, for which you only need `sizeof(void*)` embedded storage. Since the (unique) future cannot be used concurrently with `wait`, there can only ever be at most one fake continuation.

This is unfortunately not true. Std::future::wait is const, so, without an explicit prohibition, multiple threads can call it as long as no other non const function is concurrently called. This is possibly a defect of the standard.

Indeed, that is correct. I overlooked that detail, and would have to check my code if I weren't using a single implementation of waits for both `future` and `shared_future`.

...

The situation gets trickier for `wait_for/until`, where you need to remove the fake continuation on a timeout without racing.

Also needed to implement wait any.

Once `wait` returns the shared-state is ready. You don't even need to remove the fake continuation pointer, it will never be looked up again.

A dismissible, efficiently implementable 'then' protocol could be the key for future composition across libraries. I would prefer if in this case the continuation were pointer sized to allow lock free implementations.

I don't think pointer sized continuations would buy you anything, you can attach multiple continuations to a single shared-state (via `shared_future`) so you will always end up needing memory allocation eventually. That doesn't stop you from attaching the pointer to the continuation decayed-copy in a lock free way. Regards, -- Agustín K-ballo Bergé.- http://talesofcpp.fusionfenix.com

On 9/1/2015 11:43 AM, Giovanni Piero Deretta wrote:

On 1 Sep 2015 2:15 pm, "Agustín K-ballo Bergé" <kaballo86@hotmail.com> wrote:

...

On 9/1/2015 9:35 AM, Giovanni Piero Deretta wrote:

...

On 30 Aug 2015 10:04 pm, "Agustín K-ballo Bergé" <kaballo86@hotmail.com> wrote:

...

...

The situation gets trickier for `wait_for/until`, where you need to

remove

...

...

...

the fake continuation on a timeout without racing.

Also needed to implement wait any.

Once `wait` returns the shared-state is ready. You don't even need to remove the fake continuation pointer, it will never be looked up again.

You do if you want to implement a wait_any interface that blocks until the first of a number of futures is ready. In that case you must be able to reissue another wait at a later time. Think 'select'.

I did not understood what you meant by "wait any" the first time around. I do now, but I still don't see why would you ever want to wait on an already ready future. If you do, it would just return immediately; it should not even touch callbacks, continuations, condition variables, etc.

...

...

A dismissible, efficiently implementable

'then' protocol could be the key for future composition across libraries. I would prefer if in this case the continuation were pointer sized to allow lock free implementations.

I don't think pointer sized continuations would buy you anything, you can attach multiple continuations to a single shared-state (via `shared_future`) so you will always end up needing memory allocation eventually. That doesn't stop you from attaching the pointer to the continuation decayed-copy in a lock free way.

Only if you use shared state. I'm looking at optimising unique futures. My shared future implementation currently is internally N unique futures plus a multiplexer.

You always have a shared state (the concept, in one form or the other), it is part of the specification of the semantics.

Also for waits you can always allocate the waiter/signaler on the stack. The tricky part is handling the race between a dismissal and a signal.

This is exactly the trickiness I was referring to for `wait_for/until` vs `wait`.

To implement the generic 'then' I do always heap allocate, but I put the callback inline with the returned future shared state. I use the pointer sized dismissible 'then' to chain the two shared states.

Nod, and after the allocation you can attach the pointer to the callback in a lock free way. You can further chain multiple continuations for a `shared_future` in a similar way. Regards, -- Agustín K-ballo Bergé.- http://talesofcpp.fusionfenix.com

On 9/1/2015 1:23 PM, Giovanni Piero Deretta wrote:

On 1 Sep 2015 4:06 pm, "Agustín K-ballo Bergé" <kaballo86@hotmail.com> wrote:

...

On 9/1/2015 11:43 AM, Giovanni Piero Deretta wrote:

...

On 1 Sep 2015 2:15 pm, "Agustín K-ballo Bergé" <kaballo86@hotmail.com> wrote:

...

On 9/1/2015 9:35 AM, Giovanni Piero Deretta wrote:

...

On 30 Aug 2015 10:04 pm, "Agustín K-ballo Bergé" <kaballo86@hotmail.com

...

...

...

wrote:

...

...

The situation gets trickier for `wait_for/until`, where you need to

remove

...

...

...

the fake continuation on a timeout without racing.

Also needed to implement wait any.

Once `wait` returns the shared-state is ready. You don't even need to

remove the fake continuation pointer, it will never be looked up again.

You do if you want to implement a wait_any interface that blocks until

the

...

...

first of a number of futures is ready. In that case you must be able to reissue another wait at a later time. Think 'select'.

I did not understood what you meant by "wait any" the first time around. I do now, but I still don't see why would you ever want to wait on an already ready future. If you do, it would just return immediately; it should not even touch callbacks, continuations, condition variables, etc.

Uh? You do not wait on a ready future. You might wait for one of the other futures that are not ready.