[RPC] Boost.Asio to write *concurrency ready* code
On 27 Aug 2013 at 12:00, boost-request@lists.boost.org wrote:
I am *not sure* if anyone else has made such discovery, anyway, comment is welcomed.
What you have discovered is well known I am afraid. Indeed, proposed Boost.AFIO from this year's GSoC implements an asynchronous chainable closure execution engine on ASIO, and you can read documentation on how AFIO is equivalent to Google's and Microsoft's WG21 closure execution engine proposals here: https://ci.nedprod.com/job/Boost.AFIO%20Build%20Documentation/Boost.AF IO_Documentation/doc/html/afio/quickstart/closure_execution_engines/ho w_boost_afio_relates_to_exisit.html Paul did up a great matrix mapping APIs from those engines to AFIO equivalents at https://ci.nedprod.com/job/Boost.AFIO%20Build%20Documentation/Boost.AF IO_Documentation/doc/html/afio/quickstart/closure_execution_engines/ho w_boost_afio_relates_to_exisit/comparison_of_afio_with_other_cl.html and https://ci.nedprod.com/job/Boost.AFIO%20Build%20Documentation/Boost.AF IO_Documentation/doc/html/afio/quickstart/closure_execution_engines/ho w_boost_afio_relates_to_exisit/other_useful_comparisons.html. If we get time before GSoC closes, we may just go ahead and implement N3562 (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3562.pdf) using AFIO as it would be incredibly trivial to do so. However, the Boost community only voted closure execution support as third most important - they ranked directory change monitoring as highest, so that's what me and Paul are implementing now. (FYI to those interested, we're going to deliver some *seriously* performant directory monitoring. Testing of our direct kernel syscall based implementation suggests watching a 1 million file entry directory for changes will actually work on Windows, albeit pegging a 3.5Ghz 4 core machine at 75-85% of all CPUs. Linux sadly lacks the syscalls to match Windows, but a 100,000 entry directory may well work just fine) Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
2013/8/28 Niall Douglas
On 27 Aug 2013 at 12:00, boost-request@lists.boost.org wrote:
I am *not sure* if anyone else has made such discovery, anyway, comment is welcomed.
What you have discovered is well known I am afraid. Indeed, proposed
Don't say "afraid", I am actually very very happy to see that this ideal was well known. I kinda think that what they called *closer execution engine* can change the way we write softwares. But I am afraid I don't understand why they wrote Boost.AFIO instead of helping Boost.Asio.
Boost.AFIO from this year's GSoC implements an asynchronous chainable closure execution engine on ASIO, and you can read documentation on how AFIO is equivalent to Google's and Microsoft's WG21 closure execution engine proposals here:
So AFIO is an extension to asio ? That helped some sort of explanations ...... Again, I don't understand why they wrote Boost.AFIO instead of helping Boost.Asio.
2013/8/28 Niall Douglas
On 27 Aug 2013 at 12:00, boost-request@lists.boost.org wrote:
I am *not sure* if anyone else has made such discovery, anyway, comment is welcomed.
What you have discovered is well known I am afraid. Indeed, proposed Boost.AFIO from this year's GSoC implements an asynchronous chainable closure execution engine on ASIO, and you can read documentation on how AFIO is equivalent to Google's and Microsoft's WG21 closure execution engine proposals here:
https://ci.nedprod.com/job/Boost.AFIO%20Build%20Documentation/Boost.AF IO_Documentation/doc/html/afio/quickstart/closure_execution_engines/ho w_boost_afio_relates_to_exisit.html
Paul did up a great matrix mapping APIs from those engines to AFIO equivalents at https://ci.nedprod.com/job/Boost.AFIO%20Build%20Documentation/Boost.AF IO_Documentation/doc/html/afio/quickstart/closure_execution_engines/ho w_boost_afio_relates_to_exisit/comparison_of_afio_with_other_cl.html and https://ci.nedprod.com/job/Boost.AFIO%20Build%20Documentation/Boost.AF IO_Documentation/doc/html/afio/quickstart/closure_execution_engines/ho w_boost_afio_relates_to_exisit/other_useful_comparisons.html.
If we get time before GSoC closes, we may just go ahead and implement N3562 (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3562.pdf) using AFIO as it would be incredibly trivial to do so. However, the Boost community only voted closure execution support as third most important - they ranked directory change monitoring as highest, so that's what me and Paul are implementing now.
(FYI to those interested, we're going to deliver some *seriously* performant directory monitoring. Testing of our direct kernel syscall based implementation suggests watching a 1 million file entry directory for changes will actually work on Windows, albeit pegging a 3.5Ghz 4 core machine at 75-85% of all CPUs. Linux sadly lacks the syscalls to match Windows, but a 100,000 entry directory may well work just fine)
Niall
After digging into the proposed Boost.Afio library, I am afraid that Boost.Afio *does not* meet my standard of what a *good* library is. Boost.Afio seems to support both async and sync operation, but obversely they failed to design the proper API. Boost.Afio relay on Boost.Asio, but they seems to ignore the elegant Asio Style. They didn't even understand what makes Boost.Asio a good library. A good library is not about good features (although its importand) but about element API. For async file I/O, the asio style API like this boost::asio::fstream file(io_service); file.async_open("test.txt", "r", & handle_open ); that is way more powerfull that the complex auto mkfile(dispatcher->file(async_path_op_req(mkdir, "testdir/foo", file_flags::Create|file_flags::ReadWrite))); The api that Boost.Afio choose is too complex yet too stupid. You might argue on me that the job which Boost.Afio tries to solve is complex, but that doesn't be a good excuse to design such stupid API. Boost.Afio is stupid. As the "closure execution engine", Boost.Asio already have that, there is absolutly no point in investigating yet another Boost.Asio for what ever reasons.
On 28 Aug 2013 at 12:48, microcai wrote:
After digging into the proposed Boost.Afio library, I am afraid that Boost.Afio *does not* meet my standard of what a *good* library is.
Boost.Afio seems to support both async and sync operation, but obversely they failed to design the proper API. Boost.Afio relay on Boost.Asio, but they seems to ignore the elegant Asio Style. They didn't even understand what makes Boost.Asio a good library.
"They" would be me. I designed the AFIO API. I also implemented most of AFIO before the port to Boost by Paul under this year's Google Summer of Code funding.
A good library is not about good features (although its importand) but about element API.
For async file I/O, the asio style API like this
boost::asio::fstream file(io_service); file.async_open("test.txt", "r", & handle_open );
that is way more powerfull that the complex
auto mkfile(dispatcher->file(async_path_op_req(mkdir, "testdir/foo", file_flags::Create|file_flags::ReadWrite)));
The api that Boost.Afio choose is too complex yet too stupid. You might argue on me that the job which Boost.Afio tries to solve is complex, but that doesn't be a good excuse to design such stupid API.
Boost.Afio is stupid.
I designed the AFIO API. I also implemented most of AFIO. You are therefore implying that I am stupid. You are of course entitled to your opinion, but I would personally say that AFIO has an intuitive, Intellisense-friendly, elegant API which really leverages C++11 to achieve an ease of programming very hard to achieve (with performance) in C++03, and unparalleled cleanliness and elegance compared to any other async file i/o implementation. For comparison I would refer you to the libuv C library which is the nearest equivalent, and to the Windows IOCP implementation. I think you will find that AFIO compares *extremely* favourably to its nearest equivalents. It is more verbose in the single-issue API than it could be, but that is because AFIO is 100% a pure asynchronous batch API. The single issue APIs quite literally create a batch of one item, and call the batch API. The reason it is a pure asynchronous batch API is because AFIO is designed for exceptional file i/o performance: if you only need to write a few files mostly sequentially, STL iostreams are pretty good. If you need to scatter gun random writes and reads across dozens of files simultaneously and memory mapped file i/o isn't an option, STL iostreams - or Boost.Iostreams for that matter - will max out the CPU long before maxing out the storage. The reason AFIO "ignores the elegant Asio Style" is because the ASIO design is intended for non-seekable devices such as sockets and pipes. With file i/o, you very specifically need to specify the *ordering* *constraints* of reads and writes or else you will lose data (equally, not being able to say "I don't care about the commit order of this particular batch, so write it as fast as possible" is bad for performance). That can be done by hand with ASIO, but it involves lots of error and race condition prone verbose ordering control logic. AFIO relieves the programmer of all that marshalling work: you simply tell AFIO your dependencies, and AFIO figures out the ideal execution graph for you.
As the "closure execution engine", Boost.Asio already have that, there is absolutly no point in investigating yet another Boost.Asio for what ever reasons.
I think you don't understand what a closure execution engine does. AFIO implements a superset of Microsoft's closure engine (see http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3558.pdf), most specifically in the form of dependency tied completions in addition to independent completions. Most closure engines presented to WG21 to date only implement the latter, mainly because they make avoiding race conditions far easier especially if you are maintaining Abrahams exception safety guarantees. AFIO gets away with the former mainly through being so simple a design I can walk it line by line for logic errors, and as of last week I believe it to be now race condition free except in the case where one runs out of memory (unit testing for that is coming). As an example of what AFIO's unusual flexibility makes possible, I believe it makes crazy-but-sane ideas like Google's "C++ pipelines" (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3534.html) very tractable. It ought to be straightforward to combine AFIO and Boost.iostreams to implement N3534, though I admit I have no idea if performance would be acceptable. It would be extremely cool however. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
2013/8/29 Niall Douglas
On 28 Aug 2013 at 12:48, microcai wrote:
After digging into the proposed Boost.Afio library, I am afraid that Boost.Afio *does not* meet my standard of what a *good* library is.
Boost.Afio seems to support both async and sync operation, but obversely they failed to design the proper API. Boost.Afio relay on Boost.Asio, but they seems to ignore the elegant Asio Style. They didn't even understand what makes Boost.Asio a good library.
"They" would be me. I designed the AFIO API. I also implemented most of AFIO before the port to Boost by Paul under this year's Google Summer of Code funding.
A good library is not about good features (although its importand) but about element API.
For async file I/O, the asio style API like this
boost::asio::fstream file(io_service); file.async_open("test.txt", "r", & handle_open );
that is way more powerfull that the complex
auto mkfile(dispatcher->file(async_path_op_req(mkdir, "testdir/foo", file_flags::Create|file_flags::ReadWrite)));
The api that Boost.Afio choose is too complex yet too stupid. You might argue on me that the job which Boost.Afio tries to solve is complex, but that doesn't be a good excuse to design such stupid API.
Boost.Afio is stupid.
I designed the AFIO API. I also implemented most of AFIO. You are therefore implying that I am stupid.
Oh, I am sorry to hear that. Sorry that I didn't notice the fact that you designed the API. By saying , Boost.Afio is stupid, I don't mean the author is stupid. Don't over react. Most of the time I wrote stupid code. I would be rather happy if some time, by accident, I wrote some elegant code.
You are of course entitled to your opinion, but I would personally say that AFIO has an intuitive, Intellisense-friendly, elegant API which really leverages C++11 to achieve an ease of programming very hard to achieve (with performance) in C++03, and unparalleled cleanliness and elegance compared to any other async file i/o implementation. For comparison I would refer you to the libuv C library which is the nearest equivalent, and to the Windows IOCP implementation. I think you will find that AFIO compares *extremely* favourably to its nearest equivalents.
It is more verbose in the single-issue API than it could be, but that is because AFIO is 100% a pure asynchronous batch API. The single issue APIs quite literally create a batch of one item, and call the batch API.
The reason it is a pure asynchronous batch API is because AFIO is designed for exceptional file i/o performance: if you only need to write a few files mostly sequentially, STL iostreams are pretty good. If you need to scatter gun random writes and reads across dozens of files simultaneously and memory mapped file i/o isn't an option, STL iostreams - or Boost.Iostreams for that matter - will max out the CPU long before maxing out the storage.
The reason AFIO "ignores the elegant Asio Style" is because the ASIO design is intended for non-seekable devices such as sockets and pipes. With file i/o, you very specifically need to specify the *ordering* *constraints* of reads and writes or else you will lose data (equally, not being able to say "I don't care about the commit order of this particular batch, so write it as fast as possible" is bad for performance). That can be done by hand with ASIO, but it involves lots of error and race condition prone verbose ordering control logic. AFIO relieves the programmer of all that marshalling work: you simply tell AFIO your dependencies, and AFIO figures out the ideal execution graph for you.
Have you ever read the Boost.Asio document? Of-course asio supports seekable devices. You can write file_services to extent io_services if you want more features.
As the "closure execution engine", Boost.Asio already have that, there is absolutly no point in investigating yet another Boost.Asio for what ever reasons.
I think you don't understand what a closure execution engine does.
Yes, I don't understand what a closure execution engine does by your definition. But I do understand that, Boost.Asio may not be the same API as that Microsoft provided, it does have the most elegant API. It might not have the same feature set, but , hey , why we should care for that?
AFIO implements a superset of Microsoft's closure engine (see http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3558.pdf), most specifically in the form of dependency tied completions in addition to independent completions. Most closure engines presented to WG21 to date only implement the latter, mainly because they make avoiding race conditions far easier especially if you are maintaining Abrahams exception safety guarantees. AFIO gets away with the former mainly through being so simple a design I can walk it line by line for logic errors, and as of last week I believe it to be now race condition free except in the case where one runs out of memory (unit testing for that is coming).
As an example of what AFIO's unusual flexibility makes possible, I believe it makes crazy-but-sane ideas like Google's "C++ pipelines" (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3534.html) very tractable. It ought to be straightforward to combine AFIO and Boost.iostreams to implement N3534, though I admit I have no idea if performance would be acceptable. It would be extremely cool however.
By saying Boost.Afio's API is stupid, I mean the API encourage bad programing style, and punish a good programing style. If you look at the example of Boost.Afio, you feel like to be fucked by the code. I like to use stack-less coroutines, but with Boost.Afio, seems to be impossible.
On Thursday 29 August 2013 01:11:02 microcai wrote:
By saying Boost.Afio's API is stupid, I mean the API encourage bad programing style, and punish a good programing style.
If you look at the example of Boost.Afio, you feel like to be fucked by the code. I like to use stack-less coroutines, but with Boost.Afio, seems to be impossible.
I'd like to ask you to choose your words more carefully. I can understand the frustration and disappointment but please, keep the conversation civil and constructive. Foul language certainly doesn't help. http://www.boost.org/community/policy.html
I'd like to ask you to choose your words more carefully. I can understand the frustration and disappointment but please, keep the conversation civil and constructive. Foul language certainly doesn't help.
If I had read it carefully, it allow you to use Foul language on code. Did I miss some words? Cause English is not my first language.
On Thursday 29 August 2013 02:15:32 microcai wrote:
I'd like to ask you to choose your words more carefully. I can understand the frustration and disappointment but please, keep the conversation civil and constructive. Foul language certainly doesn't help.
If I had read it carefully, it allow you to use Foul language on code.
Code or not, the particular tone and wording of your statements were unnecessarily offending. Quotes from the discussion policy: Flame wars Personal insults, argument for the sake of argument, and all the other behaviors which fall into the "flame war" category are prohibited. Discussions should focus on technical arguments, not the personality traits or motives of participants. Culture In addition to technical skills, Boost members value collaboration, acknowledgement of the help of others, *and a certain level of politeness*.
Did I miss some words? Cause English is not my first language.
English is not my native language either. I assume, this is also the case for many other readers and participants of the list. It's one more reason to be more careful in a public conversation.
2013/8/29 Andrey Semashev
On Thursday 29 August 2013 02:15:32 microcai wrote:
I'd like to ask you to choose your words more carefully. I can understand the frustration and disappointment but please, keep the conversation civil and constructive. Foul language certainly doesn't help.
If I had read it carefully, it allow you to use Foul language on code.
Code or not, the particular tone and wording of your statements were unnecessarily offending. Quotes from the discussion policy:
Flame wars
Personal insults, argument for the sake of argument, and all the other behaviors which fall into the "flame war" category are prohibited. Discussions should focus on technical arguments, not the personality traits or motives of participants.
Culture
In addition to technical skills, Boost members value collaboration, acknowledgement of the help of others, *and a certain level of politeness*.
Did I miss some words? Cause English is not my first language.
English is not my native language either. I assume, this is also the case for many other readers and participants of the list. It's one more reason to be more careful in a public conversation.
What is wrong with you guys ( sorry, I assume that you are all male ) ? NO ONE, NOT A SINGLE ONE COMMENT ON MY CODE. Instead, attack on my words. Come on, I AM NOT POLITICS. I DON'T KNOW HOW TO PLEASE AUDIENCE. PLEASE COMMENT ON MY CODE. Boost.Afio IS OFF TOPIC HERE. I DISLIKE Boost.Afio's API, DON'T FORCE ME TO LIKE IT. AND DON'T TURN THIS THREAD INTO Boost.Afio wars. !!! Boost.Afio IS OFF TOPIC HERE. !!!
(I took the liberty of normalizing the case of the message I replied to) On Thu, Aug 29, 2013 at 03:46:40AM +0800, microcai wrote:
No one, not a single one comment on my code. Instead, attack on my words.
Yes, your code demonstrates a conversion of a dependency-less recursive algorithm into a sequence of chaining asynchronous operations. That's nice, and an example of how a generic enough library with a well-defined scope can be used to implement problems that can be solved by dispatching asynchronous events. If your original post had some underlying untold message like "is this a good way to do this" or "is there any unknown problems with my approach", it did not quite get through.
Come on, I am not politics. I don't know how to please audience.
A lot of us do not have English as a first language, but when in doubt, we try our best to not be insulting and avoid cursewords and other strong language. Communication with your peers in person in your own language is often a hard thing. Doing it over the internet with people from different cultures, backgrounds and native languages, requires a lot of patience, simplified language and great care in not using offensive concepts or words. For example, this mail is very mellow and refined from my initial stream of conciousness that first went into it. As a mail is a near-permanent record of ones words, it's often wise to choose them well and to not make enemies unnecessarily.
Please comment on my code. Boost.AFIO is off topic here.
Starting a thread does not confer any right to dictate how the thread evolves and forks. You can steer the direction by not participating in the parts of the thread you do not care about, or by participating more in the parts that you do care about.
I dislike Boost.AFIO's API, Don't force me to like it. And don't turn this thread into Boost.AFIO wars.
There is a difference between constructive criticism or suggestions on how to improve or adapt the interface, and just proclaiming it bad/ill-formed. Maybe you could elaborate on what parts of the interface are unsuitable, and how it could be expressed in a more elegant way, without losing the benefits it brings, particularly the sequencing of operations mentioned before?
Boost.AFIO is off topic here.
You're free to filter and killfile your mailbox in any way you want. -- Lars Viklund | zao@acc.umu.se
2013/8/29 Lars Viklund
(I took the liberty of normalizing the case of the message I replied to) On Thu, Aug 29, 2013 at 03:46:40AM +0800, microcai wrote:
No one, not a single one comment on my code. Instead, attack on my words.
Yes, your code demonstrates a conversion of a dependency-less recursive algorithm into a sequence of chaining asynchronous operations.
That's nice, and an example of how a generic enough library with a well-defined scope can be used to implement problems that can be solved by dispatching asynchronous events.
Yeah, it is nice. I don't understand why this pattern is not common. Is this pattern "only works on paper" kinds of stuff? I am enjoy this, it only makes the code just a little bit more complex but you gained so much. For those how concern about the overhead of handler allocate and invoking: well, if you can go parallel, a little bit overhead is comfortable.
Communication with your peers in person in your own language is often a hard thing. Doing it over the internet with people from different cultures, backgrounds and native languages, requires a lot of patience, simplified language and great care in not using offensive concepts or words.
For example, this mail is very mellow and refined from my initial stream of conciousness that first went into it. As a mail is a near-permanent record of ones words, it's often wise to choose them well and to not make enemies unnecessarily.
After a long discussion with zao and kalven on #boost , I apologizes to Niall for using afoul words. Thank you all guys that help me. Niall: I didn't say you are not stupid. Just because I disagree with your library does not mean you are stupid. please don't miss-interpret, For what ever I said about Bosot.Afio did not imply anything on you. I apologizes to Niall, because I should have said "Boost.Afio is not elegant." I should not use the word "stupid".
participants (4)
-
Andrey Semashev -
Lars Viklund -
microcai -
Niall Douglas