[afio][v1.30 "C++ Now edition" released]
Dear Boost, I am very pleased to announce the v1.30 release of proposed Boost AFIO, a Boost library implementing asynchronous file input/output which has been in the review queue since 2013. This is the version of AFIO which will be overviewed in my talk at C++ Now 2015. It also demonstrates the validity of the clang AST driven BindLib standalone Boost library conversion technique which will also be overviewed at C++ Now. Source distro: https://github.com/BoostGSoC13/boost.afio/archive/v1.30_Boost_v1.57.zi p Docs and CI dashboard: https://boostgsoc13.github.io/boost.afio/ Major new features in the v1.30 release: * Performance was improved by removing the transactional memory support in the v1.2 engine which is now compile time enabled only. Throughput now reaches up to 85% of raw ASIO and average latencies were reduced by 48%, yielding a 15 microsecond average latency with +/- 0.15 microsecond at a 95% confidence interval. This should enable AFIO to handle the latest generation PCIe SSDs with their 80 microsecond latencies reasonably well. * AFIO is now capable of complete standalone usage no Boost dependencies whatsoever, including for its unit testing via a Boost.Test emulation. * Due to its use of libclang AST generated template aliasing to allow user swapping of STL 11 implementation, minimum compilers required are now VS2013, GCC 4.7 and clang 3.2. All the VS2010 workarounds baggage has been eradicated. * All things related to async_data_op_req are now fully metaprogrammed with automatic generation by the compiler of suitable ASIO scatter-gather buffers from input types and STL containers with correct scatter-gather buffer coalescing where safe to do so. This automatic generation is user customisable via free function specialisation. * Support for relative paths from an open file descriptor was added. This lets you specify a file descriptor as a base from which a path fragment operates with the path resolution atomically performed in the kernel, and therefore the path of the base file descriptor may rapidly change safely. * AFIO is now optimally filing system race condition safe on all supported platforms, with per-API documentation of how much race safety is guaranteed per platform. A soak unit test verifies per commit that perfect race safety under severe and rapid filing system change is maintained. * A new afio::path type which very thinly wraps filesystem::path was added. The new afio specific path type makes the management of NT kernel path to Win32 path transition and conversion explicit, and therefore much harder to accidentally get wrong. A new normalise_path() function can convert a NT kernel path to a traditional Win32 path, complete with drive letter. * New platforms supported: Apple OS X and Android 5.0. * New filing system features support: All platforms: sparse file support, storage extent enumeration, extent deallocation ("hole punching"), volume metadata fetching (statfs), open file handle path discovery, hard link support, atomic relink (rename) support. Windows only: file symlinking, NT lock file performance flags (lock file performance is now not far from POSIX), workaround semantics for all the ways Windows will surprise you on the filing system such as files and directories which won't delete. AFIO now does a fair job of replicating POSIX filing system semantics on Windows (and better than Mingw/Cygwin presently). Linux: symlinks are now opened as file descriptors on kernels supporting that feature. * Doubled the size of the tutorial, with all the new material being non-toy examples. There is a particularly nifty "File locking via only atomic appending and hole punching" algorithm implementation in there, and it holds up surprisingly well compared to traditional file locks except on ZFS. Plus, it's completely portable. * Some utility routines in preparation for AFIO v1.4: a cryptographically strong random filename generator, a hugepage TLB STL allocator useful for efficient file copy buffers, a very highly optimised C++ 14 constexpr constructing arbitrary Hamming code SECDED template class (an amazing 22 cycles per byte for the 32784,32768 code), and secure to-hex and from-hex validating converters. * No shortage of bug fixes, helped hugely by additional unit testing for correct error handling. Detailed changelog can be found at https://boostgsoc13.github.io/boost.afio/doc/html/afio/release_notes.h tml. I'll tag a Boost 1.58 release on github once Boost 1.58 is out. This release took far longer than I originally intended mainly due to the extensive debugging of weird filing system races, and for six weeks now I've been working till 5am every night after my day job. I cannot tell you how glad I am to see this release out! My enormous thanks to my family who have put up with a lot for this release, and to my employer for a fairly dopey employee recently. Bed, here I come! Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On 18 Mar 2015, at 04:47, Niall Douglas <s_sourceforge@nedprod.com> wrote:
Dear Boost,
I am very pleased to announce the v1.30 release of proposed Boost AFIO, a Boost library implementing asynchronous file input/output which has been in the review queue since 2013.
[...]
This release took far longer than I originally intended mainly due to the extensive debugging of weird filing system races, and for six weeks now I've been working till 5am every night after my day job. I cannot tell you how glad I am to see this release out!
Hi Niall, You are obviously very proud of this library and I believe rightly so. You've worked hard on it and in terms of things like soak testing it probably sets a new high watermark for Boost libraries. You're also a highly active member of this community. So why is the library always "proposed".... When is there going to be a review? :) Regards, Pete
On 18 Mar 2015 at 9:41, Pete Bartlett wrote:
This release took far longer than I originally intended mainly due to the extensive debugging of weird filing system races, and for six weeks now I've been working till 5am every night after my day job. I cannot tell you how glad I am to see this release out!
You are obviously very proud of this library and I believe rightly so. You've worked hard on it and in terms of things like soak testing it probably sets a new high watermark for Boost libraries.
I'm proud of what I've been able to do with the very limited resources available to me, but I feel not especially proud of shipping a library which isn't well tested in absolute terms. For example, almost all the testing is functional rather than unit testing (lack of time to write more), plus the soak tests run for minutes instead of the 6 or 24 hours I'd much prefer and on only quad core hardware with local drives rather than the 16-32 core distributed over Samba/NFS network shares that it should be. I'm also not happy with the testing of the (too) many build options, the lack of testing of std::bad_alloc correctness (right now I claim exception safety except for bad_alloc), and there is no testing of localised resource leaks, only global testing. Ok, so like most Boost maintainers I see all that I failed to do before any of the achievements. Still, it could be a lot better than it is, and a bug in AFIO means data loss for users. In the end, we all have day jobs and families.
You're also a highly active member of this community. So why is the library always "proposed".... When is there going to be a review? :)
When a review manager volunteers themselves! Antony nearly did a few months ago, but this release was in progress and at that time I had no ABI nor API versioning in AFIO, so I could not propose an even semi-stable API at that time. In the v1.3 release, both ABI and API are versioned and there is a unit test which compiles two intentionally dimorphic and ABI incompatible copies of AFIO into the same translation unit and runs both sets of unit tests for each to make sure they don't collide. They don't, so I am now sure v1.3 of AFIO can coexist without issue with v1.4 of AFIO or any other version. So I now have no objections, and furthermore no major work will be done to AFIO until after CppCon in September. Still, asynchronous file i/o and race free filesystem is an extremely niche use case. I am not aware of anyone who uses AFIO in their code. That has surely to count against the wisdom of including it into Boost official as without a user base, who can really say if a design or implementation is any good during review? Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On 18/03/2015 05:47, Niall Douglas wrote:
Dear Boost,
I am very pleased to announce the v1.30 release of proposed Boost AFIO, a Boost library implementing asynchronous file input/output which has been in the review queue since 2013. This is the version of AFIO which will be overviewed in my talk at C++ Now 2015. It also demonstrates the validity of the clang AST driven BindLib standalone Boost library conversion technique which will also be overviewed at C++ Now.
Source distro: https://github.com/BoostGSoC13/boost.afio/archive/v1.30_Boost_v1.57.zi p
Docs and CI dashboard: https://boostgsoc13.github.io/boost.afio/
Hi, this is my first time actually looking at the documentation of the library. I think it would be interesting to develop a couple more things in the design or FAQ sections. Why use a specific asynchronous model with its own when_all etc.? There is a standard way of defining asynchronous tasks, which is getting extended by compiler support using await/yield_value, so why not do something using that? What about the relationship with ASIO? ASIO has its own event-driven main-loop system for asynchronous tasks, can AFIO use that and can ASIO use AFIO's future method for other types of I/O too?
On 19 Mar 2015 at 19:58, Mathias Gaunard wrote:
Hi, this is my first time actually looking at the documentation of the library.
I think it would be interesting to develop a couple more things in the design or FAQ sections.
Suggested topics are welcome.
Why use a specific asynchronous model with its own when_all etc.? There is a standard way of defining asynchronous tasks, which is getting extended by compiler support using await/yield_value, so why not do something using that?
It's not too obvious from first glance, but an async_io_op is a struct with a shared_future inside it which is publicly accessible called 'h'. So it uses the standard STL11 asynchronous model, just slightly hidden. The when_all etc is purely due to legacy reasons. Indeed the reason I couldn't use shared_future directly is purely legacy, back in the day I had to use shared_ptr<shared_future<T>> due to STL11 bugs, and it was easier to wrap that into an async_io_op, plus back in 2013 there was no fully functioning when_any/when_all implementation so I had to write my own and I needed type sugar to stop my when_any/all conflicting with the ones in Thread. I think the v1.2 engine replaced the shared_ptr<shared_future> with shared_future, so that was the major refactor for v1.2. Hence, especially as the Concurrent TS continuations API has finally firmed up, async_io_op can go away next release now, and an afio::future<T> with .then() etc replaces it. I've tried to bridge the APIs with a get() and a few other future like member functions to make porting the code easier, but it's going to be a major break, not least that the current central hash table is going away too in favour of a lock free linked list. So basically AFIO currently looks the way it does mostly due to legacy baggage and problems with C++ 11 back in 2013. Each release I refactor some of the baggage out. v1.3 was the first to drop < VS2013 support for example, it was a ton of work purging the VS2010 custom code paths. v1.4 will purge async_io_op + when_any/all and replace those with a proper C++ 17 compliant custom continuable future type which understands resumable functions. A big part of why that didn't enter v1.3 is because VS2015 and its resumable functions support I can test against isn't released yet, plus I was waiting for the Concurrency TS to firm up, it's only really stabilised these past six months.
What about the relationship with ASIO? ASIO has its own event-driven main-loop system for asynchronous tasks, can AFIO use that and can ASIO use AFIO's future method for other types of I/O too?
AFIO uses ASIO's event loop. It doesn't use the async_result idiom of ASIO because for file i/o you want strong graph ordering, and that's a pain to do with async_result. A more functional idiom, for which I chose future-promise, makes strong ordering much easier to program. Once I get Fibers/resumable function support in there next release, writing strongly graph ordered filing system algorithms ought to become optimally easy, and then I can get down to writing race free tree visitors etc which right now are very messy to write and debug. The AFIO-ASIO integration isn't deep right now, though of course ASIO's async_result can work with any arbitrary sync object, and that includes an async_io_op. Writing code (mostly completion lambdas) which arbitrates between the two is straightforward, if verbose and tedious. What is needed is zero copy kernel side support, and AFIO's design intentionally allows that (i.e. async write this gather list from this file to this socket all in the kernel using DMA). I'll be honest it's not high on my priority list given no one is using AFIO anyway, but potential deep AFIO-ASIO integration is very high on the radar. I will say though I'll need changes to ASIO to make that work, so it's not a trivial feature add. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On 19/03/2015 20:37, Niall Douglas wrote:
So basically AFIO currently looks the way it does mostly due to legacy baggage and problems with C++ 11 back in 2013. Each release I refactor some of the baggage out. v1.3 was the first to drop < VS2013 support for example, it was a ton of work purging the VS2010 custom code paths. v1.4 will purge async_io_op + when_any/all and replace those with a proper C++ 17 compliant custom continuable future type which understands resumable functions. A big part of why that didn't enter v1.3 is because VS2015 and its resumable functions support I can test against isn't released yet, plus I was waiting for the Concurrency TS to firm up, it's only really stabilised these past six months.
Thanks for the clarification. So v1.4 will be the interface that will be submitted for inclusion into Boost?
AFIO uses ASIO's event loop. It doesn't use the async_result idiom of ASIO because for file i/o you want strong graph ordering, and that's a pain to do with async_result. A more functional idiom, for which I chose future-promise, makes strong ordering much easier to program. Once I get Fibers/resumable function support in there next release, writing strongly graph ordered filing system algorithms ought to become optimally easy, and then I can get down to writing race free tree visitors etc which right now are very messy to write and debug.
I think it would be useful to be able to make use of the future-promise idiom for other ASIO code as well. Is it possible to use your futures for any ASIO async_result?
On 19 Mar 2015 at 21:26, Mathias Gaunard wrote:
So v1.4 will be the interface that will be submitted for inclusion into Boost?
v1.0 was the interface submitted for inclusion into Boost. It reflected the state of C++ in 2013 where XP compatibility was still very important, and therefore so was VS2010 compatibility. Every version thence is also submitted for inclusion until a review happens, which I assume will come back with a long list of changes needed anyway. If you're really asking when would I expect to finish the major refactoring, currently I expect v1.4 to be the last. However, I thought that about v1.3 during v1.2. A lot of this is outside my control of course. Had WG21 gone for Chris's concurrency proposal for example, I'd be refactoring in a very different direction. This is a big consequence of being the first C++ 11 only Boost library to enter the community review queue - you build in not just your own technical debt, but also the technical debt of the C++ language itself, *plus* that of the Microsoft compiler team, the libstdc++ team, the mingw team and so on. In fact the oldest compiler from v1.0 still supported identically is clang, because they had the most complete C++ 11 support the earliest, so clang 3.2 still works perfectly. As the ecosystem collectively reaches closer to conformance to the standard and therefore to each other, I can keep ripping out workaround patterns which used to make sense before but don't anymore. I'll put this another way. Back in v1.0 future-promise looked fast and buggy, but then the engine could only push 150k ops/sec. By v1.2 future-promise looked slow because it could now push 1.5m ops/sec. So by me ripping out the shared_ptr workarounds, I made the performance spotlight fall on other code. As I fix that, spotlights shine elsewhere and more fixes are needed. At some point both the C++ language ecosystem changes will settle down, and nails will stop sticking up to be hammered down. I'd expect API maturity to start then, same as any other Boost library. At that point I'll start building the embedded graph database AFIO is intended for.
AFIO uses ASIO's event loop. It doesn't use the async_result idiom of ASIO because for file i/o you want strong graph ordering, and that's a pain to do with async_result. A more functional idiom, for which I chose future-promise, makes strong ordering much easier to program. Once I get Fibers/resumable function support in there next release, writing strongly graph ordered filing system algorithms ought to become optimally easy, and then I can get down to writing race free tree visitors etc which right now are very messy to write and debug.
I think it would be useful to be able to make use of the future-promise idiom for other ASIO code as well.
ASIO already provides future-promise support via asio::use_future.
Is it possible to use your futures for any ASIO async_result?
Absolutely. Mine has just standard shared_futures, same as the ones ASIO uses. The custom afio::future in v1.4 will also work perfectly in ASIO, indeed I'll be providing the async_result specialisations so it's just as seamless to use. ASIO has Fiber support too, so Fiber should seamlessly multiplex ASIO and AFIO i/o for you. You just glue together the logic with resumable functions, and let the runtime figure it out for you. Which will be neat. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
participants (3)
-
Mathias Gaunard -
Niall Douglas -
Pete Bartlett