[Boost.AFIO] Formal Review
1. Should Boost.AFIO be accepted into Boost?
The topic of asynchronous file operation is very very important and quite critical in our days. Boost needs such a library but I fear *current* AFO is not there yet. I vote NO for its inclusion in Boost in its current form.
2. What is your evaluation of the design?
I think the intent are goods but there's some icky parts. The use of future and other asynchronous element in the API cudl be made more clear and easy. Thomas Heller made a compelling lsit of gotcha that shoudl be followed. Currently i had to fight against my habit of writing future absed code to get the stuff working.
3. What is your evaluation of the implementation?
Next to small scale issues like mammoth headers and liberal use of allocation in the implementation of some functions, I think the main culprit is to simplify the implementation by splitting the ABI stable part into another component. Also, the way asynchronous cmponents are used internally hould be aligned with the required fix on the public API.
4. What is your evaluation of the documentation?
The documentation is hard to follow and some 'simple example' are clearly not that simple. This should be the second major point of improvement.
5. What is your evaluation of the potential usefulness of the library?
THis kind of library is very useful and as I said, we need one with proper design and implementation.
6. Did you try to use the library?
Played around with the examle and the doc.
7. How much effort did you put into your evaluation?
Roughly 10-12 hours
8. Are you knowledgeable about the problem domain?
Not with async file I/O but I have a fair amount of experience in asynchronous computation and parallel programming in general. As a concluding note, I *really* want Niall to keep going on making AFIO the reference async file I/O library. Just take the needed time to reflect on the afformentionned PAI and implementation issues and remember that there is no rush to get there.
On 1 Sep 2015 a 11:41, Joel FALCOU wrote:
As a concluding note, I *really* want Niall to keep going on making AFIO the reference async file I/O library. Just take the needed time to reflect on the afformentionned PAI and implementation issues and remember that there is no rush to get there.
Thanks for the review Joel. This final point is an interesting one, and I think it is worth discussing a bit further, though everyone note I start my holidays away from Boost on Thursday and you won't see me again here until GSoC starts up again in January 2016. Race free filesystem is far more useful to the C++ community than async file i/o which really does not deliver what people think it does without completely rearchitecting their application. Race free filesystem would be a big value add for reliability and predictability for any code using the file system because it very substantially reduces the chance for surprise and surface for security attacks. As Beman mentioned a long time ago, for race free filesystem we need to standardise on some method of representing open file handles, and then somehow figure out how that fits nicely with STL iostreams. This is a big mess of difficult and hard, because everybody will have an opinion right from Boost up to WG21, and moreover most people's opinions will have no technical basis for being better than most other people's opinions. There are many, technical merit equivalent, ways to skin that cat, and piloting a compromise suitable to enough people to be standardisation capable would be a big multi-year commitment. Nevertheless, such a library would be of general value to the C++ community, and moreover WG21 is keen on the idea. It would probably be just as slow as AFIO's file path consuming APIs i.e. a lot slower than Boost.Filesystem or STL iostreams path operations, and therefore probably cannot wholly replace the existing solutions (unless POSIX fixes the missing functionality we have to workaround with inode comparing loops - I really wish POSIX would take note of Windows more on file system, Windows has for the most part a far superior implementation). But I think the knock on consequences on modernising STL iostreams to better fit today's C++ world would be enormously valuable, and indeed the source of much disagreement and debate. If anyone wished to volunteer to design and write such a library and take it eventually for standardisation, I would be more than happy to act as an advisor with hard experience from the coal face, but not as an implementor. You may be surprised to hear that I don't much care for conflict, and getting that library past review would involve a whole ton of conflict and I just don't have the energy given my plans for the next few years. Other fish to fry, as I will explain. Regarding AFIO, as you all now realise it is not the library you thought it would be, but my big hope was and is that this perhaps too early review has given you all a chance to ponder what a next generation standardised C++ file system and file i/o library ought to be, and that might bear fruit in any later review of AFIO in months or years to come. Many of the reviews felt something like what ASIO provides for seekable async streams makes a lot more sense through virtue of being much ligher weight - yet, as I countered, if you want that then ASIO already has that ready to go for you. I aim for AFIO to bring something very new to the table: a toolkit of fundamental portable primitives for bare metal programming of file systems where "bare metal" refers to the direct, unmodified exposure of portable filing system semantics to the programmer rather than "bare metal" in terms of least implementation. Where a particular platform differs from the standardised abstract portable model presented by AFIO to the programmer, emulation is performed to make up the shortfall, and hence the apparent complexity and overheads relative to a least implementation solution. That comes, unavoidably, with a very different weighting of what complexities and relative tradeoffs various operations have than you are used to, and it will wreck you head initially. But I am confident that after a few months of it getting into your head space, it'll start to make a lot more sense especially given the relative fixed and variable costs of operations on the file system relative to one another, and the next time you see AFIO it'll hopefully all just make more sense why it does what it does in the way it does it. *Especially* when you see the standardised C++ transactional versioned key-value store based on AFIO I have planned, and the enormous world of new C++ program designs you can do when you can just *assume* you have always reliable transactional persistent versioned storage built into the language runtime and which works perfectly on everything from embedded C++ right up to the largest mainframes, works during process bootstrap before main() is called, copes well with power loss and store damage, and has no issue with concurrent users on heterogeneous platforms making scaling up from tiny SSDs to large distributed SANs trivial. Anyone who has ever programmed on Zope (https://en.wikipedia.org/wiki/Zope) will need no convincing of just what a paradigm shift a standardised scalable transactional persistent store in the runtime affords, and this is why I came out of lurking on boost-dev to develop these libraries after being adjunct to the community here since 2002 or so. I have had enough dealing with inherently unreliable data storage in my applications. I want truly reliable storage supplied as standard in every major programming language runtime available. That's my end goal, and AFIO is but a vital step along that path. Anyway, I'll stop boring you now. I'll be around here until Thursday morning British Standard Time for any questions etc, then I'll be deactivating my boost-dev subscriptions until Jan 2016 to take a long overdue break from Boost library development which has sucked down something close to 700 hours since February. Private email will continue to work of course, and I'll be tipping away at no more than eight hours per week on Boost.Outcome and the AFIO engine rewrite until they're done. I'll monitor the Fiber mini-review discussion starting shortly, but I can already give my vote: it's a YES to acceptance (I have been following Fiber closely to ensure lightweight futures will work with it. It's a definite yes, and my congrats to Oliver for doing such a great job). Thank you *everybody* for the reviews of AFIO. They have been enormously valuable, and I am tremendously grateful to you all for taking the time to review my library. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On Tue, Sep 1, 2015, Niall Douglas wrote:
Regarding AFIO, as you all now realise it is not the library you thought it would be, but my big hope was and is that this perhaps too early review has given you all a chance to ponder what a next generation standardised C++ file system and file i/o library ought to be, and that might bear fruit in any later review of AFIO in months or years to come.
[snip]
Private email will continue to work of course, and I'll be tipping away at no more than eight hours per week on Boost.Outcome and the AFIO engine rewrite until they're done.
Hi Niall, I noticed that you referred to AFIO without the Boost suffix now (unlike "Boost.Outcome"). Does this mean that while you will continue working on AFIO you don't plan to resubmit it for review? (I wasn't sure whether the uncertainty in the phrase "that might bear fruit in any later review of AFIO in months or years to come" was meant to imply that). Glen
On 1 Sep 2015 at 23:56, Glen Fernandes wrote:
Private email will continue to work of course, and I'll be tipping away at no more than eight hours per week on Boost.Outcome and the AFIO engine rewrite until they're done.
I noticed that you referred to AFIO without the Boost suffix now (unlike "Boost.Outcome"). Does this mean that while you will continue working on AFIO you don't plan to resubmit it for review?
That would depend on the review manager's verdict. If Ahmed recommends rejection, the next time you'll see AFIO is as a purely internal implementation library inside another Boost library. It'll be exposed in the reference documentation, but otherwise be unmentioned as it's been made very clear this review the documentation must only refer to the Boost library at hand. From my personal perspective, once the surrounding Boost library is in Boost, that's AFIO into Boost as well and I'm planning to leave it at that. Life is too short to be fighting people over unimportant minutiae when the end goal of a transactional key store is far more important.
(I wasn't sure whether the uncertainty in the phrase "that might bear fruit in any later review of AFIO in months or years to come" was meant to imply that).
If Ahmed recommends acceptance, I would have surely thought an additional review will be needed. I am hoping in that case that AFIO would not look so alien in say six months time once people's minds have had a chance to get used to the idea. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
participants (3)
-
Glen Fernandes -
Joel FALCOU -
Niall Douglas