Hi Paul, On Sat, Mar 15, 2008 at 6:02 PM, Paul Baxter <pauljbaxter@hotmail.com> wrote:
I'm really excited about this library but is it ready for review yet?
Thanks for your comments - I understand your concerns, please see my thoughts below.
Stjepan has kindly kept us up to date with a v0.80/81 and later 0.90 with a fledgling blueprint layer but, playing devil's advocate, perhaps more discussion of what it's trying to solve and how it might make use of other libraries might be useful prior to review. I'm concerned that its fate might be similar to the first revision of the logging library otherwise. I suspect there are many differing/conflicting requirements for a dataflow library.
I would definitely agree that there are many differing requirements for a dataflow library. That is why I am only proposing the Signals layer of the review - it was the original focus of the library, and even though the code under and over the Signals layer has been in turmoil for a long time, the interface of the Signals layer itself has been mostly stable for months now. Realizing that made me think I should focus on getting it out for review first. It is possible that there would also be different requirements even for a Boost.Signals-based library - but I'm not as concerned with the fate of the library in terms of acceptance as I am with the feedback I get.
When I've done similar things in the past the critical decision has been whether dataflows are defined at compile time or run-time and what the strategies are for information, buffer and connection management.
One thing that I tried to do is build the Signals layer on top of something generic enough to (eventually) accommodate all of the above. For now, the generic layer is still biased towards the characteristics of Boost.Signals - connectability is determined at compile-time but dataflows are constructed at runtime; connection management and the the operation of the network is left completely to the components and "the outside world". I hope that by review time I can expand the generic layer so it can at least accommodate some different scenarios. That way anyone can adapt or implement their ideal dataflow framework to work with the library.
Even in the last couple of days several areas of functionality and interface have been discussed. Compiler support seems weak too on very popular compilers.
These discussions have been great - and the features discussed would be great additions, but I don't think that their current lack makes the library inadequate. Also, I can make a reasonable effort to expand the compiler support, but I don't see this as a critical issue for review either. I don't want to put too much effort into doing detail work on something that might be shown to need a complete redesign.
Perhaps Stjepan could comment further on his plans between now and review. Its a worthwhile library and something I'm personally very interested in but I'd love to see it fleshed out a bit more with a reasonable application example. This is a library that is not really core to many tasks and probably needs to show its mettle through use.
My plans between now and review mostly have to do with expanding the documentation, with a few implementation pushes as well. On the documentation side it's: * document more of what is implemented * provide better descriptions of examples, and add more examples * reorganize the docs for review so that it is clear what is finished and ready for review, and what is just to be considered an example/proof of concept) On the implementation side it's: * tweak the Dataflow.Signals layer so that components can be composed at compile-time. * address recently discussed ideas / requests. Most of the things above are already in the works, and it might take a long time to get a review manager + time slot anyway. Now, the ideal situation would be that I also manage to do the following: * expand the generic layer to better support non-Boost.Signals-like dataflow frameworks (e.g., determining connectability and other properties at run-time) * add basic support layers for existing dataflow frameworks out there to show that the generic layer can support them. If that is also done by review time, then the review can cover both the Dataflow.Signals layer and the generic layer - otherwise the generic layer can be considered at some other time.
Is it intended to tackle building processing sequences on the fly. Is it lightweight enough to support fine-grained building blocks where connection setup could end up dominating run time if not done well. Is it simply about ease of use in putting together blocks? Is it's uniqueness about supporting dynamic run-time sequence configurability rather than compile-time?
The Dataflow.Signals layer does what boost::signal does - it is not fine grained, it's compile-time connectability-checking with run-time connecting, it's producers with imperfect tracking of consumers and consumers with no tracking of producers, and it couples data-transfer with component invocation. The generic layer, on the other hand, should eventually support all of the scenarios you mention (and I really hope that it too will be ready by review time). Then anyone can take an existing dataflow framework that has the desired characteristics (or develop a new one), make a Dataflow support layer for it, and take advantage of anything that is built on top of Dataflow (like the blueprint layer or the GUI, which are still just in a proof-of-concept stage, but for review I am more interested in showing that they /can/ be built rather than providing full-featured versions).
I guess I'd like to see the rationale for what its good at and then see compelling evidence (in the form of an example?)
I hope this e-mail made things a bit clearer. If you still have questions please let me know - but also I'd love to hear about what your personal dataflow wishlist is, and what sorts of dataflow frameworks/scenarios you've worked with in the past (offline or here). Kind regards, Stjepan