On Fri, Sep 5, 2008 at 4:30 AM, Phil Endecott <spam_from_boost_dev@chezphil.org> wrote:
Stjepan Rajko wrote:
I really appreciate you bringing up MAX - it was actually my exposure to MAX that eventually lead me to write the Dataflow library. I had previously seen dataflow programming in LabVIEW, but it wasn't until MAX that I realized how easily people (with no formal programming training) did really cool things with such an environment.
This is some very interesting rationale that you hadn't previously shared with us ;-)
A question that had been forming in my mind was, "Who is this library for?". Do you see this library useful primarily for people with no (or little) formal programming training?
Thanks for bringing this up - it is a very important piece of information that I completely neglected to address in the documentation. At it's current state, I think the library is more geared towards people that are designing plugin-based Dataflow systems (like MAX or LabVIEW), or developing an application / library for which the dataflow paradigm is particularly suitable. In the context of Dataflow.Signals, that means providing facilities for programmers to create new components, and providing different ways in which the components can be connected (e.g., through GUI, using operators, using the connect function, using the bind_mem_fn helper). Once someone develops a reasonably complete set of components for a particular domain, *then* a person with less programming experience could use those components to create their own applications, assuming they are also comfortable with whatever facility for connecting is given to them (GUI vs. connecting in C++).
If so, do you really think that Boost, or for that matter C++, is the right starting point? Or perhaps you believe that the benefits that those people see would also apply to mainstream software developers? Are you hoping that "regular C++ programmers like us" will start to use the dataflow style with the help of the library, or is there an existing body of dataflow programmers who currently use some other language who can be "converted"?
I have a few points related to these questions. I can tell you from my personal experience as a "regular C++ programmer" that, after I started playing with MAX, I found that way of programming (meaning dataflow, not necessarily needing a GUI editor) very well suited for certain kinds of applications. I worked with signal processing quite a bit (in C++), and after seeing MAX found myself trying to implement them in a dataflow sort of ways - developing components, and then connecting them. I came up with a precursor to Dataflow.Signals, and found it very useful for signal processing (I used the operator syntax with it). It eased development and maintainability, simply because it allowed me to program in a paradigm that matched the application very well. To address its applicability to boost, I have two angles on that. For one, I would assume that a good number of programmers that use boost have clients with limited programming expertise. They could use Dataflow.Signals like I mentioned above - by developing a relevant set of components, and training the client on how to connect them and use them. Again, from my experience, people with absolutely no programming experience can understand the dataflow paradigm very quickly. I've seen it happen. I think it is because it uses metaphors that are readily accessible to many people, as we tend to connect things in our daily lives (e.g., connecting headphones to an iPod, a DVD player to a TV, a guitar to a pedal to an amp), and we are also used to tweaking the behavior of the components to get the desired result (e.g., turning the volume knob). As far as the other angle... Bjarne Stroustrup gave a very interesting talk at the last BoostCon, and it had to do with providing different levels of accessibility / configurability trade-offs to C++ programmers. E.g., an entry level version of a library (or set of libraries) that is super easy to use (and debug with) but maybe not very flexible (although providing enough features to make it useful & fun), vs. a fully templated super powerful version of the same library that can assume you really know what you are doing. I actually do see the Dataflow library as something that can address the high accessibility / low configurability (of individual components) end of the spectrum, especially when combined with a GUI. Furthermore, I think it can be conducive to leading people to learn more about C++ (because the system itself could be used without any C++ knowledge whatsoever). Sooner or later, many users of a GUI/dataflow system will find the need to extend a component in some new way or develop a new component. It wouldn't be difficult to show someone how to take an existing component that adds two numbers, and change it to something that multiplies the two numbers (granted, they would have to learn how to compile C++ / boost which can be non-trivial). If that sparks their interest and they learn more C++, perhaps they will advance to wanting to develop a component that works with files, and learn Boost.Filesystem :-)
I also wonder how much of the benefit of a graphical environment like LabVIEW carry over into your textual dataflow description (even with 2D operator overloading). It seems to me that one of the main benefits of a GUI is that the user is somewhat guided towards a "syntactically correct" program by the help of, for example, labelled parameter fields to fill in on the components. That, and other aspects, are lost.
Yes, the operator syntax is a gross simplification / approximation and probably only suitable for simple networks. As far as labeling fields, the library offers some help. It is possible to provide appropriately named member functions that return a port - for example, the storage component has a member function send_slot(), which returns the port corresponding to the send() member function. So, if I do, storage<void()> initiate_send_of_zero; storage<void(int)> zero(0); storage<void(int)> receiver; connect (initiate_send_of_zero, zero.send_slot()); connect (zero, receiver); initiate_send_of_zero.send(); what will happen is: initiate_send_of_zero will send a void() signal to the zero.send() function (i.e., the zero.send() function will be called); when zero.send() is called, it will send it's stored value (0) to receiver. receiver now holds a 0. In addition, the library offers a way to enumerate all of the ports - you can do things like connect(get_port_c<0>(a), get_port_c<1>(b)); there are also mpl versions: connect(get_port<mpl::int_<0> >(a), get_port_c<mpl::int_<1> >(b)); So, you can have member typedefs for the ports that carry the name of the port (e.g., typedef mpl::int<0> output;), in which case you could do: connect(get_port<A::output>(a), get_port<B::input>(b)); Unfortunately, registering all the ports so they can be enumerated is an advanced topic and not discussed in the documentation yet :-(
It may be significant that systems like LabView and MAX have not escaped from their niche application areas. "Real programmers prefer text" perhaps. Note that over the last couple of decades, chip design has almost entirely moved from GUI input (schematic capture) to textual input (hardware description languages).
The Dataflow library as a whole is an attempt to provide dataflow functionality regardless of the programming medium (text or visual).
And while thinking about hardware description languages, note also that they don't expose any sort of dataflow model even though the underlying circuit often has that sort of structure. Similarly, when people were building "dataflow computers" back in the '80s they wrote compilers that hid the dataflow nature of the hardware behind a more conventional programming language (e.g. SISAL).
I have found dataflow programming useful precisely when it matches the underlying problem structure.
The closest to what you're proposing that I have seen is the stuff that the GNU Radio project is doing; I mentioned them on this list once before. (They have been in the news recently after operating a GSM base station using their software radio at Burning Man.) It would be really helpful if you could perhaps try to re-implement some of their stuff using your library, and see how it compares in terms of ease of coding, performance etc. You could ask them to compare it themselves and submit reviews here.
I tried suggesting a Google Summer of Code project along these lines: http://lists.gnu.org/archive/html/discuss-gnuradio/2008-02/msg00247.html ... but got very little response. In terms of reimplementing - when it comes to an existing dataflow system, the purpose of the dataflow library wouldn't be so much in reimplementing it, but helping build on the core of the dataflow framework. I.e., we can take the base GNU Radio framework which provides the component / connection code, and develop a Dataflow support layer for it. Then, we get things like the GUI for free. Unfortunately, it seems like GNU radio doesn't support pure C++ programs fully yet (I will give it a try when it does). I have done a similar experiment with VTK though - in the dataflow library, there is an example in libs/dataflow/example/glv_gui/glvgui_vtk which brings up the GUI editor with the 5 components from the following example: http://www.dancinghacker.com/code/dataflow/dataflow/support/examples/new_lay... With the GUI editor, you can instantiate the 5 components, connect them in the correct chain, and click on the final component to invoke the rendered scene.
I think you really need to justify to us why this library is useful, and to whom.
This has definitely been lacking, but I hope I did a little bit of a better job in my response here.
- can you create push networks with cycles?
Yes, as long as the components are designed in such a way that doesn't propagate the signal in an infinitely recursive loop.
I am reminded here of the design of asynchronous circuits using handshake signalling. You might like to have a look at Kees van Berkel's thesis, which seems to be visible at books.google.com.
Will do - thanks for the reference. And thank you for the continuing discussion about the library, it is very helpful. Best, Stjepan