On Thu, Sep 4, 2008 at 8:24 PM, Hervé Brönnimann <hervebronnimann@mac.com> wrote:
Stjepan: I'm just curious, because my wife uses such a dataflow program for interactive music performance.
The program is called MAX and is distributed by a company named Cyclic74
I really appreciate you bringing up MAX - it was actually my exposure to MAX that eventually lead me to write the Dataflow library. I had previously seen dataflow programming in LabVIEW, but it wasn't until MAX that I realized how easily people (with no formal programming training) did really cool things with such an environment. It lead me to experiment with the dataflow paradigm within C++, and eventually realized that Boost.Signals is a good way of connecting components together. That's how the Dataflow.Signals layer got started.
One of the worse quirks when flowing the data (in a push mode, I guess) is that components are processed in (get this) geometric order, from left to right (breaking ties with top to bottom, I think). I guess it's important to know what the order i, when you allow feedback loops (cycles). But the left-to-right has been a never-ending source of bugs, in my wife's experience :-)
Yep :-)
Which prompts me to ask: - can you create push networks with cycles?
Yes, as long as the components are designed in such a way that doesn't propagate the signal in an infinitely recursive loop. For example, MAX solves some situations with cycles using a convention - signals received on the leftmost inlet are typically processed and propagated, while those received on other inlets are just stored and not propagated. Hence, connecting something to a non-leftmost inlet can break the infinite recursion in cycles. Another way of dealing with cycles is using threading (with some components / sets of components executing in their own threads). The example that was developed at the end of the following thread would allow cycles through the use of threading: http://tinyurl.com/5huap7 [nabble] I hope to expand on that example soon.
- in the presence of cycles, how is the signal flowing to the data sinks?
In the case of the above threading example, at some point of the cycle a component would submit a task to a thread pool corresponding to the signal call (all data communication in the Dataflow.Signals layer happens through Boost.Signals).
- in the absence of cycles, do you process the components in topological sorting order? or is the evaluation order arbitrary?
Currently, arbitrary. In practice, I believe the signals get sent out according to the order in which the consumers were connected to the producer, but I don't think that Boost.Signals guarantees that. Boost.Signals does have a way of ordering signals: http://www.boost.org/doc/libs/1_36_0/doc/html/signals/tutorial.html#id347051... ... Unfortunatelly, Dataflow.Signals doesn't take advantage of it (yet). The library currently offers no way of customizing a connection (e.g., to specify the signaling order), which is a serious limitation that I need to fix.
Surely I could deduce that from your docs if I had read carefully the interface of the components, but why not discuss it up front? I'm lacking a general high-level view of what I can and can't do with this library.
This review is helping me tremendously in understanding which parts of the documentation are more useful than others, and what is missing. I think the docs are definitely going to get restructured :-) To answer your question though, I think the most useful thing about the Dataflow.Signals layer (this layer is the focus of the review) is that it provides tools to implement components that can be used in a dataflow network - both specific ones (e.g., something that generates a particular type of image), and generic ones (something that can be used with any signal signature / any number of arguments, e.g., a component that doubles each of the arguments it receives before passing them on). It also makes connecting components easier, IMO. As far as applications where a library like this might be useful, Giovanni Piero Deretta mentioned a few examples earlier in this thread. The specific domains would be determined by what component one has available / is willing to develop.
A nice feature of Max/MSP though, is that you can create a "patch" which recursively acts as an atomic component, and can be compiled separately, so you can create libraries of patches some of which are opaque to enforce intellectual property (not that I approve, but...). Is there an abstract base class for component? This would enable one conceivably to distribute one's own components as DLLs (i.e., not source).
Yes - the Dataflow.Blueprint layer offers such functionality (but note that this layer is still very much a prototype). As you're familiar with MAX, you might find interesting a sample visual editor I developed for the Dataflow library: http://dancinghacker.com/code/dataflow/dataflow/blueprint/examples/fltk_gui.... If you haven't seen them, there are some videos of the editor in action: http://dancinghacker.blip.tv/posts?view=archive&nsfw=dc
These are the ones I'm more familiar with. I just found the WikiPedia page on my own (http://en.wikipedia.org/wiki/Dataflow_programming) and *then* (only then) noticed the link in your description (second section of Introduction page documentation). It's way too discrete imho. Bring it out, write a short paragraph about some known examples, fire up the imagination, something. As it is, it's a really steep curve in the introduction (which is an introduction to your library, but not to the topic).
OK, I can try to improve that part of the documentation.
Beyond that, I've just started browsing, but I was a bit put off by the steepness of the documentation and the use of operator overloading. It would really benefit from a gentler introduction that is not based on the existing library but on the general idea of dataflow programming. It gets more concrete in the examples, but that's too late, I've already been hopelessly confused by that point :)
Oh oh :-( I was hoping that the "Dataflow programming in C++" section would serve as a gentler introduction, but perhaps it does jump into the library rather quickly and steeply. I am curious, what did you find off-putting about the operator overloading? I am finding this to be a rather contentious issue. Thank you for your feedback, Stjepan