Hi Roland, On Nov 12, 2007 4:00 AM, Roland Schwarz <roland.schwarz@chello.at> wrote:
I just skimmed over the docs, and I have some questions:
Great - thanks for taking the time!
1) How would I specify the control flow? I.e. How would I choose between "source driven" and "sink driven"?
Right now, the generic layer doesn't deal with this at all - it is completely up to the underlying mechanism (just to clarify, by "mechanism" I mean a particular dataflow framework / data transport mechanism, such as Boost.Signals or VTK). The layer based on Boost.Signals is source driven - it sort of assumes that the dominant dataflow direction is from the caller to the callee. Although, if in actuality the data flowed the other way (via the return value), then it would be sink driven. The Dataflow.Signals Example "pull-based networks" illustrates this alternative. On the other hand, I believe the VTK mechanism is sink driven (I am complete newb with VTK - I just learned enough to provide a small example support layer).
2) Is there a mechanism that will give support for block oriented processing, or will a function be called on every data item? To clarify: I do not mean a couple of primitive data items blocked together into a larger container which is itself a data item on its own, but something like the fread() function is able to do: *) request a number of items *) return as much as currently available (may be less than wanted) *) block process what has been received *) wait for more data
You are using VTK as an example and so I suspect you are using the demand driven processing in your library. Are you also providing a caching and timestamping mechanism to hold intermediary results that need no updating?
Which special concepts for memory management are you suggesting (if at all)? Memory management is one of the crucial parts of data flow programming. Will you provide a single copy mechanism?
About all these - the generic dataflow layer is currently really small. It basically deals with connecting the dataflow network together, and that's it. Everything else is up to the underlying mechanism. I'm not sure whether the issues you mention should be dealt with at the generic level, but I guess we'll see as things get built on top of it. As far as the Dataflow.Signals layer is concerned, it currently addresses little of the above features/issues. There is a "storage" component which can serve as cache (but offers no timestamping). It would be useful to add components which dealt with these issues though (e.g., a timestamped storage, and a component (or boost::signals Combiner) which only forwards a signal when it has changed). There is no memory management support here either - because of the nature of dataflow networks built on top of boost::signals (there is no global knowledge of the network - each component only holds connections to its sinks), it's basically all up to the components. There could be some memory management components added if necessary (like now there are some threading related components). And I'll think about fread-like processing components in Dataflow.Signals, seems like it would be a good idea as well. I am not understanding the term "single copy mechanism" - could you help me out with a reference?
I always found that in data flow paradigm these are the hard parts. It is not the hard part to come up with a "nice syntax".
A while back, Tobias Schwinger suggested a dataflow framework which does deal with a lot of the issues you bring up here. It's discussed briefly in the "future directions" of the documentation. I think that such a framework could be supported by the generic layer of this library, after it grows somewhat perhaps. My personal goal in providing a generic layer is to provide a visual programming environment on top of it. That way, anything that has a Dataflow support layer can have it's networks be manipulated visually, serialized, instantiated, etc. In the meantime, while there isn't much built on top of the generic layer, the "nice syntax" is unfortunately the biggest practical benefit. Again, thanks for taking the time! Best regards, Stjepan