Review Request: Dataflow.Signals
Hello, I would like to request a formal review of the Dataflow.Signals layer of the Dataflow library. The Dataflow.Signals layer provides dataflow functionality based on function calls (using Boost.Signals), and is a result of the Signal Networks GSoC project from 2007. (The other layers of the Dataflow library are not quite ready, so for now I'd like to keep them as implementation details and/or examples of the Dataflow.Signals layer, and submit them for review at a later point.) The documentation for the Dataflow library is located at: http://dancinghacker.com/code/dataflow/ Version 0.9.0 has been uploaded to the vault, in the Dataflow directory. There are a few more things I plan to add before the library comes up for review, especially regarding the documentation and motivating examples. Description: The Dataflow library is a generic library for dataflow programming. Dataflow programs can typically be expressed as a graph in which vertices represent components that process data, and edges represent the flow of data between the components. As such, dataflow programs can be easily reconfigured by changing the components and/or the connections. The Dataflow.Signals layer provides dataflow functionality based on function calls - which is one of the fundamental ways of dataflow in C++. For its data transport mechanism, Dataflow.Signals uses Boost.Signals which can be used to make lasting dataflow connections based on function calls. Dataflow.Signals provides the following to facilitate signals-based dataflow networks: * A number of useful general-purpose components, and building blocks for implementing new components. * Various free functions and operators for connecting and using components. Kind regards, Stjepan
Stjepan, very good news! I'm glad to see that an important milestone has been reached. This library is of great interest to me. Though, I wasted no time to and started to play around with the library. As a start I'm trying to get the simple example compiled which can be found in your documentation: http://dancinghacker.com/code/dataflow/dataflow/introduction/dataflow.html But unfortunately including #include <boost/dataflow/signals/component/filter.hpp> brings my compiler to explode. Here are a list of the first few errors produced: c:\boost_trunk\boost\fusion\mpl\clear.hpp(26) : error C2039: 'tag_of' : is not a member of 'boost::fusion::detail' c:\boost_trunk\boost\fusion\mpl\clear.hpp(28) : see reference to class template instantiation 'boost::mpl::clear_impl<boost::fusion::fusion_sequence_tag>::apply<Sequence>' being compiled c:\boost_trunk\boost\fusion\mpl\clear.hpp(26) : error C2146: syntax error : missing ',' before identifier 'tag_of' c:\boost_trunk\boost\fusion\mpl\clear.hpp(26) : error C2065: 'tag_of' : undeclared identifier c:\boost_trunk\boost\fusion\mpl\clear.hpp(26) : error C2976: 'boost::fusion::detail::clear' : too few template arguments c:\boost_trunk\boost\fusion\mpl\detail\clear.hpp(27) : see declaration of 'boost::fusion::detail::clear' [...] I'm working with the boost trunk and use the MSVC compiler 7.1. I think I remember those errors from an earlier attempt to compile the dataflow lib. Is it my compiler or is my fusion version outdated. Although, as I mentioned I'm working with the trunk. Regards, Christian On Mon, Mar 10, 2008 at 9:25 PM, Stjepan Rajko <stipe@asu.edu> wrote:
Hello,
I would like to request a formal review of the Dataflow.Signals layer of the Dataflow library. The Dataflow.Signals layer provides dataflow functionality based on function calls (using Boost.Signals), and is a result of the Signal Networks GSoC project from 2007. (The other layers of the Dataflow library are not quite ready, so for now I'd like to keep them as implementation details and/or examples of the Dataflow.Signals layer, and submit them for review at a later point.)
The documentation for the Dataflow library is located at: http://dancinghacker.com/code/dataflow/
Version 0.9.0 has been uploaded to the vault, in the Dataflow directory.
There are a few more things I plan to add before the library comes up for review, especially regarding the documentation and motivating examples.
Description:
The Dataflow library is a generic library for dataflow programming. Dataflow programs can typically be expressed as a graph in which vertices represent components that process data, and edges represent the flow of data between the components. As such, dataflow programs can be easily reconfigured by changing the components and/or the connections.
The Dataflow.Signals layer provides dataflow functionality based on function calls - which is one of the fundamental ways of dataflow in C++. For its data transport mechanism, Dataflow.Signals uses Boost.Signals which can be used to make lasting dataflow connections based on function calls.
Dataflow.Signals provides the following to facilitate signals-based dataflow networks:
* A number of useful general-purpose components, and building blocks for implementing new components. * Various free functions and operators for connecting and using components.
Kind regards,
Stjepan _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Mon, Mar 10, 2008 at 8:02 PM, Christian Henning <chhenning@gmail.com> wrote:
Stjepan,
very good news! I'm glad to see that an important milestone has been reached. This library is of great interest to me. Though, I wasted no time to and started to play around with the library.
Great - thanks for taking the time!
As a start I'm trying to get the simple example compiled which can be found in your documentation:
http://dancinghacker.com/code/dataflow/dataflow/introduction/dataflow.html
But unfortunately including #include <boost/dataflow/signals/component/filter.hpp>
brings my compiler to explode. Here are a list of the first few errors produced:
[snip errors]
I'm working with the boost trunk and use the MSVC compiler 7.1. I think I remember those errors from an earlier attempt to compile the dataflow lib. Is it my compiler or is my fusion version outdated. Although, as I mentioned I'm working with the trunk.
Sorry this didn't take you very far. I should have mentioned that I only tested with GCC 4.0.1 (OS X) and MSVC8, and I'm not sure how the lib behaves on other platforms. I might find a machine that has 7.1 on it - if I do I'll see if I can get the lib to run on it. I am testing against boost trunk, so you're good on that at least. There's a decent chance I might have more info tomorrow - I'll keep you posted. Kind regards, Stjepan
Stjepan, I've MSVC 8 at work and will do the same. I'll keep you posted. Thanks, Christian On Mon, Mar 10, 2008 at 11:30 PM, Stjepan Rajko <stipe@asu.edu> wrote:
On Mon, Mar 10, 2008 at 8:02 PM, Christian Henning <chhenning@gmail.com> wrote:
Stjepan,
very good news! I'm glad to see that an important milestone has been reached. This library is of great interest to me. Though, I wasted no time to and started to play around with the library.
Great - thanks for taking the time!
As a start I'm trying to get the simple example compiled which can be found in your documentation:
http://dancinghacker.com/code/dataflow/dataflow/introduction/dataflow.html
But unfortunately including #include <boost/dataflow/signals/component/filter.hpp>
brings my compiler to explode. Here are a list of the first few errors produced:
[snip errors]
I'm working with the boost trunk and use the MSVC compiler 7.1. I think I remember those errors from an earlier attempt to compile the dataflow lib. Is it my compiler or is my fusion version outdated. Although, as I mentioned I'm working with the trunk.
Sorry this didn't take you very far. I should have mentioned that I only tested with GCC 4.0.1 (OS X) and MSVC8, and I'm not sure how the lib behaves on other platforms. I might find a machine that has 7.1 on it - if I do I'll see if I can get the lib to run on it. I am testing against boost trunk, so you're good on that at least.
There's a decent chance I might have more info tomorrow - I'll keep you posted.
Kind regards,
Stjepan _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
I tried MSVC8 and was able to go small steps forward. Basically, I was able to compile all header files, albeit explicitely setting /Zm1000. Which is fine with me but might be helpful to know for other people. Where I get into troubles is when defining the data processor class. For demonstration purposes I just create an empty class for now. But I have the same errors when I copy and paste your processor from the documentation. class processor : public signals::filter<processor, void (double)> {}; Here is what my compiles outputs: 1>c:\boost\boost\type_traits\function_traits.hpp(170) : error C2504: 'boost::detail::function_traits_helper<Function>' : base class undefined 1> with 1> [ 1> Function=processor * 1> ] 1> c:\boost\boost\dataflow\signals\component\filter.hpp(55) : see reference to class template instantiation 'boost::function_traits<Function>' being compiled 1> with 1> [ 1> Function=processor 1> ] 1>c:\boost\boost\dataflow\signals\component\filter.hpp(55) : error C2039: 'result_type' : is not a member of 'boost::function_traits<Function>' 1> with 1> [ 1> Function=processor 1> ] 1>c:\boost\boost\dataflow\signals\component\filter.hpp(55) : error C2146: syntax error : missing ',' before identifier 'result_type' 1>c:\boost\boost\dataflow\signals\component\filter.hpp(55) : error C2065: 'result_type' : undeclared identifier 1>c:\code samples\boost\dataflow\first\first.cpp(11) : error C3203: 'last_value' : unspecialized class template can't be used as a template argument for template parameter 'Combiner', expected a real type 1>c:\code samples\boost\dataflow\first\first.cpp(12) : error C2504: 'boost::signals::filter<Signature,OutSignal>' : base class undefined 1> with 1> [ 1> Signature=processor, 1> OutSignal=void (double) 1> ] This time I'm using MSVC 8 with the updated boost trunk. BTW, why to have your own boost/functional lib? Are you overriding or extending it? Regards, Christian On Tue, Mar 11, 2008 at 10:05 AM, Christian Henning <chhenning@gmail.com> wrote:
Stjepan,
I've MSVC 8 at work and will do the same. I'll keep you posted.
Thanks, Christian
On Mon, Mar 10, 2008 at 11:30 PM, Stjepan Rajko <stipe@asu.edu> wrote:
On Mon, Mar 10, 2008 at 8:02 PM, Christian Henning <chhenning@gmail.com> wrote:
Stjepan,
very good news! I'm glad to see that an important milestone has been reached. This library is of great interest to me. Though, I wasted no time to and started to play around with the library.
Great - thanks for taking the time!
As a start I'm trying to get the simple example compiled which can be found in your documentation:
http://dancinghacker.com/code/dataflow/dataflow/introduction/dataflow.html
But unfortunately including #include <boost/dataflow/signals/component/filter.hpp>
brings my compiler to explode. Here are a list of the first few errors produced:
[snip errors]
I'm working with the boost trunk and use the MSVC compiler 7.1. I think I remember those errors from an earlier attempt to compile the dataflow lib. Is it my compiler or is my fusion version outdated. Although, as I mentioned I'm working with the trunk.
Sorry this didn't take you very far. I should have mentioned that I only tested with GCC 4.0.1 (OS X) and MSVC8, and I'm not sure how the lib behaves on other platforms. I might find a machine that has 7.1 on it - if I do I'll see if I can get the lib to run on it. I am testing against boost trunk, so you're good on that at least.
There's a decent chance I might have more info tomorrow - I'll keep you posted.
Kind regards,
Stjepan _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Christian, Thank you so much for helping get this figured out! On Tue, Mar 11, 2008 at 8:40 AM, Christian Henning <chhenning@gmail.com> wrote:
I tried MSVC8 and was able to go small steps forward. Basically, I was able to compile all header files, albeit explicitely setting /Zm1000. Which is fine with me but might be helpful to know for other people.
Thanks - I'll mention that. I've been testing with VC Express 8.0.50727 but didn't need that switch. Or at least I didn't need to set it explicitly - I did test builds with bjam (the build/vc8ide folder has a vc8 project that runs bjam for tests and examples), and I also tried the intro_example in a typical VC8 console project.
Where I get into troubles is when defining the data processor class. For demonstration purposes I just create an empty class for now. But I have the same errors when I copy and paste your processor from the documentation.
class processor : public signals::filter<processor, void (double)> {};
Here is what my compiles outputs:
[snip errors]
Hmm. These errors are very suspicious. When you downloaded from the vault, did you download the latest (0_9_0) version? If so, something may have gone wrong when I archived it - I thought I checked but maybe not enough. I'll do a clean download of boost and a clean download of the lib and see what's happening. If it's the archiving that I messed up, you can also try the version from the SVN (if you don't feel like you've already wasted enough time on this :-)): http://svn.boost.org/trac/boost/browser/sandbox/SOC/2007/signals
BTW, why to have your own boost/functional lib? Are you overriding or extending it?
One of the examples uses the factory class that was submitted for review recently. I never got around to checking whether that made it to the trunk, so I just kept a local copy. Thanks again for all your help with this! Stjepan
On Tue, Mar 11, 2008 at 2:42 PM, Stjepan Rajko <stipe@asu.edu> wrote:
Christian,
Thank you so much for helping get this figured out!
As, I said I'm very interested in this. ;-)
[snip errors]
Hmm. These errors are very suspicious. When you downloaded from the vault, did you download the latest (0_9_0) version? If so, something may have gone wrong when I archived it - I thought I checked but maybe not enough. I'll do a clean download of boost and a clean download of the lib and see what's happening.
If it's the archiving that I messed up, you can also try the version from the SVN (if you don't feel like you've already wasted enough time on this :-)): http://svn.boost.org/trac/boost/browser/sandbox/SOC/2007/signals
I have removed the dataflow folder from my boost directory and use the sandbox version now. But still I get the same errors. After work today I might be able to look under the hood.
BTW, why to have your own boost/functional lib? Are you overriding or extending it?
One of the examples uses the factory class that was submitted for review recently. I never got around to checking whether that made it to the trunk, so I just kept a local copy.
I take that for the review these file aren't needed. Christian
Stjepan, quick update. I have fixed an error on my side using MSVC 7.1. I copied now all code from the first example and get the following compiler errors: c:\Code Samples\boost\dataflow\first\first.cpp(71) : error C2893: Failed to specialize function template 'boost::enable_if<boost::mpl::and_<boost::dataflow::has_default_port<OutgoingPort,boost::dataflow::args::left,boost::dataflow::signals::connect_mechanism,boost::dataflow::signals::tag>,boost::dataflow::has_default_port<IncomingPort,boost::dataflow::args::right,boost::dataflow::signals::connect_mechanism,boost::dataflow::signals::tag>>,OutgoingPort&>::type boost::signals::operator >>=(OutgoingPort &,const IncomingPort &)' With the following template arguments: 'processor' 'output' c:\Code Samples\boost\dataflow\first\first.cpp(71) : error C2893: Failed to specialize function template 'boost::enable_if<boost::mpl::and_<boost::dataflow::has_default_port<OutgoingPort,boost::dataflow::args::left,boost::dataflow::signals::connect_mechanism,boost::dataflow::signals::tag>,boost::dataflow::has_default_port<IncomingPort,boost::dataflow::args::right,boost::dataflow::signals::connect_mechanism,boost::dataflow::signals::tag>>,OutgoingPort&>::type boost::signals::operator >>=(OutgoingPort &,IncomingPort &)' With the following template arguments: 'processor' 'output' c:\Code Samples\boost\dataflow\first\first.cpp(71) : error C2676: binary '>>=' : 'processor' does not define this operator or a conversion to a type acceptable to the predefined operator Weird, I can compile more with 7.1 than with 8. Below is my code. Christian #include <iostream> #include <tchar.h> #include <boost/dataflow/signals/component/filter.hpp> #include <boost/dataflow/signals/component/storage.hpp> #include <boost/dataflow/signals/component/timed_generator.hpp> #include <boost/dataflow/signals/connection.hpp> #include <boost/random/mersenne_twister.hpp> #include <boost/random/normal_distribution.hpp> #include <boost/random/variate_generator.hpp> using namespace boost; class processor : public signals::filter<processor, void (double)> { public: // Initialize the Gaussian noise generator. processor() : generator(mt, dist) {} // Receive void(double) signals, add some Gaussian noise, and send // out the modified value. void operator()(double x) { out(x + generator()); } private: mt19937 mt; normal_distribution<> dist; boost::variate_generator<mt19937&, boost::normal_distribution<> > generator; }; // This will be our data output. We just need to make a function object, // and specify that it is a signals::call_consumer. class output { public: typedef dataflow::signals::call_consumer<> dataflow_traits; void operator()(double x) { std::cout << x << std::endl; } }; int _tmain(int argc, _TCHAR* argv[]) { // For our data source, we will use timed_generator, // which creates its own thread and outputs it's stored value // at a specified time interval. We'll store a value of 0 to be sent out. // The signature void(double) specifies that the signal carries a double, // and that there is no return value. signals::timed_generator<void (double)> input(0); // Data processor and output: processor proc; output out; // ---Connect the dataflow network --------------------- // // ,---------. ,---------. ,---------. // | input | --> | proc | --> | out | // `---------' `---------' `---------' // // ----------------------------------------------------- input >>= proc >>= out; // If you prefer, you can also do: // connect(input, proc); // connect(proc, out); // Tell the source to start producing data, every 0.5s: input.enable(0.5); // take a little nap :-) boost::xtime xt; boost::xtime_get(&xt, boost::TIME_UTC); xt.sec += 10; boost::thread::sleep(xt); input.join(); return 0; }
On Tue, Mar 11, 2008 at 7:45 PM, Christian Henning <chhenning@gmail.com> wrote:
Stjepan, quick update. I have fixed an error on my side using MSVC 7.1.
I copied now all code from the first example and get the following compiler errors:
[snip errors]
Good news - I was able to get a copy of MSVC 7.1 and at least replicate these errors :-) When I replace input >>= proc >>= out; with connect(input, proc); connect(proc, out); I seem to avoid the error but run into an ICE in the storage component (timed_generator inherits storage). If I throw out timed_storage and just feed proc with values manually, I get errors regarding traits_of. That I might be able to deal with in time - I would conjecture that for 7.1, the basic Dataflow.Signals layer might be doable but the generic components might be really tricky to get to work.
Weird, I can compile more with 7.1 than with 8. Below is my code.
[snip code]
While I can replicate the errors with 7.1, I am still getting no errors with 8.0. I will try a fresh download on a fresh comp tomorrow (gotta put a laptop back together for it :-)). Which revision of the boost trunk are you working with? Thanks again, Stjepan
Alright, I can compile the code using MSVC 8. Cool now I can start toying with it!
I seem to avoid the error but run into an ICE in the storage component (timed_generator inherits storage). If I throw out timed_storage and just feed proc with values manually, I get errors regarding traits_of. That I might be able to deal with in time - I would conjecture that for 7.1, the basic Dataflow.Signals layer might be doable but the generic components might be really tricky to get to work.
At the risk of talking out of my ... . A library to review would at the very least need to compile with MSVC 7.1. I deal with the same problem right now. Code works for MSVC 8 but trying to make it compile with MSVC 7.1 is a pain. Also, my code looks good in a sense that I wouldn't like it to mess it up with preprocessor stuff or boost/config. Do you think you can rewrite the layer again to have a work with 7.1? Christian
On Wed, Mar 12, 2008 at 6:58 AM, Christian Henning <chhenning@gmail.com> wrote:
Alright, I can compile the code using MSVC 8. Cool now I can start toying with it!
Great! What made it change its mind?
I seem to avoid the error but run into an ICE in the storage component (timed_generator inherits storage). If I throw out timed_storage and just feed proc with values manually, I get errors regarding traits_of. That I might be able to deal with in time - I would conjecture that for 7.1, the basic Dataflow.Signals layer might be doable but the generic components might be really tricky to get to work.
At the risk of talking out of my ... . A library to review would at the very least need to compile with MSVC 7.1. I deal with the same problem right now. Code works for MSVC 8 but trying to make it compile with MSVC 7.1 is a pain. Also, my code looks good in a sense that I wouldn't like it to mess it up with preprocessor stuff or boost/config. Do you think you can rewrite the layer again to have a work with 7.1?
I'm certainly willing to add as much 7.1 support as I can if that makes it easier for you to use the library where you need it. I don't think 7.1 support would be a requirement for review though - the website states a submitted library should compile on at least two compilers, without specific compilers listed. Of course, the more supported compilers the better :-) Now that I have a copy of 7.1 installed, I can start chipping away at it, but it might take a couple of weeks before I have some serious time to devote to it. I need to tighten up the tests anyway, and doing that hand in hand with adding support for a new compiler will make writing good tests easier. Anyway, I'm glad you can at least play with the lib on 8.0 for now - if you have any feedback or requests please let me know (here or off-list), I'm most interested to hear how the library suits (or doesn't suit) your needs! Cheers, Stjepan
Hi, I've some questions related to Dataflow.Signals library: - As mentioned in the docs the library models the pipe-filter pattern. What about event handling? The arguments of the function calls could be seen as the payload of the event?! - Is it possible that one consumer instance is connected to multiple producers? - Can a consumer decide if it will disconnect form its producer inside the invoked operator()-function? best regards, Oliver
Hi Oliver, On Thu, Mar 13, 2008 at 12:31 AM, Kowalke Oliver (QD IT PA AS) <Oliver.Kowalke@qimonda.com> wrote:
Hi, I've some questions related to Dataflow.Signals library:
- As mentioned in the docs the library models the pipe-filter pattern. What about event handling? The arguments of the function calls could be seen as the payload of the event?!
Right - if the signal is used to model an event, any information associated with the event can be passed as an argument. Alternatively, if there is no information except for the event itself, you can use a signal of signature `void()`. E.g., in the GIL example (http://dancinghacker.com/code/dataflow/dataflow/signals/introduction/example...), the output of the `timer` component is analogous to an event (timer cycle) with no extra information (hence uses the `void()` signature). When `generator` receives the event, it sends out an image, etc.. There is nothing in the library that distinguishes between signals used for events and signals used for data - it's just how it's used. I'm not sure if I answered this question adequately, please let me know if I didn't.
- Is it possible that one consumer instance is connected to multiple producers?
Yes. But, when a consumer receives the data, it has no knowledge of where it came from. If you need that information, you can make it a part of the signal.
- Can a consumer decide if it will disconnect form its producer inside the invoked operator()-function?
You mean, when the consumer receives a signal, you'd like it to be able to disconnect from the producer that sent the signal? There isn't anything in the library that directly supports this, and an additional difficulty is that the consumer has no knowledge of any of the connected producers by default. There would have to some outside mechanism that would allow this - one way would be to return something to the producer that would tell it to disconnect that particular consumer. Another would be to have the consumer keep a copy of the Boost.Signals connection object (for the connection between its producer and itself), and disconnect at will. A third strategy would be to send a signal/function call to some network-managing component that would perform the disconnect. I have some unfinished business with the disconnection code - if you're interested in any of these strategies let me know and I'll implement what is necessary to make it doable and produce an example. Thanks for your interest in the library! Stjepan
Hello Stjepan,
- Can a consumer decide if it will disconnect form its producer inside the invoked operator()-function?
You mean, when the consumer receives a signal, you'd like it to be able to disconnect from the producer that sent the signal?
yes
Another would be to have the consumer keep a copy of the Boost.Signals connection object (for the connection between its producer and itself), and disconnect at will.
I would prefer this solution - maybe signals::filter can provide a bool connected() and void disconnect() function which internaly calls boost::signals::connection::disconnect()? regards, Oliver
On Thu, Mar 13, 2008 at 1:18 AM, Kowalke Oliver (QD IT PA AS) <Oliver.Kowalke@qimonda.com> wrote:
Another would be to have the consumer keep a copy of the Boost.Signals connection object (for the connection between its producer and itself), and disconnect at will.
I would prefer this solution - maybe signals::filter can provide a bool connected() and void disconnect() function which internaly calls boost::signals::connection::disconnect()?
This is a great idea. I made a little proof of concept example here: http://tinyurl.com/2zl9ar There are some minimal comments that explain it. Basically, the example defines a new type of consumer (tracking_call_consumer) for which the connect operation will register the connection object with the consumer. It then implements a simple class (tracking_consumer) which is a tracking_call_consumer and will disconnect (from all registered producers) once it has received a specified number of signals. Something like this will be a good addition to the library - when I get more time I'll try to figure out a good way to integrate it with the existing components, so that any component can be made to track connections to its consumer ports. Regards, Stjepan
Hello Stjepan,
On Thu, Mar 13, 2008 at 1:18 AM, Kowalke Oliver (QD IT PA AS) <Oliver.Kowalke@qimonda.com> wrote:
Another would be to have the consumer keep a copy of the > Boost.Signals connection object (for the connection between > its producer and itself), and disconnect at will.
I would prefer this solution - maybe signals::filter can
provide a bool connected() and void disconnect() function which internaly calls boost::signals::connection::disconnect()?
This is a great idea. I made a little proof of concept example here: http://tinyurl.com/2zl9ar
Something like this will be a good addition to the library - when I get more time I'll try to figure out a good way to integrate it with the existing components, so that any component can be made to track connections to its consumer ports.
Good news! I don't know if your dataflow library can support selectivly disconnects. As you wrote in your previous email - one consumer can be connected to multiple producers. If we can associate the boost::signal::connection with the invokation of consumers operator() we could establish a selectively disconnect. At least I think it could work this way - but I don't know if it could be supported by your implementation. What do you think? best regards, Oliver
On Thu, Mar 13, 2008 at 11:57 PM, Kowalke Oliver (QD IT PA AS) <Oliver.Kowalke@qimonda.com> wrote:
I don't know if your dataflow library can support selectivly disconnects. As you wrote in your previous email - one consumer can be connected to multiple producers. If we can associate the boost::signal::connection with the invokation of consumers operator() we could establish a selectively disconnect. At least I think it could work this way - but I don't know if it could be supported by your implementation.
I modified the previous example to allow selectivity: http://tinyurl.com/28dyk3 The change is that the registration is now done per producer, and the signal includes an ID for the producer (in this example, it sends the address of the signal as the ID). Since the producer must now identify itself, I added a custom little producer class that does so. Note though that a strategy like this is only appropriate for very simple cases (if that) - registrations go in the same bucket for all overloads of operator(), which is probably not desired in general. The long term solution might be to come up with a consumer version of the signals class - a persistent, noncopyable object that holds connections on the consumer end, and to enumerate all consumer ports for the consumer (so that each gets its own tracking object). The library already has port enumeration, but not the consumer tracking object class. It would be a nice addition though, at some point :-) Regards, Stjepan
Hi Stjepan, I have received your request and have added Dataflow.Signals to the review queue. Cheers, ron On Mar 10, 2008, at 9:25 PM, Stjepan Rajko wrote:
Hello,
I would like to request a formal review of the Dataflow.Signals layer of the Dataflow library. The Dataflow.Signals layer provides dataflow functionality based on function calls (using Boost.Signals), and is a result of the Signal Networks GSoC project from 2007. (The other layers of the Dataflow library are not quite ready, so for now I'd like to keep them as implementation details and/or examples of the Dataflow.Signals layer, and submit them for review at a later point.)
The documentation for the Dataflow library is located at: http://dancinghacker.com/code/dataflow/
Version 0.9.0 has been uploaded to the vault, in the Dataflow directory.
There are a few more things I plan to add before the library comes up for review, especially regarding the documentation and motivating examples.
Description:
The Dataflow library is a generic library for dataflow programming. Dataflow programs can typically be expressed as a graph in which vertices represent components that process data, and edges represent the flow of data between the components. As such, dataflow programs can be easily reconfigured by changing the components and/or the connections.
The Dataflow.Signals layer provides dataflow functionality based on function calls - which is one of the fundamental ways of dataflow in C++. For its data transport mechanism, Dataflow.Signals uses Boost.Signals which can be used to make lasting dataflow connections based on function calls.
Dataflow.Signals provides the following to facilitate signals-based dataflow networks:
* A number of useful general-purpose components, and building blocks for implementing new components. * Various free functions and operators for connecting and using components.
Kind regards,
Stjepan _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/ listinfo.cgi/boost
From: "Ronald Garcia" <garcia@cs.indiana.edu>
I have received your request and have added Dataflow.Signals to the review queue.
I'm really excited about this library but is it ready for review yet? Stjepan has kindly kept us up to date with a v0.80/81 and later 0.90 with a fledgling blueprint layer but, playing devil's advocate, perhaps more discussion of what it's trying to solve and how it might make use of other libraries might be useful prior to review. I'm concerned that its fate might be similar to the first revision of the logging library otherwise. I suspect there are many differing/conflicting requirements for a dataflow library. When I've done similar things in the past the critical decision has been whether dataflows are defined at compile time or run-time and what the strategies are for information, buffer and connection management. Even in the last couple of days several areas of functionality and interface have been discussed. Compiler support seems weak too on very popular compilers. Perhaps Stjepan could comment further on his plans between now and review. Its a worthwhile library and something I'm personally very interested in but I'd love to see it fleshed out a bit more with a reasonable application example. This is a library that is not really core to many tasks and probably needs to show its mettle through use. Is it intended to tackle building processing sequences on the fly. Is it lightweight enough to support fine-grained building blocks where connection setup could end up dominating run time if not done well. Is it simply about ease of use in putting together blocks? Is it's uniqueness about supporting dynamic run-time sequence configurability rather than compile-time? I guess I'd like to see the rationale for what its good at and then see compelling evidence (in the form of an example?) Paul
Hi Paul, On Sat, Mar 15, 2008 at 6:02 PM, Paul Baxter <pauljbaxter@hotmail.com> wrote:
I'm really excited about this library but is it ready for review yet?
Thanks for your comments - I understand your concerns, please see my thoughts below.
Stjepan has kindly kept us up to date with a v0.80/81 and later 0.90 with a fledgling blueprint layer but, playing devil's advocate, perhaps more discussion of what it's trying to solve and how it might make use of other libraries might be useful prior to review. I'm concerned that its fate might be similar to the first revision of the logging library otherwise. I suspect there are many differing/conflicting requirements for a dataflow library.
I would definitely agree that there are many differing requirements for a dataflow library. That is why I am only proposing the Signals layer of the review - it was the original focus of the library, and even though the code under and over the Signals layer has been in turmoil for a long time, the interface of the Signals layer itself has been mostly stable for months now. Realizing that made me think I should focus on getting it out for review first. It is possible that there would also be different requirements even for a Boost.Signals-based library - but I'm not as concerned with the fate of the library in terms of acceptance as I am with the feedback I get.
When I've done similar things in the past the critical decision has been whether dataflows are defined at compile time or run-time and what the strategies are for information, buffer and connection management.
One thing that I tried to do is build the Signals layer on top of something generic enough to (eventually) accommodate all of the above. For now, the generic layer is still biased towards the characteristics of Boost.Signals - connectability is determined at compile-time but dataflows are constructed at runtime; connection management and the the operation of the network is left completely to the components and "the outside world". I hope that by review time I can expand the generic layer so it can at least accommodate some different scenarios. That way anyone can adapt or implement their ideal dataflow framework to work with the library.
Even in the last couple of days several areas of functionality and interface have been discussed. Compiler support seems weak too on very popular compilers.
These discussions have been great - and the features discussed would be great additions, but I don't think that their current lack makes the library inadequate. Also, I can make a reasonable effort to expand the compiler support, but I don't see this as a critical issue for review either. I don't want to put too much effort into doing detail work on something that might be shown to need a complete redesign.
Perhaps Stjepan could comment further on his plans between now and review. Its a worthwhile library and something I'm personally very interested in but I'd love to see it fleshed out a bit more with a reasonable application example. This is a library that is not really core to many tasks and probably needs to show its mettle through use.
My plans between now and review mostly have to do with expanding the documentation, with a few implementation pushes as well. On the documentation side it's: * document more of what is implemented * provide better descriptions of examples, and add more examples * reorganize the docs for review so that it is clear what is finished and ready for review, and what is just to be considered an example/proof of concept) On the implementation side it's: * tweak the Dataflow.Signals layer so that components can be composed at compile-time. * address recently discussed ideas / requests. Most of the things above are already in the works, and it might take a long time to get a review manager + time slot anyway. Now, the ideal situation would be that I also manage to do the following: * expand the generic layer to better support non-Boost.Signals-like dataflow frameworks (e.g., determining connectability and other properties at run-time) * add basic support layers for existing dataflow frameworks out there to show that the generic layer can support them. If that is also done by review time, then the review can cover both the Dataflow.Signals layer and the generic layer - otherwise the generic layer can be considered at some other time.
Is it intended to tackle building processing sequences on the fly. Is it lightweight enough to support fine-grained building blocks where connection setup could end up dominating run time if not done well. Is it simply about ease of use in putting together blocks? Is it's uniqueness about supporting dynamic run-time sequence configurability rather than compile-time?
The Dataflow.Signals layer does what boost::signal does - it is not fine grained, it's compile-time connectability-checking with run-time connecting, it's producers with imperfect tracking of consumers and consumers with no tracking of producers, and it couples data-transfer with component invocation. The generic layer, on the other hand, should eventually support all of the scenarios you mention (and I really hope that it too will be ready by review time). Then anyone can take an existing dataflow framework that has the desired characteristics (or develop a new one), make a Dataflow support layer for it, and take advantage of anything that is built on top of Dataflow (like the blueprint layer or the GUI, which are still just in a proof-of-concept stage, but for review I am more interested in showing that they /can/ be built rather than providing full-featured versions).
I guess I'd like to see the rationale for what its good at and then see compelling evidence (in the form of an example?)
I hope this e-mail made things a bit clearer. If you still have questions please let me know - but also I'd love to hear about what your personal dataflow wishlist is, and what sorts of dataflow frameworks/scenarios you've worked with in the past (offline or here). Kind regards, Stjepan
Thanks Stjepan
I hope this e-mail made things a bit clearer. If you still have questions please let me know - but also I'd love to hear about what your personal dataflow wishlist is, and what sorts of dataflow frameworks/scenarios you've worked with in the past (offline or here).
The email helped a lot. The framework I designed was a generic fine-grained signal processing 'infrastructure' for sensor processing. Its premise was that future controllers would want to adjust the processing such that the dataflow would need to be re-wired on the fly. I have previously tried to go from a compile-time solution to a run-time one and found that I couldn't find an easy path for our requirements. My solution had to focus on solving the difficult 'dynamic' problem and then look for opportunities to simplify. I'm still not entirely happy with how my solution ended up. Some sensor applications really only want a largely static, basic 'chain' dataflow with a couple of discrete bolt-on goodies activated by a command flag or two. My approach can be overkill for such applications so I'm looking for 'a better way'! I've been looking to your library to inspire me! In essence I think at present we are coming at things from different ends of the spectrum, both wishing to reach the other end at some point. I'll follow and contribute as best I can. Paul
participants (5)
-
Christian Henning -
Kowalke Oliver (QD IT PA AS) -
Paul Baxter -
Ronald Garcia -
Stjepan Rajko