[gsoc] the signal network library is now the dataflow library

The summer of code signal network project has taken a turn for the generic, and is now called the Dataflow library (you can think of dataflow programming as "programming by specifying the flow of data through data processing components"). Basically, after originally focusing on using Boost.Signals as the data transport mechanism, I started exploring another data transport mechanism (using simple pointers, and processing the data using Phoenix expressions), which helped me extract some generic dataflow concepts. I have posted the documentation at: http://dancinghacker.com/code/dataflow/ A few notes about the docs, so you know what you're looking at: * The Dataflow.Signals docs are still in transition - if something is confusing, your best bet is to look at the examples. If the examples are confusing... let me know :-) * the doxygen generated reference is broken * I'm still thinking of how to best express the concepts. Even though the documentation (and the code) are still in a "slight" state of flux, at this point I would rather put something out there to get feedback than wait to get everything ironed out. By reading the docs you should get a good idea of where the project has gone, and any comments you might have are very very very welcome. If you want to look at the code, it is in the sandbox (SOC/2007/signals), the tests and examples have been tested with GCC 4.0.1/darwin and MSVC (test_disconnect is known to fail). The summary of the project contributions thus far: 1) A minimal generic dataflow layer which is data-transport mechanism independent, which gives a glimpse of what a generic dataflow library can offer. This layer currently supports component connection, invocation, and grouping. 2) A relatively complete implementation supporting Boost.Signals as the data transport mechanism. This has a number of generally useful signal-based components for dataflow networks, and provides easy connectability. 3) An experimental implementation using mostly pointers as the data transport mechanism + support for Phoenix expressions to process the data. There are some interesting things brewing here - one of the more interesting (to me) is the notion of "inferred" connections (connections that are not stored in memory but inferred from the component). There are some examples of a fibonacci sequence generator that use this (see the docs under Dataflow.Phoenix), and in the future I would also like to experiment with encoding dataflow networks in Proto expression trees. For the GSoC phase I plan to iron out items 1 and 2. Your feedback would be most helpful in deciding what to focus on - what do you find useful? what do you find confusing, awkward, or incorrect? that, and any comments on the implementation are most welcome. It would help me wrap up a more useful GSoC project outcome, as well as plan the future direction of this library. Thank you in advance! Stjepan

Stjepan, I finally finished this week's task for GSoC, so here is my review of your headers: ***1. Documentation*** 1.0: What I like: - You provide excellent motivating examples - You have good arguments for why one should use the library - Your concepts are well thought out - You have extensive code examples that clearly have taken you a lot of time and thought to produce - All of the code examples provide a good depth of what the library is capable of doing 1.1: Nit Picks - Table 1. is hard to read on my monitor, and determine what, if any, hierarchy exists - The section titled "Namespace Use" should be in a section called "How to use this documentation" - In the section named "Dataflow.Signals - based on Boost.Signals", you should indicate that the example encountered previously in the documentation is Dataflow.Signals, or provide another example - The section labelled "Creating your own signal receiver (slot)" should continue using the "fused vs unfused" table paradigm - On page "Disconnecting", 0u is not a number. - You should provide examples of compiling/linking your program for those who do not exclusively use bjam. - There should be a table for all of the components that acts as a "use me when..". When I was trying to write my example, I had only a vague idea of when I should use any of the components, and wasn't sure if I was doing something wrong. I used primarily filter<> and counter<>, as my storage needs were more complicated than I could discern the storage<> class being helpful (I may be wrong.. I only had a few hours to try to do this). ***2. The library:*** 2.1: What I Like: - The "blueprint" layer sounds fantastic. - It is very generic, allowing for all kinds of possibilities, but provides enough specific examples that one can see how it would be useful - I like the networking examples using Boost.Asio 2.2 Questions: - What is useful about a "chain"? Is it that operator() is called n times on the object it receives, where n is defined by the template parameter? 2.3 What could use improvement: - Is there any way you can include something that takes a function pointer of an existing function and makes a function object from it? // Not sure off the top of my head this would work, but something along these // lines could be nice #include <cmath> filter<double (double), unfused> my_filter = make_filter_unfused(&sin); ***3.0 Sample program*** I tried to make a sample program, but ran into problems. gcc threw errors when I attempted to connect() two components, saying "error: invalid use of undefined type 'struct boost::dataflow::extension::signals::get_slot<(void ()(const std::string&), ....>". I'm not going to ask you to debug it, and I got a pretty good feel for everything by writing the code. Overall, I felt that the experience was a little cumbersome, but that for complicated networks the benefits would far outweigh the extra time it takes to define individual components. I merely attempted to do some string processing, which could have easily been dealt with in C++ through more conventional means, but I can see how the plug-and-chug methodology makes algorithm design much easier, were one to have many pre-connected components. Good work so far, Jake

On 8/12/07, Jake Voytko <jakevoytko@gmail.com> wrote:
Stjepan,
I finally finished this week's task for GSoC, so here is my review of your headers:
Yippee!!! Thanks for doing this :-)
***1. Documentation*** 1.0: What I like: - You provide excellent motivating examples - You have good arguments for why one should use the library - Your concepts are well thought out - You have extensive code examples that clearly have taken you a lot of time and thought to produce - All of the code examples provide a good depth of what the library is capable of doing
Great
1.1: Nit Picks - Table 1. is hard to read on my monitor, and determine what, if any, hierarchy exists
Agreed - it's currently a docbook table and as such does little to visually convey the relationships - now that it is starting to stabilize I will put some effort into a better visualization.
- The section titled "Namespace Use" should be in a section called "How to use this documentation"
You are right - most of the stuff in there has more to do about the docs than about the organization. Actually, adding a "How to use this documentation" section would probably do wonders for the usability of the docs. Thanks for the suggestion.
- In the section named "Dataflow.Signals - based on Boost.Signals", you should indicate that the example encountered previously in the documentation is Dataflow.Signals, or provide another example
OK.
- The section labelled "Creating your own signal receiver (slot)" should continue using the "fused vs unfused" table paradigm
I would agree - although I might rethink the whole fused vs. unfused example convention. In so many cases they are identical except for fused vs. unfused, so I'm not sure whether having two almost identical examples is useful or confusing. When it comes to user code though, developing a fused vs. unfused component is usually much different, and in this case a side by side comparison would definitely be useful, as is the case with the doc section you mention.
- On page "Disconnecting", 0u is not a number.
I meant 0 unsigned - I think I was getting a warning if I just put down a zero because it was being compared with size_t or something like that. Is there a problem I am not seeing?
- You should provide examples of compiling/linking your program for those who do not exclusively use bjam.
[cringe]... OK :-)
- There should be a table for all of the components that acts as a "use me when..". When I was trying to write my example, I had only a vague idea of when I should use any of the components, and wasn't sure if I was doing something wrong. I used primarily filter<> and counter<>, as my storage needs were more complicated than I could discern the storage<> class being helpful (I may be wrong.. I only had a few hours to try to do this).
Ooh... I really like this idea.
***2. The library:*** 2.1: What I Like: - The "blueprint" layer sounds fantastic.
Cool! Now I just have to implement it :-) I think this one has been postponed the longest (definitely a post-gsoc undertaking) since I think it might benefit from the reflection and user-friendly graphs libs.
- It is very generic, allowing for all kinds of possibilities, but provides enough specific examples that one can see how it would be useful
I am really happy that it gives that impression.
- I like the networking examples using Boost.Asio
Oh yeah... still gotta implement that async receiver :-)
2.2 Questions: - What is useful about a "chain"? Is it that operator() is called n times on the object it receives, where n is defined by the template parameter?
A chain of, say, 10 components will actually create 10 instances of the component and chain them together (the output of the first connected the input of the 2nd, etc.). The input into the chain goes to the input of the first component, and the output of the chain comes out of the last component. With simple components, the effect is pretty much like invoking one component n times, but the difference would come in when each component could have an independent state that would influence the processing. This, of course, should all be added to the docs :-)
2.3 What could use improvement: - Is there any way you can include something that takes a function pointer of an existing function and makes a function object from it?
// Not sure off the top of my head this would work, but something along these // lines could be nice #include <cmath>
filter<double (double), unfused> my_filter = make_filter_unfused(&sin);
The "function" component does something like this. It can take anything that can be stored in a Boost.Function, and convert it into a filter so that the input to the filter is sent as an argument, and whatever the function returns is sent out as the output. I just looked at the docs for function and realized that they aren't clear at all. No wonder it went unnoticed :-)
***3.0 Sample program*** I tried to make a sample program, but ran into problems. gcc threw errors when I attempted to connect() two components, saying "error: invalid use of undefined type 'struct boost::dataflow::extension::signals::get_slot<(void ()(const std::string&), ....>". I'm not going to ask you to debug it, and I got a pretty good feel for everything by writing the code.
Actually, I would really love to look at your code, because it would give me a glimpse of how you wanted to use the library (it would be especially useful to see it since it didn't let you use it that way). It would help me make the lib more user friendly and/or find some bugs that may have been preventing you from using the lib in what should have been a legal way.
Overall, I felt that the experience was a little cumbersome, but that for complicated networks the benefits would far outweigh the extra time it takes to define individual components. I merely attempted to do some string processing, which could have easily been dealt with in C++ through more conventional means, but I can see how the plug-and-chug methodology makes algorithm design much easier, were one to have many pre-connected components.
Mmm... "plug-and-chug"... I like the sound of that :-) Thank you so very much for taking the time to check out the lib and write up your feedback! It is very insightful and helpful.
Good work so far, Jake
Ditto! Stjepan

On 8/12/07, Stjepan Rajko <stipe@asu.edu> wrote:
On 8/12/07, Jake Voytko <jakevoytko@gmail.com> wrote: I meant 0 unsigned - I think I was getting a warning if I just put down a zero because it was being compared with size_t or something like that. Is there a problem I am not seeing?
No, the problem is on my end. Sorry.
- You should provide examples of compiling/linking your program for those who do not exclusively use bjam.
[cringe]... OK :-)
I know that it is painful, but when I first started using Boost a few years back, one of the areas that gave me the most pain up-front were figuring out which files to link to from MSVC and GCC for projects that were required to be compiled from the command line of both compilers. I still remember the frustration of vague link errors, and surely neophyte users would appreciate the guidance.
The "function" component does something like this. It can take anything that can be stored in a Boost.Function, and convert it into a filter so that the input to the filter is sent as an argument, and whatever the function returns is sent out as the output. I just looked at the docs for function and realized that they aren't clear at all. No wonder it went unnoticed :-)
Ah.. that is good to know. Thank you :)
***3.0 Sample program*** I tried to make a sample program, but ran into problems. gcc threw errors when I attempted to connect() two components, saying "error: invalid use of undefined type 'struct boost::dataflow::extension::signals::get_slot<(void ()(const std::string&), ....>". I'm not going to ask you to debug it, and I got a pretty good feel for everything by writing the code.
Actually, I would really love to look at your code, because it would give me a glimpse of how you wanted to use the library (it would be especially useful to see it since it didn't let you use it that way). It would help me make the lib more user friendly and/or find some bugs that may have been preventing you from using the lib in what should have been a legal way.
I had tried to start from scratch, but I only had a few hours, so I'm guessing the mitigating factor was something simple that I overlooked due to the combination of my unfamiliarity with the library and my lack of time. My laptop isn't with me at the moment, but when I get back to it later today, I will be sure to email you the code.
Thank you so very much for taking the time to check out the lib and write up your feedback! It is very insightful and helpful.
If you need someone to look at your library/docs after GSoC, don't hesitate to send me an email. -Jake
participants (2)
-
Jake Voytko
-
Stjepan Rajko