Hi Tobias,
So basically we'll want to chain routines, that encapsulate tight loops that work on buffers based on control input.
The framework does not have to provide (at least not as a core component) or know about buffers, but rather implement memory management for arbitrary objects. From the framework's perspective there should be no distinction between buffers and control signals, but different properties of the individual /pins/ (I'm using this term for the sources&sinks here) and the /links/ between them, that control how the framework handles the associated data entities.
Connecting two processing nodes would simply set source and sink data pointer to the same memory location. Eventually (see below) we also have to set up a notify mechanism. Boost.Signal might be the way to go, but I'm not at all sure at this point.
With the library as it stands, it is possible to send references / pointers to objects through the signals, which could be used to work on shared buffers (although I see that this is slightly different from what you are proposing). But, it would be up to the objects to figure out how to share - introducing functionality that would help with shared buffers would definitelly help with signal processing apps that need to be fast. As another performance boost, it might be possible to offer "fast" versions of the provided components, which would use a boost function instead of a boost signal to handle the output. The benefit is increased speed, and the drawback is losing all of the benefits of Boost.Signals (including connecting multiple sinks to the same source directly).
So we get three basic pin types:
- in // processing node reads data from it - out // processing node writes data to it - in_out // processing node changes data in it
With the Boost.Signals base (or even Boost.Function), the "in" pin can be any function, and receiving data is analogous to the function call. "out" is a boost signal, and basically calling the "out" pin results in all of the connected "in" pins (functions) to be called. In_out functionality can be achieved by passing a reference or pointer. If I understand you correctly, you are envisioning something that makes "the wire" an object in itself - a buffer which can be placed in between in and out pins, or something manipulated by an in_out pin. Making connections would then, instead of setting up future function calls which pass the signal data in arguments, be connecting the components to the buffers (through a one-time notification like "here's your buffer"). Then, executing the network would consist of executing each component in the appropriate order. Is this correct?
I could go on like this for a while and outline my image of how such a framework should look like (which is pretty clear by now - I've been working on a design for such a thing for several years, every once in a while) but I'll stop here, for now. Let's first see whether our plans are compatible until here. Maybe you're up to something totally else...
I think we're interested in very similar things.
Further, I think we don't need an overloaded operators interface. Don't get me wrong, I'm not the "oh no - not so much tricky template stuff" kind of guy (in fact, I like template hacking a lot), but I believe the power of this thing lies in the runtime world.
I like the operator interface because you can connect objects (or change / append the network) using something that almost looks like a diagram. But that's just my preference :-)
Serialization and a careful implementation that allows to extend things at runtime by loading shared object files is much more important, than fancy compile time setup, IMO. So is multi-threading - ideally the framework is thread-aware and self-optimizing.
What do you mean by serialization? Yes, thread-aware is very important. Self-optimizing sounds great but I'm not sure what all exactly it can mean and in what all ways it could be tackled. I think as we extend the library towards multi-threading I'll be more aware of what the possible optimizations are.
Also note that I'm not only talking about a signal processing framework but something more general (signal processing gives an interesting use case, though). One could also use it to wire up the data flow in a batch system, resolve GUI dependencies, create vector graphics or use it as a general purpose programming system.
As you can see, I'm potentially interested in collaborating on this project, if I haven't scared you off already ;-).
I do think we are definitelly looking at the same sort of beast :-) Also, I'd be keen to collaborate. I'm going to try to sumbit this as a SoC proposal, and if it gets accepted I think I'll have to be careful to delineate what constitutes the project that I am to complete from things to collaborate on. Have you given the current version of the library a try? If you find that there is a way in which it lets you do the things you are interested in (or something similar), we can use it as a starting point. Otherwise, we can try to rethink the design. If you have any of the code you've been working on available, I'd be happy to try it out and do the same. Cheers, Stjepan