-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 Hi! I would like to use mpl to generate code along the following lines (I'm trying to build a reusable library for data processing in a 'pipelined' fashion): // Arg1 and Arg2 are instances of 'stages' providing results to this // stage. F is the function taking as arguments const Arg1::result_type& // and const Arg2::result_type&. template<typename Result, typename F, typename Arg1, typename Arg2> struct stage_2 { ~ typedef Result result_type; ~ Result current_; ~ F f_; ~ Arg1 input1_; ~ Arg2 input2_; ~ stage_2(F f, Arg1 a1, Arg2 a2) : f_(f), input1_(a1), input2_(a2) { } ~ const Result &operator() { ~ current_ = f_(input1_(), input2_()); ~ return current_; ~ } }; The question: is it possible not to write manually separate stage_0, stage_1, stage_2, ... for stages taking as input results from the previous 0, 1, 2, ... stages? == Actually, what I'm trying to design: S1 -> S2 -> S3 -> .. -> Sk -> ..-> Sn ~ \_______________/ This is just a rough sketch; in the bottom line the arrow goes from S2 to Sk. Each stage (S'es in the diagram) provides input to its 'children'. This is akin to UNIX shell pipes, but more powerful in that each stage can have multiple inputs. The catch: data transformation at any stage can produce data type on output different from the one received on input. The stages should (obviously?) be parametrized by the stage result, the types of input data and the data transformation to be performed. My last resort is writing a python script to generate the boiler-plate C++ code from high-level description :) Maybe I'm on the wrong way trying to use mpl? any help is appreciated! Thanks. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCyYKoFtofFpCIfhMRAzVDAJ4lMW79S4QS23ivdsLp5xEw/+G+LwCeODR7 fNweo5iwDBnvCKfUcwX3hGI= =UTnS -----END PGP SIGNATURE-----
Zeljko Vrba <zvrba@globalnet.hr> writes:
Hi!
I would like to use mpl to generate code along the following lines (I'm trying to build a reusable library for data processing in a 'pipelined' fashion):
// Arg1 and Arg2 are instances of 'stages' providing results to this // stage. F is the function taking as arguments const Arg1::result_type& // and const Arg2::result_type&. template<typename Result, typename F, typename Arg1, typename Arg2> struct stage_2 { ~ typedef Result result_type;
~ Result current_; ~ F f_; ~ Arg1 input1_; ~ Arg2 input2_;
~ stage_2(F f, Arg1 a1, Arg2 a2) : f_(f), input1_(a1), input2_(a2) { }
~ const Result &operator() { ~ current_ = f_(input1_(), input2_()); ~ return current_; ~ } };
The question: is it possible not to write manually separate stage_0, stage_1, stage_2, ... for stages taking as input results from the previous 0, 1, 2, ... stages?
I don't understand yet. What is the difference between stage_1 and stage_2? Fewer arguments in stage_1?
== Actually, what I'm trying to design:
S1 -> S2 -> S3 -> .. -> Sk -> ..-> Sn ~ \_______________/
This is just a rough sketch; in the bottom line the arrow goes from S2 to Sk. Each stage (S'es in the diagram) provides input to its 'children'.
Why don't you show an example using three or four handwritten "stages;" then we can consider how the code might be generated. -- Dave Abrahams Boost Consulting www.boost-consulting.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 David Abrahams wrote: | | I don't understand yet. What is the difference between stage_1 and | stage_2? Fewer arguments in stage_1? | exactly. e.g. stage reading from file: template<typename Result> struct file_reader { ~ typedef Result result_type; ~ istream f_; ~ const std::pair<Result, bool> &operator()() { ~ static std::pair<Result, bool> r; ~ f_.read(&r.first, sizeof(Result)); ~ r.second = f_; ~ return r; ~ } ~ // some initialization stuff to set f_ }; stage calculating unary function on its input (and input is, of course, a stage): template<typename Result, typename F1, typename Input1> struct unary_op { ~ typedef Result result_type; ~ F1 f_; ~ Input1 input1_; ~ const std::pair<Result, bool> &operator() { ~ static std::pair<Result, bool> r; ~ typename Input1::result_type const &i1 = input1_(); ~ r.first = f_(i1.first); ~ r.second = i1.second; ~ return r; ~ } ~ // some initialization to set input1_ and f_ }; stage adding its two inputs: template<typename Result, typename F2, typename Input1, typename Input2> struct binary_op { ~ typedef Result result_type; ~ F2 f_; ~ Input1 input1_; ~ Input2 input2_; ~ const std::pair<Result, bool> &operator() { ~ static std::pair<Result, bool> r; ~ typename Input1::result_type const &i1 = input1_(); ~ typename Input2::result_type const &i2 = input2_(); ~ r.first = f_(i1.first, i2.first); ~ r.second = i1.second && i2.second; ~ return r; ~ } ~ // init to set i1_ and i2_ }; this would be the intended usage: // // reading input from 2 files, producing as result x + sqrt(y) // file_reader<double> f1(file1); file_reader<double> f2(file2); unary_op<double, ..fill in..> sqrt_stage(f2, sqrt); binary_op<double, ..fill in..> add_stage(f1, sqrt_stage, ~ plus<double, double>()); // this loop would be another "stage" parametrized by the function // performing the actual work done in the body. while(1) { ~ typename adder::result_type r = ad(); ~ if(!r.second) break; ~ // do something with the data } ultimately, would it be possible to write somehting like: file_reader<some_struct> f1(file1); file_reader<another_struct> f2(file2); // note the difference in structs! stages can convert types between // input and output // // do some template magic :) // stage_op( ~ my_consumer_functor(), // obvious ~ stage_op( // producer for the consumer ~ plus<double, double>(), // operation to perform on two inputs ~ f1, // provider for the first argument ~ stage_op( ~ sqrt<double>(), // function for this stage ~ f2))) // provider for this stage If the outer stage_op(my_consumer_functor()..) is removed then I get a 'stream' from which I can extract individual element by calling operator() repeatedly. Please view this final use case as the thing I'm really trying to achieve. The example stage implementations might be on the totally wrong track. Something along current container/iterator[1]/algorithm/functor framework (no, I can't read the whole data set into memory :)), but which would work in a 'lazy' manner producing values on demand only. I believe this is called 'lazy evaluation' in some functional languages. Also I need to be able to define alternate stopping criterion (e.g. stop when ALL streams are exhausted, not the first one - this is useful for merging several streams into a single stream). [1] maybe just define new iterator classes? In the meantime I've read the Phoenix documentation and it seems I could achieve this with Phoenix. I might be mistaken. Tonight I'm writing my first prototype code. I'm open to all suggestions. Thanks in advance for all suggestions. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCye7KFtofFpCIfhMRA+I6AJsGCYmzftPKwYQfolrbryxNbPyMCQCfZo1E c+qqZL0QV4qaSlKlpV5jsXE= =O9qT -----END PGP SIGNATURE-----
Zeljko Vrba <zvrba@globalnet.hr> writes:
David Abrahams wrote: | | I don't understand yet. What is the difference between stage_1 and | stage_2? Fewer arguments in stage_1? | exactly.
e.g. stage reading from file:
template<typename Result> struct file_reader { ~ typedef Result result_type; ~ istream f_;
~ const std::pair<Result, bool> &operator()() { ~ static std::pair<Result, bool> r; ~ f_.read(&r.first, sizeof(Result)); ~ r.second = f_;
<Snip> I know this must sound very fussy, but would you mind posting this material without the leading tildes? I have a hard time reading it. -- Dave Abrahams Boost Consulting www.boost-consulting.com
OK, here it is without leading ~s. It seems GPG inserts them for some strange reason when there is leading space on the line. David Abrahams wrote: | | I don't understand yet. What is the difference between stage_1 and | stage_2? Fewer arguments in stage_1? | exactly. e.g. stage reading from file: template<typename Result> struct file_reader { typedef Result result_type; istream f_; const std::pair<Result, bool> &operator()() { static std::pair<Result, bool> r; f_.read(&r.first, sizeof(Result)); r.second = f_; return r; } // some initialization stuff to set f_ }; stage calculating unary function on its input (and input is, of course, a stage): template<typename Result, typename F1, typename Input1> struct unary_op { typedef Result result_type; F1 f_; Input1 input1_; const std::pair<Result, bool> &operator() { static std::pair<Result, bool> r; typename Input1::result_type const &i1 = input1_(); r.first = f_(i1.first); r.second = i1.second; return r; } // some initialization to set input1_ and f_ }; stage adding its two inputs: template<typename Result, typename F2, typename Input1, typename Input2> struct binary_op { typedef Result result_type; F2 f_; Input1 input1_; Input2 input2_; const std::pair<Result, bool> &operator() { static std::pair<Result, bool> r; typename Input1::result_type const &i1 = input1_(); typename Input2::result_type const &i2 = input2_(); r.first = f_(i1.first, i2.first); r.second = i1.second && i2.second; return r; } // init to set i1_ and i2_ }; this would be the intended usage: // // reading input from 2 files, producing as result x + sqrt(y) // file_reader<double> f1(file1); file_reader<double> f2(file2); unary_op<double, ..fill in..> sqrt_stage(f2, sqrt); binary_op<double, ..fill in..> add_stage(f1, sqrt_stage, plus<double, double>()); // this loop would be another "stage" parametrized by the function // performing the actual work done in the body. while(1) { typename adder::result_type r = ad(); if(!r.second) break; // do something with the data } ultimately, would it be possible to write somehting like: file_reader<some_struct> f1(file1); file_reader<another_struct> f2(file2); // note the difference in structs! stages can convert types between // input and output // // do some template magic :) // stage_op( my_consumer_functor(), // obvious stage_op( // producer for the consumer plus<double, double>(), // operation to perform on two inputs f1, // provider for the first argument stage_op( sqrt<double>(), // function for this stage f2))) // provider for this stage If the outer stage_op(my_consumer_functor()..) is removed then I get a 'stream' from which I can extract individual element by calling operator() repeatedly. Please view this final use case as the thing I'm really trying to achieve. The example stage implementations might be on the totally wrong track. Something along current container/iterator[1]/algorithm/functor framework (no, I can't read the whole data set into memory :)), but which would work in a 'lazy' manner producing values on demand only. I believe this is called 'lazy evaluation' in some functional languages. Also I need to be able to define alternate stopping criterion (e.g. stop when ALL streams are exhausted, not the first one - this is useful for merging several streams into a single stream). [1] maybe just define new iterator classes? In the meantime I've read the Phoenix documentation and it seems I could achieve this with Phoenix. I might be mistaken. Tonight I'm writing my first prototype code. I'm open to all suggestions. Thanks in advance for all suggestions.
Zeljko Vrba <zvrba@globalnet.hr> writes:
OK, here it is without leading ~s. It seems GPG inserts them for some strange reason when there is leading space on the line.
David Abrahams wrote: | | I don't understand yet. What is the difference between stage_1 and | stage_2? Fewer arguments in stage_1? | exactly.
I'm sorry to say this, but I've spent a few minutes trying to understand what you're doing, and I still don't get it. I can't afford to invest a great deal of volunteer time in a general question like this one. If you can condense it into a simple statement, I'm willing to try again, but otherwise I'm not sure I can help you. -- Dave Abrahams Boost Consulting www.boost-consulting.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 David Abrahams wrote: | | like this one. If you can condense it into a simple statement, I'm | willing to try again, but otherwise I'm not sure I can help you. | Think UNIX pipelines: ls -l | sort | gzip > /tmp/blah 1 stage == one operation over the data. ls is provider fo sort which is provider to gzip. i want to mimick the same in C++. each stage consumes data until its provider has some, does some processing on the data and feeds the result to its consumer (but only when its consumer asks it to provide the data - - the model is consumer-driven). but I need more powerful semantic: each stage can have more than one input, it can feed its output to more than one stage and types on input and output may differ. is it a bit clearer now? -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCzA0CFtofFpCIfhMRA7SXAJ9d1F+/LqYid5ZCmv5T0AuAmTaGFQCfeR1F YIuUU/GYzfe1Ny8V77dBJ94= =Yz7J -----END PGP SIGNATURE-----
Zeljko Vrba <zvrba@globalnet.hr> writes:
David Abrahams wrote: | | like this one. If you can condense it into a simple statement, I'm | willing to try again, but otherwise I'm not sure I can help you. | Think UNIX pipelines: ls -l | sort | gzip > /tmp/blah
1 stage == one operation over the data. ls is provider fo sort which is provider to gzip.
i want to mimick the same in C++. each stage consumes data until its provider has some, does some processing on the data and feeds the result to its consumer (but only when its consumer asks it to provide the data - the model is consumer-driven).
but I need more powerful semantic: each stage can have more than one input, it can feed its output to more than one stage and types on input and output may differ.
is it a bit clearer now?
Seems like you want to build a dataflow machine at compile-time. It's a nontrivial problem because, among other things -- since you may have multiple consumers for any piece of data -- you have to buffer results. There aren't any simple answers, AFAICT. If you have a specific question, I can try to answer, but solving the whole design problem is beyond my capacity in my "free" time. -- Dave Abrahams Boost Consulting www.boost-consulting.com
participants (2)
-
David Abrahams -
Zeljko Vrba