[fusion] sequence conversion
Dear Boost experts, I'm trying to copy data between different types of fusion sequences with limited success. The example below shows what I'm trying to achieve. Any good advice would be welcome. namespace fields { struct i ; struct l ; struct d ; } int main() { using namespace boost::fusion ; typedef map < pair< fields::i, int >, pair< fields::l, long >, pair< fields::d, double >
map_type ;
typedef vector< int, long, double > vect_t ; vect_t tup1( 1, 2, 3.0 ) ; map_type m = tup1 ; // OK. Works as expected vect_t tup2 = m ; // Error. Conversion not possible // expected result: tup1 == tup2 } Does fusion provide a way to achieve what I'm trying to do? As a side question, I was wondering whether it is possible to synthetize the vect_t type from map_type (with MPL?) A concrete example where this could be useful is an application that manipulates map_type objects and to persist these objects it uses the SOCI database library that supports fusion vectors. Before inserting a row in the database, the map_type object should be converted to a vect_t object and when retrieving a row from the database a vect_t object should converted to a map_type object. Thanks for your help.
AMDG Istvan Buki wrote:
Dear Boost experts,
I'm trying to copy data between different types of fusion sequences with limited success. The example below shows what I'm trying to achieve.
<snip> Does fusion provide a way to achieve what I'm trying to do?
The elements of the map are fusion pair. You need to extract the second element. #include <boost/fusion/include/vector.hpp> #include <boost/fusion/include/map.hpp> #include <boost/fusion/include/pair.hpp> #include <boost/fusion/include/transform.hpp> namespace fields { struct i; struct l; struct d; } template<class F> struct result_of_second; template<class Self, class Arg1> struct result_of_second<Self(Arg1)> { typedef const typename Arg1::second_type& type; }; template<class Self, class Arg1> struct result_of_second<Self(Arg1&)> { typedef typename Arg1::second_type& type; }; template<class Self, class Arg1> struct result_of_second<Self(const Arg1&)> { typedef const typename Arg1::second_type& type; }; struct second { template<class F> struct result { typedef typename result_of_second<F>::type type; }; template<class T> typename T::second_type& operator()(T& t) const { return(t.second); } template<class T> const typename T::second_type& operator()(const T& t) const { return(t.second); } }; int main() { using namespace boost::fusion ; typedef map< pair<fields::i, int>, pair<fields::l, long>, pair<fields::d, double> > map_type; typedef vector<int, long, double> vect_t; vect_t tup1(1, 2, 3.0); map_type m = tup1; vect_t tup2 = transform(m, second()); // expected result: tup1 == tup2 }
As a side question, I was wondering whether it is possible to synthetize the vect_t type from map_type (with MPL?)
Indeed it is possible, typedef result_of::as_vector< mpl::transform< map_type, result_of::second<mpl::_1>, mpl::back_inserter<mpl::vector0<> > >::type
::type vect_t;
In Christ, Steven Watanabe
Steven, your solution is exactly what I needed. My only dream is to be able to manipulate these concepts as well as you do. Thank you very much for your help! On Fri, Jul 18, 2008 at 1:37 AM, Steven Watanabe <watanabesj@gmail.com> wrote:
AMDG
Istvan Buki wrote:
Dear Boost experts,
I'm trying to copy data between different types of fusion sequences with limited success. The example below shows what I'm trying to achieve.
<snip> Does fusion provide a way to achieve what I'm trying to do?
The elements of the map are fusion pair. You need to extract the second element.
#include <boost/fusion/include/vector.hpp> #include <boost/fusion/include/map.hpp> #include <boost/fusion/include/pair.hpp> #include <boost/fusion/include/transform.hpp>
namespace fields { struct i; struct l; struct d; }
template<class F> struct result_of_second;
template<class Self, class Arg1> struct result_of_second<Self(Arg1)> { typedef const typename Arg1::second_type& type; }; template<class Self, class Arg1> struct result_of_second<Self(Arg1&)> { typedef typename Arg1::second_type& type; }; template<class Self, class Arg1> struct result_of_second<Self(const Arg1&)> { typedef const typename Arg1::second_type& type; };
struct second { template<class F> struct result { typedef typename result_of_second<F>::type type; }; template<class T> typename T::second_type& operator()(T& t) const { return(t.second); } template<class T> const typename T::second_type& operator()(const T& t) const { return(t.second); } };
int main() { using namespace boost::fusion ;
typedef map< pair<fields::i, int>, pair<fields::l, long>, pair<fields::d, double>
map_type;
typedef vector<int, long, double> vect_t;
vect_t tup1(1, 2, 3.0); map_type m = tup1; vect_t tup2 = transform(m, second());
// expected result: tup1 == tup2 }
As a side question, I was wondering whether it is possible to synthetize
the vect_t type from map_type (with MPL?)
Indeed it is possible,
typedef result_of::as_vector< mpl::transform< map_type, result_of::second<mpl::_1>, mpl::back_inserter<mpl::vector0<> >
::type ::type vect_t;
In Christ, Steven Watanabe
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On Fri, Jul 18, 2008 at 10:17:28PM +0200, Istvan Buki wrote:
Steven,
your solution is exactly what I needed. My only dream is to be able to manipulate these concepts as well as you do.
Just out of curiosity -- could you please describe your real underlying problem that made you turn to fusion? The fusion manual is severely lacking in motivation, esp. when it comes to transformations, so any examples from the real world are welcome. Or, to reformulate the question: which imminent problem at hand do you have and what is the advantage of solving it by fusion instead of by "traditional" means? Thank you. (PS: I have some experience with functional languages such as ML and I *do* know what *tuples* are useful for.. but the rest? [and the 'rest' does not even cover ML-style pattern matching])
Zeljko Vrba wrote:
On Fri, Jul 18, 2008 at 10:17:28PM +0200, Istvan Buki wrote:
Steven,
your solution is exactly what I needed. My only dream is to be able to manipulate these concepts as well as you do.
Just out of curiosity -- could you please describe your real underlying problem that made you turn to fusion? The fusion manual is severely lacking in motivation, esp. when it comes to transformations, so any
What do you mean by transformations?
examples from the real world are welcome.
Or, to reformulate the question: which imminent problem at hand do you have and what is the advantage of solving it by fusion instead of by "traditional" means?
Again, please define "traditional means".
Thank you.
(PS: I have some experience with functional languages such as ML and I *do* know what *tuples* are useful for.. but the rest? [and the 'rest' does not even cover ML-style pattern matching])
What is "rest" and why should fusion cover ML-style pattern matching? Anyway... If anyone can suggest a motivating example and how to improve the docs, I'd appreciate it a lot. Cheers, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net
On Sat, Jul 19, 2008 at 09:13:08PM +0800, Joel de Guzman wrote:
What do you mean by transformations?
The OP wrote: "I'm trying to copy data between different types of fusion sequences with limited success." If I understand fusion correctly, all compile-time data structures (maps, vectors, lists) are at run-time represented as structs. Is my understanding correct? If so, why provide anything else but vectors? [I have a bunch of other questions, but I'll stop here for now.]
Again, please define "traditional means".
Not using fusion.
What is "rest" and why should fusion cover ML-style pattern matching?
The "rest" is iterators, views, algorithms, ... actually, anything but simple tuples. I did not say that fusion should cover ML-style pattern matching. It was more of a curiosity in the lines of "wow, a bunch of stuff besides tuples, and I have no idea what to do with it.. is there maybe a pattern-matching facility? no.. ok, so what is it good for then?" And I haven't been able to answer the last question to myself. I've browsed through the fusion manual, I've tried to find a use for fusion on the project that I'm currently working on (mostly algorithmical stuff), and I've failed to find where fusion could be useful to me and how it would simplify the program. So, since I'm unable to find a use for fusion in the stuff I'm working on, there are two possible conclusions: 1. The problem lies in the class of problems I'm working on, so I asked the question to find out to which class of problems is fusion suited to. 2. The problem lies in me, in which case the answer to my question will give me a real-life example of other "thinking patterns".
If anyone can suggest a motivating example and how to improve the docs, I'd appreciate it a lot.
A random page from the docs: http://www.boost.org/doc/libs/1_35_0/libs/fusion/doc/html/fusion/container/s... Keys are compile-time types. What is the advantage of set over a simple struct having exactly one element of each type? An example of what is easier achieved this way (or even just a bunch of e.g. variables on the stack) rather than with a struct would be helpful. The documentation has artificial examples. Providing some context, e.g. by describing a real-world problem which caused the algorithm / function / data structure to be added to fusion would be helpful. === OK, copy-paste from some of my code: ++npeers[ret.first->first.first->vid_]; ++npeers[ret.first->first.second->vid_]; ret is a return value from std::map::insert (which is a pair of iterator into the map and boolean), where map is std::map<std::pair<S*, S*>, unsigned> Any way of using fusion to write the above in a nicer way? Though, again, I see no point in using fusion and heavy template machinery; using const references is much easier: const iterator &it = ret.first; const S *key1 = it->first.first; const S *key2 = it->first.second; ++npeers[key1->vid_]; ++npeers[key2->vid_]; So fusion is a tuple library that is supposed to ease manipulation of tuples. If the above code would be a bad use case for applying fusion, what would be a GOOD use case?
Zeljko Vrba wrote:
On Sat, Jul 19, 2008 at 09:13:08PM +0800, Joel de Guzman wrote:
What do you mean by transformations?
The OP wrote: "I'm trying to copy data between different types of fusion sequences with limited success."
If I understand fusion correctly, all compile-time data structures (maps, vectors, lists) are at run-time represented as structs. Is my understanding correct?
No. Cheers, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net
Zeljko Vrba wrote:
Again, please define "traditional means".
Not using fusion.
Huh? Interesting definition :-)
What is "rest" and why should fusion cover ML-style pattern matching?
The "rest" is iterators, views, algorithms, ... actually, anything but simple tuples.
Look at it this way: Imagine a world where STL didn't have iterators and algorithms.
I did not say that fusion should cover ML-style pattern matching. It was more of a curiosity in the lines of "wow, a bunch of stuff besides tuples, and I have no idea what to do with it.. is there maybe a pattern-matching facility? no.. ok, so what is it good for then?"
STL is still immensely useful. It does not have a pattern-matching facility. MPL is still immensely useful. It does not have a pattern-matching facility. I think you are searching the wrong problem domain.
And I haven't been able to answer the last question to myself. I've browsed through the fusion manual, I've tried to find a use for fusion on the project that I'm currently working on (mostly algorithmical stuff), and I've failed to find where fusion could be useful to me and how it would simplify the program.
So, since I'm unable to find a use for fusion in the stuff I'm working on, there are two possible conclusions:
1. The problem lies in the class of problems I'm working on, so I asked the question to find out to which class of problems is fusion suited to.
2. The problem lies in me, in which case the answer to my question will give me a real-life example of other "thinking patterns".
If anyone can suggest a motivating example and how to improve the docs, I'd appreciate it a lot.
A random page from the docs: http://www.boost.org/doc/libs/1_35_0/libs/fusion/doc/html/fusion/container/s...
Keys are compile-time types. What is the advantage of set over a simple struct having exactly one element of each type? An example of what is easier achieved
Introspection, for one. Can you iterate over the types and data of an arbitrary struct? No. Genericity, for another. I can write a generic function (algorithm) that works on *all* fusion maps, more significantly -all fusion sequences.
this way (or even just a bunch of e.g. variables on the stack) rather than with a struct would be helpful.
YMMV. You can even code in assembler, if you like. It's the abstraction!
The documentation has artificial examples. Providing some context, e.g. by describing a real-world problem which caused the algorithm / function / data structure to be added to fusion would be helpful.
This, I agree. It is an area that we need to improve. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net
On Sun, Jul 20, 2008 at 08:14:16AM +0800, Joel de Guzman wrote:
Look at it this way: Imagine a world where STL didn't have iterators and algorithms.
These are tools to get the job done. I can map my problem domain to these tools well, which is not the case with fusion. (As I said earlier -- I do *not* claim that fusion is useless -- just that I might be working in a wrong problem domain, and that's what I'm trying to find out.)
STL is still immensely useful. It does not have a pattern-matching facility. MPL is still immensely useful. It does not have a pattern-matching facility.
Neither claims to implement tuples. Ironically enough, I *do* see how zip/filter/etc. are useful -- I used them extensively when coding in ML. However, in ML they work on lists of arbitrary length, not on tuples of fixed length, and this generality, together with low barrier of entry (friendly syntax, mostly informative error messages), makes functional style of thinking easy and encourages it. These algorithms just "do not seem" useful at compile-time.. at least not when I'm thinking ML where probably most of the features provided by fusion come "for free", as nonintrusive, almost invisible, language features. I guess my expectations of fusion are too high -- not only it heavily uses templates, which I personally consider as very user-unfriendly feature of C++ (syntax, huge and obscure error messages = high barrier to entry even for experimentation), but it also comes so close, but still a bit short of ML's functionality (and comfort).
I think you are searching the wrong problem domain.
Probably yes. Most examples that have emerged so far come from constructing generic libraries. A question from another, "traditional" perspective: can some of fusion's facilities, beyond simple tuples, be useful even if one does not need any variability of data structures and algorithms at compile-time? If so, an example would be nice. Or to reformulate the question again: it would be nice to have examples of using fusion to solve "concrete problems", where data structures may have been already decided, rather than to design interfaces ("abstract problems"). The CSV parser already mentioned is a nice example "in-between" the two extremes -- the "abstract problem" is small (just the decision to use fusion for representing the CSV row), and the rest (filtering, transformations, etc.) are concrete tasks. Or the SQL library example. I would welcome more such examples: little abstraction (re. data structures), a lot of functionality gained.
Introspection, for one. Can you iterate over the types and data of an arbitrary struct? No.
No. But why would I want to? struct is a container for stuff, designed by me for a specific purpose, used by me for that specific purpose, and I *know* what stuff is in the structure and for what purpose it is used for. Arbitrary struct? What is the use of iterating over elements of an *arbitrary* struct if one does not have any semantic knowledge about the contents? OK, serialization might be one limited[1] use-case. More examples? [1] Limited because not even serialization is possible in all cases without semantic knowledge about the contents.
Genericity, for another. I can write a generic function (algorithm) that works on *all* fusion maps, more significantly -all fusion sequences.
Great! What useful could that function do, at run-time? Serialization, as we agreed, is one use case, and also the one shown in the quick start chapter of the manual. What more interesting can you do *at run-time* without semantic knowledge about the contents?
YMMV. You can even code in assembler, if you like. It's the abstraction!
The best programmers code in Fortran. Yes, it's the abstraction -- of WHAT? Abstractions for the sake of themselves are of limited usefulness... even in mathematics, abstractions are created to ease study of certain topics, i.e., they always have (one or more) underlying concrete models, which might again be built on other abstractions, but which give semantics to the abstraction. What is (one of) the underlying semantics that fusion's iterators, algorithms, etc. emerged from? Model and semantics, in this case, would be a concrete runtime problem that needed to be solved.
Zeljko Vrba wrote:
On Sun, Jul 20, 2008 at 08:14:16AM +0800, Joel de Guzman wrote:
Look at it this way: Imagine a world where STL didn't have iterators and algorithms.
These are tools to get the job done. I can map my problem domain to these tools well, which is not the case with fusion. (As I said earlier -- I do *not* claim that fusion is useless -- just that I might be working in a wrong problem domain, and that's what I'm trying to find out.)
STL is still immensely useful. It does not have a pattern-matching facility. MPL is still immensely useful. It does not have a pattern-matching facility.
Neither claims to implement tuples.
They are *all* libraries for data-structures. Data structures + algorithms==programs. That's what it is all about. Fusion is a library for dealing with heterogeneous sequences. The tuple, is just an example of a heterogeneous sequence. STL deals with data structures in the runtime world. MPL, in the compile time world. Fusion, in both. Can't see the connection yet? Don't limit yourself to just thinking in terms of tuples. Think in terms of data-structures, sequences, containers, views.
Ironically enough, I *do* see how zip/filter/etc. are useful -- I used them extensively when coding in ML. However, in ML they work on lists of arbitrary length, not on tuples of fixed length, and this generality, together with low
Wrong analogy. ML has homogeneous lists, like STL. A list (in ML) is essentially a sequence of items but they must be all of the same type.
barrier of entry (friendly syntax, mostly informative error messages), makes functional style of thinking easy and encourages it. These algorithms just "do not seem" useful at compile-time.. at least not when I'm thinking ML where
Are you saying that MPL is not useful too? Out of curiosity, have you used MPL?
probably most of the features provided by fusion come "for free", as nonintrusive, almost invisible, language features.
I guess my expectations of fusion are too high -- not only it heavily uses
No, I think your expectations of fusion is not high -- it is wrong. I am an advocate of FP. I've written FP libraries. There's nothing special about pattern matching and ML. It's just not what fusion is. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net
On Sun, Jul 20, 2008 at 07:39:02PM +0800, Joel de Guzman wrote:
Can't see the connection yet? Don't limit yourself to just thinking in terms of tuples. Think in terms of data-structures, sequences, containers, views.
OK, let's take an example of a CSV parser. Let's say I read the whole CSV file into an std::vector<F>, where F is some fusion container. So far, so good. Now I want a view (abstraction) V that will let me treat i'th column of the dataset as if it were a vector, e.g. V[17] would return me the i'th column of 17th row, and even allow me to modify it. And then this abstraction could be passed around, iterated over, used in STL algorithms, etc. It has the same properties as vector (random access container), except that it's storage is not contiguous. Does this look like something that fusion could help with? (Though, for the record, I would personally not do casual data processing and CSV parsing in C++; the time spent learning fusion and coding processing functions seems much more productively spent in learning something like R.) And no, I don't see the connection. For me, the emperor is naked (see below about maintainability). Maybe I'm stupid, maybe I'm just not easily susceptible to latest.. erm. "technologies", until they have proven their value (again, see below on maintainability to get an idea of my value metrics). Pick what you like.
Wrong analogy. ML has homogeneous lists, like STL. A list (in ML) is essentially a sequence of items but they must be all of the same type.
Except that the "same type" can actually be a variant.
just "do not seem" useful at compile-time.. at least not when I'm thinking ML where
Are you saying that MPL is not useful too? Out of curiosity, have you used MPL?
Stop reinterpreting my words to your convenience. I ignored one such previous case along those lines, but no more. I'm saying exactly what I wrote and what is quoted above: it does not seem useful. This is not the same as saying that it is not useful (which, for the record, although it should be obvious, I did NOT say). No, I have not used MPL. Yes, I've read the introduction and motivation (the example with dimensional algebra), and it seems like a rather niche application. Heck, I've even tried to read the MPL book and the question mark in my head just got bigger, not smaller. It is a reference book about manipulating typelists, but it does not say what typelists are good for. I haven't been able to find a use for MPL myself. From my POV, it's a black magic used by library writers to provide "nice" library interfaces, and which spits out hundreds (or megabytes if the user is unlucky) of error messages if the user does something wrong. And the basic question remains unanswered for me: How does MPL (or fusion for that matter) help in making more maintainable code? More maintainable as in: - less lines of code - code that is easier to understand - code that is easier to inspect at runtime with a debugger - code that can be modified by a C++ developer who does not know every dark corner of the template language - code that is written faster (i.e. takes less time to get a successful compilation) and works correctly the first time it is written How are fusion/MPL useful to an *application* programmer, not a library developer?
on Sun Jul 20 2008, Zeljko Vrba <zvrba-AT-ifi.uio.no> wrote:
just "do not seem" useful at compile-time.. at least not when I'm thinking ML where
Are you saying that MPL is not useful too? Out of curiosity, have you used MPL?
Stop reinterpreting my words to your convenience. I ignored one such previous case along those lines, but no more. I'm saying exactly what I wrote and what is quoted above: it does not seem useful. This is not the same as saying that it is not useful (which, for the record, although it should be obvious, I did NOT say).
Dude, chill; Joel is just trying to help. I know few people in the world less prone to "reinterpret words to his convenience." -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On Sun, Jul 20, 2008 at 09:28:30AM -0400, David Abrahams wrote:
on Sun Jul 20 2008, Zeljko Vrba <zvrba-AT-ifi.uio.no> wrote:
just "do not seem" useful at compile-time.. at least not when I'm thinking ML where
Are you saying that MPL is not useful too? Out of curiosity, have you used MPL?
Stop reinterpreting my words to your convenience. I ignored one such previous case along those lines, but no more. I'm saying exactly what I wrote and what is quoted above: it does not seem useful. This is not the same as saying that it is not useful (which, for the record, although it should be obvious, I did NOT say).
Dude, chill; Joel is just trying to help. I know few people in the world less prone to "reinterpret words to his convenience."
I apologize for the harsh words.
on Sun Jul 20 2008, Zeljko Vrba <zvrba-AT-ifi.uio.no> wrote:
No, I have not used MPL. Yes, I've read the introduction and motivation (the example with dimensional algebra), and it seems like a rather niche application. Heck, I've even tried to read the MPL book and the question mark in my head just got bigger, not smaller. It is a reference book about manipulating typelists, but it does not say what typelists are good for.
"C++ Template Metaprogramming" is not intended to be a reference book about manipulating typelists at all. Did you start reading from the beginning of the book, or did you skip to chapter 3 or so? The introduction lays some important conceptual foundations about what metaprogramming is all about. Typelists (not even a term we use in the book) are merely one means to reaching the many ends of metaprogramming.
I haven't been able to find a use for MPL myself. From my POV, it's a black magic used by library writers to provide "nice" library interfaces,
That's a reasonable point of view to take if you don't want to do template metaprogramming and just want to be a consumer of its benefits. However, if you're trying to use fusion beyond the most basic operations, well, you're doing template metaprogramming. Understanding MPL provides a good foundation for understanding fusion, and in fact fusion uses enough MPL idioms that it would be very difficult to learn any other way.
and which spits out hundreds (or megabytes if the user is unlucky) of error messages if the user does something wrong.
Welcome to C++ template instantiation. The same goes for just about any template library (e.g. STL).
And the basic question remains unanswered for me: How does MPL (or fusion for that matter) help in making more maintainable code? More maintainable as in:
- less lines of code - code that is easier to understand - code that is easier to inspect at runtime with a debugger - code that can be modified by a C++ developer who does not know every dark corner of the template language - code that is written faster (i.e. takes less time to get a successful compilation) and works correctly the first time it is written
Did you look at the binary number example in the first pages of our book? It addresses many of the above questions.
How are fusion/MPL useful to an *application* programmer, not a library developer?
Ah. Well, some people think every application programmer should think of himself as a library developer. If you're writing code that will be re-used by your colleagues, or even by yourself, that code is essentially library code. However, you do have a point: fusion, MPL, and templates in general are useful for building flexible, generalized components that can be configured for different applications without loss of efficiency. If you are building one-off code, you probably don't have much use for them. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On Sun, Jul 20, 2008 at 09:38:58AM -0400, David Abrahams wrote:
"C++ Template Metaprogramming" is not intended to be a reference book about manipulating typelists at all. Did you start reading from the beginning of the book, or did you skip to chapter 3 or so? The introduction lays some important conceptual foundations about what metaprogramming is all about. Typelists (not even a term we use in the book) are merely one means to reaching the many ends of metaprogramming.
Yes, I did start reading from the beginning. Actually, I have it with me. There's "1.6. When Metaprogramming". According to that section, the only thing that I'd like to take advantage of is static type checking. Section 2 has been understandable, though a bunch of reference documentation left me wondering about what all those type classification functions might be useful for. The "flyswapper" example.. nice and understandable, but not something that has really motivated *me* because it looks like a case of invoking a lot of arcane syntax (and I guess it'd have been a lot worse without MPL) to do premature optimization.
That's a reasonable point of view to take if you don't want to do template metaprogramming and just want to be a consumer of its benefits.
Well, I *do not know* whether I want to do template metaprogramming in the sense as in your book. This was the starting point for this whole discussion. I coded a van-Emde Boas tree via template metaprogramming (simple recursion), and that has had little to do with types. I have no idea where and if the MPL could have helped -- most probably not because the code is similar to the binary number example.
Did you look at the binary number example in the first pages of our book? It addresses many of the above questions.
Yes, it was a nice and understandable example which, coincidentally, does not use MPL! So in chapter 1 you begin nice and easy with an understandable example coded without any MPL machinery, and then you suddenly switch in chapter 2 to example with type classification. By the way, how would the binary number example be coded with MPL? Would it be sensible to do it at all?
Ah. Well, some people think every application programmer should think of himself as a library developer. If you're writing code that will be re-used by your colleagues, or even by yourself, that code is essentially library code.
Indeed. "Reused". QT is also reusable, yet it does not use templates heavily.
However, you do have a point: fusion, MPL, and templates in general are useful for building flexible, generalized components that can be configured for different applications without loss of efficiency. If you are building one-off code, you probably don't have much use for them.
OK, thanks. Anyway, I admire yours and Joels (and others') efforts, but.. let's declare 1:0 in favor of MPL and close the discussion. I'll steer clear of MPL/fusion for a while more. Maybe the time has come for me to switch to Java, JITs are steadily getting better :-)
on Sun Jul 20 2008, Zeljko Vrba <zvrba-AT-ifi.uio.no> wrote:
On Sun, Jul 20, 2008 at 09:38:58AM -0400, David Abrahams wrote:
"C++ Template Metaprogramming" is not intended to be a reference book about manipulating typelists at all. Did you start reading from the beginning of the book, or did you skip to chapter 3 or so? The introduction lays some important conceptual foundations about what metaprogramming is all about. Typelists (not even a term we use in the book) are merely one means to reaching the many ends of metaprogramming.
Yes, I did start reading from the beginning. Actually, I have it with me. There's "1.6. When Metaprogramming". According to that section, the only thing that I'd like to take advantage of is static type checking.
Sounds like it's not for you... yet. One day you will begin to appreciate the value of abstraction with performance. Come back and see us then :-)
Section 2 has been understandable, though a bunch of reference documentation left me wondering about what all those type classification functions might be useful for. The "flyswapper" example.. nice and understandable,
So you no longer wonder?
but not something that has really motivated *me* because it looks like a case of invoking a lot of arcane syntax (and I guess it'd have been a lot worse without MPL) to do premature optimization.
Optimizing iter_swap is hardly premature. It is a fundamental operation that governs the efficiency of many algorithms in the standard library.
That's a reasonable point of view to take if you don't want to do template metaprogramming and just want to be a consumer of its benefits.
Well, I *do not know* whether I want to do template metaprogramming in the sense as in your book. This was the starting point for this whole discussion. I coded a van-Emde Boas tree via template metaprogramming (simple recursion), and that has had little to do with types.
What for?
I have no idea where and if the MPL could have helped -- most probably not because the code is similar to the binary number example.
As you probably know, when using MPL one wraps all numeric values inside types so they can be treated polymorphically (e.g. stored in containers); if you're doing purely numeric stuff with little program logic you might find it's easier not to use MPL.
Did you look at the binary number example in the first pages of our book? It addresses many of the above questions.
Yes, it was a nice and understandable example which, coincidentally, does not use MPL!
Well, it's not a coincidence. Doing stuff with MPL requires a small amount of background, which we get to in the following chapters. The upshot, though, is that MPL is a library that makes it much easier to do much more sophisticated things that fall into the same category. If you want to see one of those, you might look at the state machine example in chapter 11.
So in chapter 1 you begin nice and easy with an understandable example coded without any MPL machinery, and then you suddenly switch in chapter 2 to example with type classification.
By the way, how would the binary number example be coded with MPL? Would it be sensible to do it at all?
Probably not. I mean, you could do it, but it would likely add quite a bit of needless syntax.
Ah. Well, some people think every application programmer should think of himself as a library developer. If you're writing code that will be re-used by your colleagues, or even by yourself, that code is essentially library code.
Indeed. "Reused". QT is also reusable, yet it does not use templates heavily.
Yes, but its use of runtime polymorphism incurs a fairly large abstraction penalty.
However, you do have a point: fusion, MPL, and templates in general are useful for building flexible, generalized components that can be configured for different applications without loss of efficiency. If you are building one-off code, you probably don't have much use for them.
OK, thanks. Anyway, I admire yours and Joels (and others') efforts, but.. let's declare 1:0 in favor of MPL and close the discussion.
Was there a contest? Did I miss something?
I'll steer clear of MPL/fusion for a while more. Maybe the time has come for me to switch to Java, JITs are steadily getting better :-)
Ohhh... don't do that. Or at least download and read Stepanov's "elements of programming" slides before you do. :-) -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On Sun, Jul 20, 2008 at 11:36:54AM -0400, David Abrahams wrote:
Sounds like it's not for you... yet. One day you will begin to appreciate the value of abstraction with performance. Come back and see us then :-)
"Abstraction" might be what is bothering me.. but let's postpone that. (i.e. it might be that i'm working on a too concrete level, without planning for parametrization.. as you've already mentioned.)
So you no longer wonder?
Well, no :)
Optimizing iter_swap is hardly premature. It is a fundamental operation that governs the efficiency of many algorithms in the standard library.
Ok, I've never used it myself.
Well, I *do not know* whether I want to do template metaprogramming in the sense as in your book. This was the starting point for this whole discussion. I coded a van-Emde Boas tree via template metaprogramming (simple recursion), and that has had little to do with types.
What for?
Hmm, what "what for"? vEB-tree? I needed O(lg lg n) priority queue. Why metaprogramming: it's a recursive data structure where the same algorithm is executed at every level, with specialization at 0th level. It had to be fast, so I unrolled the recursion at compile-time.
OK, thanks. Anyway, I admire yours and Joels (and others') efforts, but.. let's declare 1:0 in favor of MPL and close the discussion.
Was there a contest? Did I miss something?
A figure of speech. As in, "given my situation and time constraints, I have to postpone tackling MPL and fusion for later". I asked about fusion because I've seen the other mail, and thought it might be easier to start doing something with it (more precisely: the manual mentions compile-time and *run-time*, and that made it look "easier"). Now that you've said that it heavily builds on MPL.. the situation is rather clear: postpone.
I'll steer clear of MPL/fusion for a while more. Maybe the time has come for me to switch to Java, JITs are steadily getting better :-)
Ohhh... don't do that. Or at least download and read Stepanov's "elements of programming" slides before you do. :-)
Oh, the slides have grown very much lately, thanks for the tip! (I've browsed through an early draft of 300-ish pages a while back ago..)
Zeljko Vrba wrote:
On Sun, Jul 20, 2008 at 07:39:02PM +0800, Joel de Guzman wrote:
Can't see the connection yet? Don't limit yourself to just thinking in terms of tuples. Think in terms of data-structures, sequences, containers, views.
OK, let's take an example of a CSV parser. Let's say I read the whole CSV file into an std::vector<F>, where F is some fusion container. So far, so good. Now I want a view (abstraction) V that will let me treat i'th column of the dataset as if it were a vector, e.g. V[17] would return me the i'th column of 17th row, and even allow me to modify it. And then this abstraction could be passed around, iterated over, used in STL algorithms, etc. It has the same properties as vector (random access container), except that it's storage is not contiguous.
Does this look like something that fusion could help with? (Though, for the record, I would personally not do casual data processing and CSV parsing in C++; the time spent learning fusion and coding processing functions seems much more productively spent in learning something like R.)
In that case, it's pointless to answer, or even try to comprehend your question before this paragraph. So, I won't bother. For the record, I would personally spend my time on something more productive.
And no, I don't see the connection. For me, the emperor is naked (see below about maintainability). Maybe I'm stupid, maybe I'm just not easily susceptible to latest.. erm. "technologies", until they have proven their value (again, see below on maintainability to get an idea of my value metrics). Pick what you like.
Wrong analogy. ML has homogeneous lists, like STL. A list (in ML) is essentially a sequence of items but they must be all of the same type.
Except that the "same type" can actually be a variant.
Sure. Like std::vector<boost::variant<int, char double> >. Sorry, again, it's not the same.
just "do not seem" useful at compile-time.. at least not when I'm thinking ML where Are you saying that MPL is not useful too? Out of curiosity, have you used MPL?
Stop reinterpreting my words to your convenience. I ignored one such previous case along those lines, but no more. I'm saying exactly what I wrote and what is quoted above: it does not seem useful. This is not the same as saying that it is not useful (which, for the record, although it should be obvious, I did NOT say).
Ok, pardon me. That's what I really wanted to ask, but I missed a word. Yeah, it's not the same thing.
No, I have not used MPL. Yes, I've read the introduction and motivation (the example with dimensional algebra), and it seems like a rather niche application. Heck, I've even tried to read the MPL book and the question mark in my head just got bigger, not smaller. It is a reference book about manipulating typelists, but it does not say what typelists are good for. I haven't been able to find a use for MPL myself. From my POV, it's a black magic used by library writers to provide "nice" library interfaces, and which spits out hundreds (or megabytes if the user is unlucky) of error messages if the user does something wrong.
Seems Dave gave a better answer than the one I was about to type. Well, if MPL and the TMP book does not connect with you, no amount of Fusion documentation will ever convince you of its merits. On one hand, I want to answer your questions and queries, improve the docs, write examples, etc. On the other hand, I'm starting to feel that no amount of effort will ever make you change your mind, since, it seems, it's already been made. If that's the case, and correct me if I'm wrong, I'd rather spend my time on something more productive. At the very least, however, I *do* agree with you that the docs need improvement in the area you just took mention of: motivation. I do intend to work on it. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net
On Sun, Jul 20, 2008 at 10:32:59PM +0800, Joel de Guzman wrote:
In that case, it's pointless to answer, or even try to comprehend your question before this paragraph. So, I won't bother. For the record, I would personally spend my time on something more productive.
That's understandable -- you have no personal interest in convincing me that fusion is something worth looking into. That's why I asked about real-world, non-abstract examples without further justifications or explanations. And a CSV parser *is* a nice example. (Just that I personally would not choose C++ as the tool for the job. From my current POV, "R" seems like knowledge that is more useful for my long-term professional development than fusion and MPL -- that's what I tried to say. Whether this "seeming" is true or not, is what I'm trying to find out with this discussion.)
Well, if MPL and the TMP book does not connect with you, no amount of Fusion documentation will ever convince you of its merits. On one hand, I want to answer your questions and queries, improve the docs, write examples, etc. On the other hand, I'm starting to feel that no amount of effort will ever make you change your mind, since, it seems, it's already been made. If that's the case, and correct me if I'm wrong, I'd rather spend my time on something more productive.
This is not about *me* -- I guess that there are many others in similar situation like I am. I am an independent learner, and I do not like to ask too many questions. + when one knows little, one can only ask very few sensible questions. In my case, the only sensible question I can ask is "What is it good for?" I *am* convinced that it is good for *something*, but for *what* ? Yes, I'm tough to convince, and I do not expect *you* nor anybody else to convince me into anything. I'm actually already convinced that fusion and MPL are useful and have merits, otherwise I wouldn't be trying to tackle them, and I wouldn't be spending half of the day (sunday!) writing emails -- I'd be doing something "more productive" :-) [except that I consider self-education as being productive, so I'm writing emails] The problem with MPL/fusion is that there are not enough _simple_ real-world examples available to learn from. The existing use-cases are buried deep inside advanced library code that is not suited for learning. The small amounts of code that get posted to the list are very cryptic to the uninitiated, to say the least, and again of little help without the context (like the CSV parser). So how was I (or anybody) supposed to start learning, when all I have is reference documentation with artificial examples, huge amounts of library code, and short snippets without a context mailed to the list together with abstract questions? And *no* examples showing how to *model* real-world problems. Anyway, I apologize once again, and thanks for your time.
On Jul 20, 2008, at 9:21 AM, Zeljko Vrba wrote:
And the basic question remains unanswered for me: How does MPL (or fusion for that matter) help in making more maintainable code? More maintainable as in:
- less lines of code - code that is easier to understand - code that is easier to inspect at runtime with a debugger - code that can be modified by a C++ developer who does not know every dark corner of the template language - code that is written faster (i.e. takes less time to get a successful compilation) and works correctly the first time it is written
How are fusion/MPL useful to an *application* programmer, not a library developer?
I'm not really into the whole MPL scene myself, but I'll give you examples where I used it. A while ago, Boost.Test added support for automatic unit tests. I tried them out, and converted Boost.Rational to use them. The test come in the form of a function you write with a special macro. A similar macro was provided that made a function template instead, and you supply a list of types to apply on that template for testing. The list is in the form of a MPL list: typedef boost::mpl::list<short, int, long, MyInt> rational_testing_list; And I could apply the same tests for different versions of boost::rational<T>. Recently, I redid the integer-selection class templates to use Boost.Test. Before this (like in Boost 1.35) that test (integer_test.cpp) used a LOT of macros that assumed 32-bit values. I wanted something more maintainable by the compiler proper. (For instance, someone extended a macro for 64-bit support, although I never supported that for any other code.) I decided to go with automatic test templates and lists. The base list was like: typelist boost::mpl::vector<unsigned char, unsigned short, unsigned, unsigned long, unsigned long long> unsigned_integer_list; with some alterations: 1. Use "__int64" instead of "long long" on Windows systems 2. Exclude both if I'm not on a 64-bit system 3. Exclude types that are strong-typedefs of a "smaller" type (so "unsigned char" is the only type guaranteed) done by the preprocessor. I use MPL to synthesize the data I actually needed for the tests from this list. 1. Create a list of corresponding signed types (help from Boost.TypeTraits) 2. Create a list of bit lengths used for each integral type, using a custom meta-function I made with std::numeric_limits. This means that I didn't not hard-code the {8, 16, 32, 64} values and the tests will work on platforms without a 64-bit type, or even on old systems that don't use the 8/16/32 system (like 36-bit ints with 9- bit char). 3. A list of all the median bit lengths between the values in [2], so I can test lengths between the exact ones. 4. A combined list of [2] and [3] 5. [4] extended with a length beyond all valid bit lengths In addition, I changed the implementation of the integer-selection classes. Before they could give a compiler error if a given input length was invalid. Now they're MPL-compatible and just lack a member "type" if the input is invalid. (The old templates now call these new ones and still cause a hard error.) Without this, tests using [5] aren't possible. Said tests use SFINAE to make decisions. (I think that MPL lists are limited to 20 elements total. So I made list [3] to be selective. Otherwise, I could have carpet-bombed with range_c<int, 0, 65> or such.) All of these lists are compile-time, but based on a list I couldn't hard-code. It would have been harder without MPL manipulations. -- Daryle Walker Mac, Internet, and Video Game Junkie darylew AT hotmail DOT com
on Sun Jul 20 2008, Daryle Walker <darylew-AT-hotmail.com> wrote:
I think that MPL lists are limited to 20 elements total.
MPL lists are infinitely extensible (at least up to the limitations of your compiler.) There are numbered forms for lists of any practical length longer than 20 elements, and you can always construct longer lists by catenating them. However, it's nearly always more efficient to use MPL vector than MPL list. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On Jul 20, 2008, at 9:04 PM, David Abrahams wrote:
on Sun Jul 20 2008, Daryle Walker <darylew-AT-hotmail.com> wrote:
I think that MPL lists are limited to 20 elements total.
MPL lists are infinitely extensible (at least up to the limitations of your compiler.) There are numbered forms for lists of any practical length longer than 20 elements, and you can always construct longer lists by catenating them. However, it's nearly always more efficient to use MPL vector than MPL list.
I meant "list" in a generic sense. (I actually did start with a mpl::list, but switched to a mpl::vector because some operations weren't supported otherwise.) I think my needs are "impractical"; the numbered-form lists stop at around 50, and I'll need to go to nearly 70 (until 128-bit types come forth). How do you concatenate lists, hopefully so I wouldn't have to change the code that expects a "normal" sized list? -- Daryle Walker Mac, Internet, and Video Game Junkie darylew AT hotmail DOT com
on Mon Jul 21 2008, Daryle Walker <darylew-AT-hotmail.com> wrote:
On Jul 20, 2008, at 9:04 PM, David Abrahams wrote:
on Sun Jul 20 2008, Daryle Walker <darylew-AT-hotmail.com> wrote:
I think that MPL lists are limited to 20 elements total.
MPL lists are infinitely extensible (at least up to the limitations of your compiler.) There are numbered forms for lists of any practical length longer than 20 elements, and you can always construct longer lists by catenating them. However, it's nearly always more efficient to use MPL vector than MPL list.
I meant "list" in a generic sense. (I actually did start with a mpl::list, but switched to a mpl::vector because some operations weren't supported otherwise.) I think my needs are "impractical"; the numbered-form lists stop at around 50, and I'll need to go to nearly 70 (until 128-bit types come forth). How do you concatenate lists, hopefully so I wouldn't have to change the code that expects a "normal" sized list?
I might user joint_view if I were you. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
Zeljko Vrba wrote:
So fusion is a tuple library that is supposed to ease manipulation of tuples. If the above code would be a bad use case for applying fusion, what would be a GOOD use case?
Sure: See spirit2, see Phoenix, see Proto. These are early adopters of fusion. Well, these are probably not what you mean by "traditional", but we couldn't have written these libraries without fusion (and MPL). Dave (Abrahams) has a very nice motivating example, I recall now. In one of his slides, he showed the zip iterator (an iterator adapter) implemented with and without fusion. The fusion version collapsed the code to a fraction of the size without it. Lemme see what I can do. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net
Hi, On Sat, Jul 19, 2008 at 3:13 PM, Joel de Guzman <joel@boost-consulting.com> wrote:
Zeljko Vrba wrote:
On Fri, Jul 18, 2008 at 10:17:28PM +0200, Istvan Buki wrote:
Steven,
your solution is exactly what I needed. My only dream is to be able to manipulate these concepts as well as you do.
Just out of curiosity -- could you please describe your real underlying problem that made you turn to fusion? The fusion manual is severely lacking in motivation, esp. when it comes to transformations, so any
What do you mean by transformations?
examples from the real world are welcome.
Or, to reformulate the question: which imminent problem at hand do you have and what is the advantage of solving it by fusion instead of by "traditional" means?
Again, please define "traditional means".
Thank you.
(PS: I have some experience with functional languages such as ML and I *do* know what *tuples* are useful for.. but the rest? [and the 'rest' does not even cover ML-style pattern matching])
What is "rest" and why should fusion cover ML-style pattern matching?
Anyway...
If anyone can suggest a motivating example and how to improve the docs, I'd appreciate it a lot.
I do not know for sure if what I'm working on is a good example for using fusion. It started as an experiment to learn more about some of the boost libraries. The application is a very small ETL tool which allows me to load and save data from CSV files or database tables. Each record is represented by a fusion map. Between the load and save components I can insert all kind of filters to transform the data contained in the map. The question I asked the other day was more specifically related to the database component that is using the SOCI library which is making use of fusion vectors to represent the rows of a table. I needed a way to copy the data contained in a fusion map into a fusion vector before storing it into the database. One of the next component I will probably develop is a something to load XML messages from a file. I'm not yet decided how I'll represent the XML data in the application. I saw that there is a fusion tree in the ext directory but I'm not sure how stable it is. Steven already helped me a while ago to understand how to work with nested fusion maps. If you have suggestions regarding this subject I would like to hear them. I hope this can help you. Istvan Buki
Istvan Buki wrote:
One of the next component I will probably develop is a something to load XML messages from a file. I'm not yet decided how I'll represent the XML data in the application. I saw that there is a fusion tree in the ext directory but I'm not sure how stable it is. Steven already helped me a while ago to understand how to work with nested fusion maps. If you have suggestions regarding this subject I would like to hear them.
BTW, you might want to check out the Spirit2 example directory. Spirit2 will be an experimental branch in the next 1.36 release and you can find it in both the boost SVN trunk and release branch. There you'll see a toy XML parser that loads its data to a struct adapted as a fusion sequence: struct mini_xml; typedef boost::variant< boost::recursive_wrapper<mini_xml> , std::string > mini_xml_node; struct mini_xml { std::string name; // tag name std::vector<mini_xml_node> children; // children }; //] // We need to tell fusion about our mini_xml struct // to make it a first-class fusion citizen BOOST_FUSION_ADAPT_STRUCT( mini_xml, (std::string, name) (std::vector<mini_xml_node>, children) ) After adapting the mini_xml struct, I was able to work on it just like any fusion sequence. There's some docs (tutorials) in there detailing this. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net
Hi Joel, On Sun, Jul 20, 2008 at 2:26 AM, Joel de Guzman <joel@boost-consulting.com> wrote:
Istvan Buki wrote:
One of the next component I will probably develop is a something to load
XML messages from a file. I'm not yet decided how I'll represent the XML data in the application. I saw that there is a fusion tree in the ext directory but I'm not sure how stable it is. Steven already helped me a while ago to understand how to work with nested fusion maps. If you have suggestions regarding this subject I would like to hear them.
BTW, you might want to check out the Spirit2 example directory. Spirit2 will be an experimental branch in the next 1.36 release and you can find it in both the boost SVN trunk and release branch. There you'll see a toy XML parser that loads its data to a struct adapted as a fusion sequence:
Thanks for the information. I'll work on that as soon as I finish the component that extract/store information from/to an OpenOffice Calc file. Istvan
On Sat, Jul 19, 2008 at 06:16:53PM +0200, Istvan Buki wrote:
Hi,
Hi, thanks for the answer!
I do not know for sure if what I'm working on is a good example for using fusion. It started as an experiment to learn more about some of the boost libraries. The application is a very small ETL tool which allows me to load and save data from CSV files or database tables. Each record is represented by a fusion map. Between the load and save components I can insert all kind of filters to transform the data contained in the map.
This looks like a nice examples, yes. But why do you use maps instead of vectors?
On Sun, Jul 20, 2008 at 8:41 AM, Zeljko Vrba <zvrba@ifi.uio.no> wrote:
On Sat, Jul 19, 2008 at 06:16:53PM +0200, Istvan Buki wrote:
Hi,
Hi, thanks for the answer!
I do not know for sure if what I'm working on is a good example for using fusion. It started as an experiment to learn more about some of the boost
libraries.
The application is a very small ETL tool which allows me to load and save data from CSV files or database tables. Each record is represented by a fusion map. Between the load and save components I can insert all kind of filters to transform the data contained in the map.
This looks like a nice examples, yes. But why do you use maps instead of vectors?
For example, I use it to implement a join component. It works more or less like a join between two database table but here the table records are represented by fusion maps and in each map there is a field used to simulate the where clause. Using the fusion::at_key<> function on a map makes that very easy. It also makes it very easy to filter out a column in a table with the fusion::erase_key<> function. Istvan
On Sun, Jul 20, 2008 at 02:45:04PM +0200, Istvan Buki wrote:
For example, I use it to implement a join component. It works more or less like a join between two database table but here the table records are represented by fusion maps and in each map there is a field used to simulate the where clause. Using the fusion::at_key<> function on a map makes that very easy.
This "where" simulation - what does the field contain? A function object? In that case, which arguments does the function object take? ("Join" is actually a cartesian product, and "where a.x = b.y" over primary keys is just a special case of a predicate that creates "natural join".)
It also makes it very easy to filter out a column in a table with the fusion::erase_key<> function.
What is erasing keys useful for? Which concrete task does it solve? The structure already exists in its fullness in memory after it has been loaded from the file (or am I mistaken there?), so I doubt that it is for runtime efficiency. I mean, isn't "filtering out" more expensive than just ignoring the column? Why do you list this as an advantage? == BTW, I'm not challenging your choices in any way -- I'm just trying to learn something on a concrete example.
On Sun, Jul 20, 2008 at 5:24 PM, Zeljko Vrba <zvrba@ifi.uio.no> wrote:
On Sun, Jul 20, 2008 at 02:45:04PM +0200, Istvan Buki wrote:
For example, I use it to implement a join component. It works more or
less
like a join between two database table but here the table records are represented by fusion maps and in each map there is a field used to simulate the where clause. Using the fusion::at_key<> function on a map makes that very easy.
This "where" simulation - what does the field contain? A function object? In that case, which arguments does the function object take? ("Join" is actually a cartesian product, and "where a.x = b.y" over primary keys is just a special case of a predicate that creates "natural join".)
join was probably not the best analogy. Here is the core function of my "join" component that will make it more clear. InMetaData contain, among other things, the type of the input data. More specifically, InMetaData::type is a fusion map describing all the fields of the input data. CacheMetaData does the same for the cache. IndexType is the type used as the common field in the input data and the cache. I retrieve the value of this field in the input data and use it to do a lookup in the cache. The result of that lookup will be a sond fusion map that I'll join to the map of the input data. This new map will be the result of the "join" component I will pass along to other components that will apply other transformation. typename result_of::join< InMetaData, CacheMetaData, IndexType
::type apply( const typename InMetaData::type &r ) { typename IndexType::value_type key_value = fusion::at_key< IndexType >( r ) ; return fusion::as_map( fusion::join( r, fusion::erase_key< IndexType >( cache_.lookup( key_value ) ) ) ) ; }
It also makes it very easy to filter out a column in a table with the fusion::erase_key<> function.
What is erasing keys useful for? Which concrete task does it solve? The structure already exists in its fullness in memory after it has been loaded from the file (or am I mistaken there?), so I doubt that it is for runtime efficiency.
I mean, isn't "filtering out" more expensive than just ignoring the column? Why do you list this as an advantage?
Take a very simple example. Your input data is a CSV file and you need to create another CSV file from it but with one or more columns removed. Here is the chain of components I would use. +------------------+ +--------------+ +-------------------+ | input CSV | ---->| filter out |----->| output CSV | +------------------+ +--------------+ +-------------------+ If the filter component erase the column, I can just give a template parameter to my component to let it know which column to remove. On the other hand if I work by ignoring what is not needed I need explicitly to tell what I'll copy to the output file which requires more work. In fact it is the base for a more general component that is mapping input fields to output fields either directly or by aggregating the value of several input fields using some function that produce a result which will be written to an output field. You can achieve the same result using one or the other component but in some cases it is easier to go with the less general solution because it is simpler to code.
BTW, I'm not challenging your choices in any way -- I'm just trying to learn something on a concrete example.
No problem. I'm in the same situation as you are: learning fusion and MPL. I just decided that the best way for me to learn is to start working on something concrete which gives me a clear goal. I'll try to use fusion and MPL as much as possible and see how far I can go with them. At this point I do not care much if fusion and MPL are the best choices for this kind of application, I just try to use them in all possible (crazy?) ways. At the end I hope to have a better view about what they are good for, what are the limitations and when I should not use them. I also found that it makes it easier for other people to help you because you ask very specific questions about what you are struggling to implement and it can be answered with few lines of code. I hope this can be of some help.
On Mon, Jul 21, 2008 at 03:27:39PM +0200, Istvan Buki wrote:
InMetaData contain, among other things, the type of the input data. More specifically, InMetaData::type is a fusion map describing all the fields of the input data.
Given that you're using a map, I guess that the user has to define "dummy" types as field names (unless he's satisfied with exactly one value for each primitive type)? Hm, a crazy question: what would happen if I decided to declare map<pair<int, char>, pair<char, char>, pair<int, double> > (note the repeated use of int key).. (That was a rhetoric question, I can try that myself :-))
typename result_of::join< InMetaData, CacheMetaData, IndexType
::type apply( const typename InMetaData::type &r ) { typename IndexType::value_type key_value = fusion::at_key< IndexType >( r ) ; return fusion::as_map( fusion::join( r, fusion::erase_key< IndexType >( cache_.lookup( key_value ) ) ) ) ; }
Given your explanation, I have several questions about this code: - the name apply -- is this function a part of some larger class, so the name is prescribed by some protocol? - You have CacheMetaData as template parameter, yet you do not use it -- you have an instance variable cache_ instead. Is this just for illustration purposes or some other reason? - the return statement prejudices "map" as implementation, i.e., the code does not look generic enough :-) is there some metafunction that could replace fusion::as_map with something more generic, in case the user would like to use e.g. vectors? ditto for erase_key? And now a perceived inconsistency within fusion itself -- vector is clearly a map from a set of integers [0..n-1] to types, yet it is not a model of associative sequence. I.e. after having read the documentation for erase_key, it turns out that for vectors one has to use erase(), but erase_key() for maps.. i'm not saying that it's good or bad, it just strikes me as weird. Could you please also write a simple example of composition of two elements? (I.e. how it would be used by the 'end-user' programmer?)
On the other hand if I work by ignoring what is not needed I need explicitly to tell what I'll copy to the output file which requires more work. In fact
Yes, the same reason occurred to me a bit later too.
something concrete which gives me a clear goal. I'll try to use fusion and MPL as much as possible and see how far I can go with them. At this point I
nice attitude :)
I hope this can be of some help.
Yes, it was helpful. Thank you.
Zeljko Vrba wrote:
Hm, a crazy question: what would happen if I decided to declare
map<pair<int, char>, pair<char, char>, pair<int, double> >
http://www.boost.org/doc/libs/1_35_0/libs/fusion/doc/html/fusion/container/m... "A map may contain at most one element for each key." You'll get a compiler error.
(note the repeated use of int key).. (That was a rhetoric question, I can try that myself :-))
And now a perceived inconsistency within fusion itself -- vector is clearly a map from a set of integers [0..n-1] to types, yet it is not a model of associative sequence.
Then you might also perceive an inconsistency within STL too when a std::vector is not a model of an associative container and a std::vector is not a map? ;-) This is not the place to complain about that. Try the C++ committee :-P
I.e. after having read the documentation for erase_key, it turns out that for vectors one has to use erase(), but erase_key() for maps.. i'm not saying that it's good or bad, it just strikes me as weird.
erase can be applied to maps too. With erase_key, you supply the key. With erase, you supply a range or a position. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net
On Tue, Jul 22, 2008 at 12:08:24AM +0800, Joel de Guzman wrote:
Then you might also perceive an inconsistency within STL too when a std::vector is not a model of an associative container and a std::vector is not a map? ;-) This is not the place to complain about that. Try the C++ committee :-P
To be honest, I've never given it a second thought in the case of STL. I guess that's because nobody tried to "sell" me STL as something abstract and generic, so I've never critically judged it in that light :-P Well, whether this is not the place to complain about that is questionable -- the future needs not repeat the mistakes of the past.. (of course, whether this particular issue is a "mistake" is also debatable, so best to drop it, things are as they are, for better or for worse :))
Zeljko Vrba wrote:
On Tue, Jul 22, 2008 at 12:08:24AM +0800, Joel de Guzman wrote:
Then you might also perceive an inconsistency within STL too when a std::vector is not a model of an associative container and a std::vector is not a map? ;-) This is not the place to complain about that. Try the C++ committee :-P
To be honest, I've never given it a second thought in the case of STL. I guess that's because nobody tried to "sell" me STL as something abstract and generic, so I've never critically judged it in that light :-P
Really? Seems that you know very little of STL. It's been sold by countless people, in books, in mags, as "something abstract and generic". It's the very essence of generic programming in C++.
Well, whether this is not the place to complain about that is questionable -- the future needs not repeat the mistakes of the past.. (of course, whether this particular issue is a "mistake" is also debatable, so best to drop it, things are as they are, for better or for worse :))
Perhaps. But next time you rant on something, better cover the bases and histories first. It seems you haven't even covered STL concepts yet, judging from your words. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net
On Tue, Jul 22, 2008 at 06:57:59AM +0800, Joel de Guzman wrote:
Really? Seems that you know very little of STL. It's been sold by countless people, in books, in mags, as "something abstract and generic". It's the very essence of generic programming in C++.
Yes, I'm aware that it's been sold as "something abstract and generic". Despite this, it boils down to a small set of easy to understand and manually implemented (to me, at this level -- this is an advantage because I understand what's going on, and compiler errors are still understandable to a human!) conventions that enable writing of reusable algorithms for general and special cases. And I don't need a big imagination to see what std::map is good for or when to use std::find. And now, the last sentence brings us back to square one: me trying to understand what MPL and fusion are / could be good for and with you jumping right into the middle of my discussion with Istvan, and you becoming (seemingly at least) frustrated with trying to help me when I never even explicitly asked for help -- it was a question about real-world usage! == Anyway, Dave has already recommended Stepanov's slides. Can you please (yes, *now* I'm asking for help :-)) recommend something else that explains generic programming in C++ [*] and which is not so verbose that it uses several pages and examples to explain why does an adaptable unary function need the "result_type" typedef. (This was just an example, and *not* a reference to Stepanov's slides.. just to give you an idea about what kind of text I'm after.) There's a bunch of papers at Stepanov's site, but what to read and in which order to read it? [*] My basic mistake is, I guess, drawing parallels with ML. Generic programming in C++ is (obviously?) sufficiently different / harder that analogies do not work well.
Zeljko Vrba wrote:
And now, the last sentence brings us back to square one: me trying to understand what MPL and fusion are / could be good for and with you jumping right into the middle of my discussion with Istvan, and you becoming (seemingly at least) frustrated with trying to help me when I never even explicitly asked for help -- it was a question about real-world usage!
Sorry for jumping in the middle of your discussion with Istvan. Pardon me if I seemed frustrated.
Anyway, Dave has already recommended Stepanov's slides. Can you please (yes, *now* I'm asking for help :-)) recommend something else that explains generic programming in C++ [*] and which is not so verbose that it uses several pages and examples to explain why does an adaptable unary function need the "result_type" typedef. (This was just an example, and *not* a reference to Stepanov's slides.. just to give you an idea about what kind of text I'm after.) There's a bunch of papers at Stepanov's site, but what to read and in which order to read it?
[*] My basic mistake is, I guess, drawing parallels with ML. Generic programming in C++ is (obviously?) sufficiently different / harder that analogies do not work well.
IMO, there's nothing hard in GP. It's just a different way of doing things. C++ is multiparadigm. You can do FP, OOP, GP all at the same time. The Stepanov site is a goldmine of information about the subject. Just be patient. There's a good reason why the examples need to explain "result_type". I reckon you can skip the part if you are already familiar with it. Alas, I don't know which to recommend reading first. The papers are listed by date, more or less giving you a historical perspective. I'd also recommend Doug Gregor's "An Introduction to Generic Programming" http://www.generic-programming.org/about/intro/ and his talk on Concept C++: http://video.google.com/videoplay?docid=-1790714981047186825 If you like patterns as I do, I find this one enjoyable: http://www.coldewey.com/europlop2000/papers/geraud+duret.zip and of course: http://www.boost.org/community/generic_programming.html well, those are just a few. HTH, FWIW. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net
On Mon, Jul 21, 2008 at 5:02 PM, Zeljko Vrba <zvrba@ifi.uio.no> wrote:
On Mon, Jul 21, 2008 at 03:27:39PM +0200, Istvan Buki wrote:
InMetaData contain, among other things, the type of the input data. More specifically, InMetaData::type is a fusion map describing all the fields
of
the input data.
Given that you're using a map, I guess that the user has to define "dummy" types as field names (unless he's satisfied with exactly one value for each primitive type)?
Yes, Indeed. But they are not as dummy as one could first think, I use them to store the database field name in case I use one. For example: struct id { typedef fields::id type ; typedef int value_type ; static std::string value_name( void ) { return "id" ; } // return the database field name } ;
typename result_of::join< InMetaData, CacheMetaData, IndexType
::type apply( const typename InMetaData::type &r ) { typename IndexType::value_type key_value = fusion::at_key< IndexType >( r ) ; return fusion::as_map( fusion::join( r, fusion::erase_key< IndexType ( cache_.lookup( key_value ) ) ) ) ; }
Given your explanation, I have several questions about this code:
- the name apply -- is this function a part of some larger class, so the name is prescribed by some protocol?
Yes, the join component derives from a class whose responsibility is to manage input and output connections. In the base class I do: void exec( void ) { output_connection_->value() = static_cast< FuncType *>( this )->apply( input_connection_.value() ) ; }
- You have CacheMetaData as template parameter, yet you do not use it -- you have an instance variable cache_ instead. Is this just for illustration purposes or some other reason?
It is used to create the cache_ object which is a reference to a cache object . Here is the code of the join class that shows more clearly how it works: template < typename InMetaData, typename CacheMetaData, typename IndexType
struct join { typedef typename data_cache< CacheMetaData, IndexType >::impl join_cache ; struct impl : public one2one_component< InMetaData, result_of::join< InMetaData, CacheMetaData, IndexType >, impl > { impl( join_cache &c ) : cache_( c ) { } typename result_of::join< InMetaData, CacheMetaData, IndexType >::type apply( const typename InMetaData::type &r ) { typename IndexType::value_type key_value = fusion::at_key< IndexType >( r ) ; return fusion::as_map( fusion::join( r, fusion::erase_key< IndexType >( cache_.lookup( key_value ) ) ) ) ; } private: join_cache &cache_ ; } ; } ;
- the return statement prejudices "map" as implementation, i.e., the code does not look generic enough :-) is there some metafunction that could replace fusion::as_map with something more generic, in case the user would like to use e.g. vectors? ditto for erase_key?
I did not think about it. Currently my design is to pass map only between components. However, I use some transformation between map and vector in the database component (see beginning of this thread for more detail). This can perhaps be used as the starting point to achieve exactly that. [...]
Could you please also write a simple example of composition of two elements?
(I.e. how it would be used by the 'end-user' programmer?)
Here is a small example that read data from a CSV file, filter out a column and write the result to another CSV file. namespace fields { struct first_name { typedef fields::first_name type ; typedef std::string value_type ; } ; struct last_name { typedef fields::last_name type ; typedef std::string value_type ; } ; struct age { typedef fields::age type ; typedef int value_type ; } ; struct address { typedef fields::address type ; typedef std::string value_type ; } ; struct postal_code { typedef fields::postal_code type ; typedef std::string value_type ; } ; } struct AddressBook { typedef fusion::map < fusion::pair< fields::first_name::type, fields::first_name::value_type
, fusion::pair< fields::last_name::type, fields::last_name::value_type >, fusion::pair< fields::age::type, fields::age::value_type >, fusion::pair< fields::address::type, fields::address::value_type >, fusion::pair< fields::postal_code::type, fields::postal_code::value_type
type ; } ;
int main() { metl::csv_istream< AddressBook >::impl in_file( "addrbook.csv" ) ; typedef metl::filter< AddressBook, fields::age >::impl filter_t ; // remove the 'age' column typedef metl::result_of::filter< AddressBook, fields::age > filter_result_t ; filter_t my_filter ; metl::csv_ostream< filter_result_t >::impl out_file( "out.csv" ) ; // connect the components. in_file.output( my_filter.input() ) ; my_filter.output( out_file.input() ) ; while ( in_file.good() ) { in_file.exec() ; my_filter.exec() ; out_file.exec() ; } return 0 ;
On Mon, Jul 21, 2008 at 07:08:15PM +0200, Istvan Buki wrote:
Here is a small example that read data from a CSV file, filter out a column and write the result to another CSV file.
struct AddressBook { typedef fusion::map < fusion::pair< fields::first_name::type, fields::first_name::value_type
, fusion::pair< fields::last_name::type, fields::last_name::value_type >, fusion::pair< fields::age::type, fields::age::value_type >, fusion::pair< fields::address::type, fields::address::value_type >, fusion::pair< fields::postal_code::type, fields::postal_code::value_type
type ; } ;
Hm, this looks like a lot of duplication and grunt work. I wonder whether MPL or fusion could be used to generate the above given just the type list: <first_name, last_name, age, address, postal_code> (because the pattern is very regular.. each type expands to two types (base type suffixed by ::type and ::value_type), wrapped in fusion::pair, concatenated into a list handed over to fusion::map>)
{ metl::csv_istream< AddressBook >::impl in_file( "addrbook.csv" ) ;
typedef metl::filter< AddressBook, fields::age >::impl filter_t ; // remove the 'age' column
typedef metl::result_of::filter< AddressBook, fields::age > filter_result_t ;
Why do you have to use metl::result_of? Wouldn't it be simpler to have a result typedef inside the filter? ('typedef filter_t::result filter_result_t') Thanks for the example, a while back I did something similar: http://zvrba.net/software/data_stream_toolkit.html The code was used to process huge amounts of data, but it was at the same time my own experiment with "declarative programming" in C++. I wanted to extend the toolkit to be able to reuse the same "declarative computation" at several places without actually redoing the computation several times, but that required storing the functors into variables of unwieldy types.. I also tried experimenting with Phoenix-2 at that time, but it was too difficult (my first experience with megabyte-sized messages from the compiler.) So I ended up with my own conventions for argument and result-types and coding something less general that fullfiled my needs at that time. I'd like to extend the above idea to something like a "subset of SQL in C++", but.. that'll have to wait better times :-(
On Tue, Jul 22, 2008 at 8:45 AM, Zeljko Vrba <zvrba@ifi.uio.no> wrote:
On Mon, Jul 21, 2008 at 07:08:15PM +0200, Istvan Buki wrote:
Here is a small example that read data from a CSV file, filter out a
column
and write the result to another CSV file.
struct AddressBook { typedef fusion::map < fusion::pair< fields::first_name::type,
fields::first_name::value_type
, fusion::pair< fields::last_name::type, fields::last_name::value_type , fusion::pair< fields::age::type, fields::age::value_type >, fusion::pair< fields::address::type, fields::address::value_type >, fusion::pair< fields::postal_code::type, fields::postal_code::value_type
type ; } ;
Hm, this looks like a lot of duplication and grunt work. I wonder whether MPL or fusion could be used to generate the above given just the type list:
<first_name, last_name, age, address, postal_code>
(because the pattern is very regular.. each type expands to two types (base type suffixed by ::type and ::value_type), wrapped in fusion::pair, concatenated into a list handed over to fusion::map>)
Yes, indeed, it is lot of "not so interesting" code to write and MPL could probably come to the rescue here. However, at this point I did not want to work on that because I don't know in which direction my tool will evolve. One of the ideas I have in mind is to create a small GUI that would allow creating the data flow by placing symbols on the window and connecting them with arrows. Then the GUI would generate the corresponding fusion maps, the components, the connections,... and finally call the compiler to create an executable. I'm far from that goal but as you can see I've plenty of crazy ideas ;-)
{ metl::csv_istream< AddressBook >::impl in_file( "addrbook.csv" ) ;
typedef metl::filter< AddressBook, fields::age >::impl filter_t ; // remove the 'age' column
typedef metl::result_of::filter< AddressBook, fields::age > filter_result_t ;
Why do you have to use metl::result_of? Wouldn't it be simpler to have a result typedef inside the filter? ('typedef filter_t::result filter_result_t')
Again, this started as a fusion learning experiment and I decided to use the same convention as fusion to construct the result types. I admit it is not clear to me what reason decided the fusion developers to use the result_of namespace instead of a typedef inside the algorithm classes.
Thanks for the example, a while back I did something similar: http://zvrba.net/software/data_stream_toolkit.html
The code was used to process huge amounts of data, but it was at the same time my own experiment with "declarative programming" in C++. I wanted to extend the toolkit to be able to reuse the same "declarative computation" at several places without actually redoing the computation several times, but that required storing the functors into variables of unwieldy types.. I also tried experimenting with Phoenix-2 at that time, but it was too difficult (my first experience with megabyte-sized messages from the compiler.) So I ended up with my own conventions for argument and result-types and coding something less general that fullfiled my needs at that time.
I'd like to extend the above idea to something like a "subset of SQL in C++", but.. that'll have to wait better times :-(
Thanks for the link. It looks very interesting and shows that there are many ways to solve the same kind of problems. I hope you will find the time to extend it as you wish.
Istvan Buki wrote:
Why do you have to use metl::result_of? Wouldn't it be simpler to have a result typedef inside the filter? ('typedef filter_t::result filter_result_t')
Again, this started as a fusion learning experiment and I decided to use the same convention as fusion to construct the result types. I admit it is not clear to me what reason decided the fusion developers to use the result_of namespace instead of a typedef inside the algorithm classes.
Various reasons: 1) A typedef is not good enough[x]. It has to be a template because the result-type is potentially dependent on the argument types. IOTW, the actual result type can change depending on what you pass. 2) The result_of namespace contains all the metafunction counterparts of all the functions. They mimic, as much as possible, what you have in MPL. So, if you want to do pure type computations, you can use the result_of namespace. [x]: Take a simple function: template <typename T> T square (T val) { return val * val; } - What is the result type of square? Can it be a typedef? - How can you write a C++ utility that will give you this type? E.g. I want to call this function with an argument of int, but I want to know the result type beforehand so I can put the result in a vector. It's obvious what the result is with our example above. But say I have another function with a more complicated result type. How do you automate this task for the user? Cheers, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net
On Tue, Jul 22, 2008 at 10:38 AM, Joel de Guzman <joel@boost-consulting.com>wrote:
Istvan Buki wrote:
Why do you have to use metl::result_of? Wouldn't it be simpler to
have a result typedef inside the filter? ('typedef filter_t::result filter_result_t')
Again, this started as a fusion learning experiment and I decided to use the same convention as fusion to construct the result types. I admit it is not clear to me what reason decided the fusion developers to use the result_of namespace instead of a typedef inside the algorithm classes.
Various reasons: 1) A typedef is not good enough[x]. It has to be a template because the result-type is potentially dependent on the argument types. IOTW, the actual result type can change depending on what you pass.
2) The result_of namespace contains all the metafunction counterparts of all the functions. They mimic, as much as possible, what you have in MPL. So, if you want to do pure type computations, you can use the result_of namespace.
[x]: Take a simple function:
template <typename T> T square (T val) { return val * val; }
- What is the result type of square? Can it be a typedef? - How can you write a C++ utility that will give you this type? E.g. I want to call this function with an argument of int, but I want to know the result type beforehand so I can put the result in a vector.
It's obvious what the result is with our example above. But say I have another function with a more complicated result type. How do you automate this task for the user?
Thanks for explaining. It makes perfect sense now.
On Fri, Jul 18, 2008 at 11:36 PM, Zeljko Vrba <zvrba@ifi.uio.no> wrote:
[snip]
The fusion manual is severely lacking in motivation, esp. when it comes to transformations, so any examples from the real world are welcome.
Hi Zeljko, I have been using Fusion extensively in the Dataflow library (http://www.dancinghacker.com/code/dataflow/), and for many of the uses I can't think of a good replacement for using fusion. Here are some examples: * components of the Dataflow.Signals layer use fusion containers with functional adapters to deal with variable numbers of parameters. Furthermore, once the parameters are received as a fusion Sequence, the component can use fusion transformations to do its job. The storage component, for example, whose job is to store the values of the arguments it receives, has to transform the Sequence vector type into something storable (if it receives fusion::vector<const float, int &> it needs to store the values in a fusion::vector<float, int>). Fusion transformations take care of the type transform, and fusion takes care of getting the values from the parameter vector to the storage vector. * the generic dataflow layer has a KeyedPort concept - which describes a port that actually contains a number of subports. The subport that gets used for a particular operation is determined by the type of the port that the keyed port is interacting with. For example, I might have a keyed port type K that has subports of type KA and type KB. KA is intended for connections with ports of type A, and KB is intended for connections with ports of type B. The keyed port takes care of resolving the right connection automatically - if I try to connect a K port to an A port, it will know to connect KA to A. If I try to connect a K port to a B port, it will know to connect KB to B. This is accomplished using a fusion map. * the components of the Dataflow.Managed layer (implemented recently) uses fusion vectors to store all of its ports (All I have to say is I want a component that consumes a bool and produces a float and an int, e.g., managed::component<bool, mpl::vector<float,int> >, and the component figures out what types of ports it needs and keeps them in a fusion vector). The problem is, the ports are not default constructible. Fusion transformations take care of getting the right argument to the constructors of the ports. These are just some of the examples that come to mind... Kind regards, Stjepan
Stjepan Rajko wrote:
On Fri, Jul 18, 2008 at 11:36 PM, Zeljko Vrba <zvrba@ifi.uio.no> wrote:
[snip]
The fusion manual is severely lacking in motivation, esp. when it comes to transformations, so any examples from the real world are welcome.
Hi Zeljko,
I have been using Fusion extensively in the Dataflow library (http://www.dancinghacker.com/code/dataflow/), and for many of the uses I can't think of a good replacement for using fusion. Here are some examples:
[...]
Stjepan, these are all very nice real world examples. The uses I have Spirit/Phoenix/Proto doesn't seem to connect well with the "traditional" world, but your examples do. Can I use those as part of the "motivation" section of fusion? Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net
On Sat, Jul 19, 2008 at 5:31 PM, Joel de Guzman <joel@boost-consulting.com> wrote:
Stjepan Rajko wrote:
Hi Zeljko,
I have been using Fusion extensively in the Dataflow library (http://www.dancinghacker.com/code/dataflow/), and for many of the uses I can't think of a good replacement for using fusion. Here are some examples:
[...]
Stjepan, these are all very nice real world examples. The uses I have Spirit/Phoenix/Proto doesn't seem to connect well with the "traditional" world, but your examples do. Can I use those as part of the "motivation" section of fusion?
Please feel free :-) Stjepan
participants (7)
-
Daryle Walker -
David Abrahams -
Istvan Buki -
Joel de Guzman -
Steven Watanabe -
Stjepan Rajko -
Zeljko Vrba