On Fri, Jul 18, 2008 at 10:17:28PM +0200, Istvan Buki wrote:

Steven,

your solution is exactly what I needed. My only dream is to be able to manipulate these concepts as well as you do.

Just out of curiosity -- could you please describe your real underlying problem that made you turn to fusion? The fusion manual is severely lacking in motivation, esp. when it comes to transformations, so any examples from the real world are welcome. Or, to reformulate the question: which imminent problem at hand do you have and what is the advantage of solving it by fusion instead of by "traditional" means? Thank you. (PS: I have some experience with functional languages such as ML and I *do* know what *tuples* are useful for.. but the rest? [and the 'rest' does not even cover ML-style pattern matching])

On Sat, Jul 19, 2008 at 09:13:08PM +0800, Joel de Guzman wrote:

What do you mean by transformations?

The OP wrote: "I'm trying to copy data between different types of fusion sequences with limited success." If I understand fusion correctly, all compile-time data structures (maps, vectors, lists) are at run-time represented as structs. Is my understanding correct? If so, why provide anything else but vectors? [I have a bunch of other questions, but I'll stop here for now.]

Again, please define "traditional means".

Not using fusion.

What is "rest" and why should fusion cover ML-style pattern matching?

The "rest" is iterators, views, algorithms, ... actually, anything but simple tuples. I did not say that fusion should cover ML-style pattern matching. It was more of a curiosity in the lines of "wow, a bunch of stuff besides tuples, and I have no idea what to do with it.. is there maybe a pattern-matching facility? no.. ok, so what is it good for then?" And I haven't been able to answer the last question to myself. I've browsed through the fusion manual, I've tried to find a use for fusion on the project that I'm currently working on (mostly algorithmical stuff), and I've failed to find where fusion could be useful to me and how it would simplify the program. So, since I'm unable to find a use for fusion in the stuff I'm working on, there are two possible conclusions: 1. The problem lies in the class of problems I'm working on, so I asked the question to find out to which class of problems is fusion suited to. 2. The problem lies in me, in which case the answer to my question will give me a real-life example of other "thinking patterns".

If anyone can suggest a motivating example and how to improve the docs, I'd appreciate it a lot.

A random page from the docs: http://www.boost.org/doc/libs/1_35_0/libs/fusion/doc/html/fusion/container/s... Keys are compile-time types. What is the advantage of set over a simple struct having exactly one element of each type? An example of what is easier achieved this way (or even just a bunch of e.g. variables on the stack) rather than with a struct would be helpful. The documentation has artificial examples. Providing some context, e.g. by describing a real-world problem which caused the algorithm / function / data structure to be added to fusion would be helpful. === OK, copy-paste from some of my code: ++npeers[ret.first->first.first->vid_]; ++npeers[ret.first->first.second->vid_]; ret is a return value from std::map::insert (which is a pair of iterator into the map and boolean), where map is std::map, unsigned> Any way of using fusion to write the above in a nicer way? Though, again, I see no point in using fusion and heavy template machinery; using const references is much easier: const iterator &it = ret.first; const S *key1 = it->first.first; const S *key2 = it->first.second; ++npeers[key1->vid_]; ++npeers[key2->vid_]; So fusion is a tuple library that is supposed to ease manipulation of tuples. If the above code would be a bad use case for applying fusion, what would be a GOOD use case?

Zeljko Vrba wrote:

...

Again, please define "traditional means".

Not using fusion.

Huh? Interesting definition :-)

...

What is "rest" and why should fusion cover ML-style pattern matching?

The "rest" is iterators, views, algorithms, ... actually, anything but simple tuples.

Look at it this way: Imagine a world where STL didn't have iterators and algorithms.

I did not say that fusion should cover ML-style pattern matching. It was more of a curiosity in the lines of "wow, a bunch of stuff besides tuples, and I have no idea what to do with it.. is there maybe a pattern-matching facility? no.. ok, so what is it good for then?"

STL is still immensely useful. It does not have a pattern-matching facility. MPL is still immensely useful. It does not have a pattern-matching facility. I think you are searching the wrong problem domain.

And I haven't been able to answer the last question to myself. I've browsed through the fusion manual, I've tried to find a use for fusion on the project that I'm currently working on (mostly algorithmical stuff), and I've failed to find where fusion could be useful to me and how it would simplify the program.

So, since I'm unable to find a use for fusion in the stuff I'm working on, there are two possible conclusions:

1. The problem lies in the class of problems I'm working on, so I asked the question to find out to which class of problems is fusion suited to.

2. The problem lies in me, in which case the answer to my question will give me a real-life example of other "thinking patterns".

...

If anyone can suggest a motivating example and how to improve the docs, I'd appreciate it a lot.

A random page from the docs: http://www.boost.org/doc/libs/1_35_0/libs/fusion/doc/html/fusion/container/s...

Keys are compile-time types. What is the advantage of set over a simple struct having exactly one element of each type? An example of what is easier achieved

Introspection, for one. Can you iterate over the types and data of an arbitrary struct? No. Genericity, for another. I can write a generic function (algorithm) that works on *all* fusion maps, more significantly -all fusion sequences.

this way (or even just a bunch of e.g. variables on the stack) rather than with a struct would be helpful.

YMMV. You can even code in assembler, if you like. It's the abstraction!

The documentation has artificial examples. Providing some context, e.g. by describing a real-world problem which caused the algorithm / function / data structure to be added to fusion would be helpful.

This, I agree. It is an area that we need to improve. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net

Zeljko Vrba wrote:

On Sun, Jul 20, 2008 at 08:14:16AM +0800, Joel de Guzman wrote:

...

Look at it this way: Imagine a world where STL didn't have iterators and algorithms.

These are tools to get the job done. I can map my problem domain to these tools well, which is not the case with fusion. (As I said earlier -- I do *not* claim that fusion is useless -- just that I might be working in a wrong problem domain, and that's what I'm trying to find out.)

...

STL is still immensely useful. It does not have a pattern-matching facility. MPL is still immensely useful. It does not have a pattern-matching facility.

Neither claims to implement tuples.

They are *all* libraries for data-structures. Data structures + algorithms==programs. That's what it is all about. Fusion is a library for dealing with heterogeneous sequences. The tuple, is just an example of a heterogeneous sequence. STL deals with data structures in the runtime world. MPL, in the compile time world. Fusion, in both. Can't see the connection yet? Don't limit yourself to just thinking in terms of tuples. Think in terms of data-structures, sequences, containers, views.

Ironically enough, I *do* see how zip/filter/etc. are useful -- I used them extensively when coding in ML. However, in ML they work on lists of arbitrary length, not on tuples of fixed length, and this generality, together with low

Wrong analogy. ML has homogeneous lists, like STL. A list (in ML) is essentially a sequence of items but they must be all of the same type.

barrier of entry (friendly syntax, mostly informative error messages), makes functional style of thinking easy and encourages it. These algorithms just "do not seem" useful at compile-time.. at least not when I'm thinking ML where

Are you saying that MPL is not useful too? Out of curiosity, have you used MPL?

probably most of the features provided by fusion come "for free", as nonintrusive, almost invisible, language features.

I guess my expectations of fusion are too high -- not only it heavily uses

No, I think your expectations of fusion is not high -- it is wrong. I am an advocate of FP. I've written FP libraries. There's nothing special about pattern matching and ML. It's just not what fusion is. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net

on Sun Jul 20 2008, Zeljko Vrba wrote:

...

...

just "do not seem" useful at compile-time.. at least not when I'm thinking ML where

Are you saying that MPL is not useful too? Out of curiosity, have you used MPL?

Stop reinterpreting my words to your convenience. I ignored one such previous case along those lines, but no more. I'm saying exactly what I wrote and what is quoted above: it does not seem useful. This is not the same as saying that it is not useful (which, for the record, although it should be obvious, I did NOT say).

Dude, chill; Joel is just trying to help. I know few people in the world less prone to "reinterpret words to his convenience." -- Dave Abrahams BoostPro Computing http://www.boostpro.com

on Sun Jul 20 2008, Zeljko Vrba wrote:

No, I have not used MPL. Yes, I've read the introduction and motivation (the example with dimensional algebra), and it seems like a rather niche application. Heck, I've even tried to read the MPL book and the question mark in my head just got bigger, not smaller. It is a reference book about manipulating typelists, but it does not say what typelists are good for.

"C++ Template Metaprogramming" is not intended to be a reference book about manipulating typelists at all. Did you start reading from the beginning of the book, or did you skip to chapter 3 or so? The introduction lays some important conceptual foundations about what metaprogramming is all about. Typelists (not even a term we use in the book) are merely one means to reaching the many ends of metaprogramming.

I haven't been able to find a use for MPL myself. From my POV, it's a black magic used by library writers to provide "nice" library interfaces,

That's a reasonable point of view to take if you don't want to do template metaprogramming and just want to be a consumer of its benefits. However, if you're trying to use fusion beyond the most basic operations, well, you're doing template metaprogramming. Understanding MPL provides a good foundation for understanding fusion, and in fact fusion uses enough MPL idioms that it would be very difficult to learn any other way.

and which spits out hundreds (or megabytes if the user is unlucky) of error messages if the user does something wrong.

Welcome to C++ template instantiation. The same goes for just about any template library (e.g. STL).

And the basic question remains unanswered for me: How does MPL (or fusion for that matter) help in making more maintainable code? More maintainable as in:

- less lines of code - code that is easier to understand - code that is easier to inspect at runtime with a debugger - code that can be modified by a C++ developer who does not know every dark corner of the template language - code that is written faster (i.e. takes less time to get a successful compilation) and works correctly the first time it is written

Did you look at the binary number example in the first pages of our book? It addresses many of the above questions.

How are fusion/MPL useful to an *application* programmer, not a library developer?

Ah. Well, some people think every application programmer should think of himself as a library developer. If you're writing code that will be re-used by your colleagues, or even by yourself, that code is essentially library code. However, you do have a point: fusion, MPL, and templates in general are useful for building flexible, generalized components that can be configured for different applications without loss of efficiency. If you are building one-off code, you probably don't have much use for them. -- Dave Abrahams BoostPro Computing http://www.boostpro.com

on Sun Jul 20 2008, Zeljko Vrba wrote:

On Sun, Jul 20, 2008 at 09:38:58AM -0400, David Abrahams wrote:

...

"C++ Template Metaprogramming" is not intended to be a reference book about manipulating typelists at all. Did you start reading from the beginning of the book, or did you skip to chapter 3 or so? The introduction lays some important conceptual foundations about what metaprogramming is all about. Typelists (not even a term we use in the book) are merely one means to reaching the many ends of metaprogramming.

Yes, I did start reading from the beginning. Actually, I have it with me. There's "1.6. When Metaprogramming". According to that section, the only thing that I'd like to take advantage of is static type checking.

Sounds like it's not for you... yet. One day you will begin to appreciate the value of abstraction with performance. Come back and see us then :-)

Section 2 has been understandable, though a bunch of reference documentation left me wondering about what all those type classification functions might be useful for. The "flyswapper" example.. nice and understandable,

So you no longer wonder?

but not something that has really motivated *me* because it looks like a case of invoking a lot of arcane syntax (and I guess it'd have been a lot worse without MPL) to do premature optimization.

Optimizing iter_swap is hardly premature. It is a fundamental operation that governs the efficiency of many algorithms in the standard library.

...

That's a reasonable point of view to take if you don't want to do template metaprogramming and just want to be a consumer of its benefits.

Well, I *do not know* whether I want to do template metaprogramming in the sense as in your book. This was the starting point for this whole discussion. I coded a van-Emde Boas tree via template metaprogramming (simple recursion), and that has had little to do with types.

What for?

I have no idea where and if the MPL could have helped -- most probably not because the code is similar to the binary number example.

As you probably know, when using MPL one wraps all numeric values inside types so they can be treated polymorphically (e.g. stored in containers); if you're doing purely numeric stuff with little program logic you might find it's easier not to use MPL.

...

Did you look at the binary number example in the first pages of our book? It addresses many of the above questions.

Yes, it was a nice and understandable example which, coincidentally, does not use MPL!

Well, it's not a coincidence. Doing stuff with MPL requires a small amount of background, which we get to in the following chapters. The upshot, though, is that MPL is a library that makes it much easier to do much more sophisticated things that fall into the same category. If you want to see one of those, you might look at the state machine example in chapter 11.

So in chapter 1 you begin nice and easy with an understandable example coded without any MPL machinery, and then you suddenly switch in chapter 2 to example with type classification.

By the way, how would the binary number example be coded with MPL? Would it be sensible to do it at all?

Probably not. I mean, you could do it, but it would likely add quite a bit of needless syntax.

...

Ah. Well, some people think every application programmer should think of himself as a library developer. If you're writing code that will be re-used by your colleagues, or even by yourself, that code is essentially library code.

Indeed. "Reused". QT is also reusable, yet it does not use templates heavily.

Yes, but its use of runtime polymorphism incurs a fairly large abstraction penalty.

...

However, you do have a point: fusion, MPL, and templates in general are useful for building flexible, generalized components that can be configured for different applications without loss of efficiency. If you are building one-off code, you probably don't have much use for them.

OK, thanks. Anyway, I admire yours and Joels (and others') efforts, but.. let's declare 1:0 in favor of MPL and close the discussion.

Was there a contest? Did I miss something?

I'll steer clear of MPL/fusion for a while more. Maybe the time has come for me to switch to Java, JITs are steadily getting better :-)

Ohhh... don't do that. Or at least download and read Stepanov's "elements of programming" slides before you do. :-) -- Dave Abrahams BoostPro Computing http://www.boostpro.com

Zeljko Vrba wrote:

On Sun, Jul 20, 2008 at 07:39:02PM +0800, Joel de Guzman wrote:

...

Can't see the connection yet? Don't limit yourself to just thinking in terms of tuples. Think in terms of data-structures, sequences, containers, views.

OK, let's take an example of a CSV parser. Let's say I read the whole CSV file into an std::vector<F>, where F is some fusion container. So far, so good. Now I want a view (abstraction) V that will let me treat i'th column of the dataset as if it were a vector, e.g. V[17] would return me the i'th column of 17th row, and even allow me to modify it. And then this abstraction could be passed around, iterated over, used in STL algorithms, etc. It has the same properties as vector (random access container), except that it's storage is not contiguous.

Does this look like something that fusion could help with? (Though, for the record, I would personally not do casual data processing and CSV parsing in C++; the time spent learning fusion and coding processing functions seems much more productively spent in learning something like R.)

In that case, it's pointless to answer, or even try to comprehend your question before this paragraph. So, I won't bother. For the record, I would personally spend my time on something more productive.

And no, I don't see the connection. For me, the emperor is naked (see below about maintainability). Maybe I'm stupid, maybe I'm just not easily susceptible to latest.. erm. "technologies", until they have proven their value (again, see below on maintainability to get an idea of my value metrics). Pick what you like.

...

Wrong analogy. ML has homogeneous lists, like STL. A list (in ML) is essentially a sequence of items but they must be all of the same type.

Except that the "same type" can actually be a variant.

Sure. Like std::vector >. Sorry, again, it's not the same.

...

...

just "do not seem" useful at compile-time.. at least not when I'm thinking ML where Are you saying that MPL is not useful too? Out of curiosity, have you used MPL?

Stop reinterpreting my words to your convenience. I ignored one such previous case along those lines, but no more. I'm saying exactly what I wrote and what is quoted above: it does not seem useful. This is not the same as saying that it is not useful (which, for the record, although it should be obvious, I did NOT say).

Ok, pardon me. That's what I really wanted to ask, but I missed a word. Yeah, it's not the same thing.

No, I have not used MPL. Yes, I've read the introduction and motivation (the example with dimensional algebra), and it seems like a rather niche application. Heck, I've even tried to read the MPL book and the question mark in my head just got bigger, not smaller. It is a reference book about manipulating typelists, but it does not say what typelists are good for. I haven't been able to find a use for MPL myself. From my POV, it's a black magic used by library writers to provide "nice" library interfaces, and which spits out hundreds (or megabytes if the user is unlucky) of error messages if the user does something wrong.

Seems Dave gave a better answer than the one I was about to type. Well, if MPL and the TMP book does not connect with you, no amount of Fusion documentation will ever convince you of its merits. On one hand, I want to answer your questions and queries, improve the docs, write examples, etc. On the other hand, I'm starting to feel that no amount of effort will ever make you change your mind, since, it seems, it's already been made. If that's the case, and correct me if I'm wrong, I'd rather spend my time on something more productive. At the very least, however, I *do* agree with you that the docs need improvement in the area you just took mention of: motivation. I do intend to work on it. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net

On Jul 20, 2008, at 9:21 AM, Zeljko Vrba wrote:

And the basic question remains unanswered for me: How does MPL (or fusion for that matter) help in making more maintainable code? More maintainable as in:

- less lines of code - code that is easier to understand - code that is easier to inspect at runtime with a debugger - code that can be modified by a C++ developer who does not know every dark corner of the template language - code that is written faster (i.e. takes less time to get a successful compilation) and works correctly the first time it is written

How are fusion/MPL useful to an *application* programmer, not a library developer?

I'm not really into the whole MPL scene myself, but I'll give you examples where I used it. A while ago, Boost.Test added support for automatic unit tests. I tried them out, and converted Boost.Rational to use them. The test come in the form of a function you write with a special macro. A similar macro was provided that made a function template instead, and you supply a list of types to apply on that template for testing. The list is in the form of a MPL list: typedef boost::mpl::list rational_testing_list; And I could apply the same tests for different versions of boost::rational<T>. Recently, I redid the integer-selection class templates to use Boost.Test. Before this (like in Boost 1.35) that test (integer_test.cpp) used a LOT of macros that assumed 32-bit values. I wanted something more maintainable by the compiler proper. (For instance, someone extended a macro for 64-bit support, although I never supported that for any other code.) I decided to go with automatic test templates and lists. The base list was like: typelist boost::mpl::vector unsigned_integer_list; with some alterations: 1. Use "__int64" instead of "long long" on Windows systems 2. Exclude both if I'm not on a 64-bit system 3. Exclude types that are strong-typedefs of a "smaller" type (so "unsigned char" is the only type guaranteed) done by the preprocessor. I use MPL to synthesize the data I actually needed for the tests from this list. 1. Create a list of corresponding signed types (help from Boost.TypeTraits) 2. Create a list of bit lengths used for each integral type, using a custom meta-function I made with std::numeric_limits. This means that I didn't not hard-code the {8, 16, 32, 64} values and the tests will work on platforms without a 64-bit type, or even on old systems that don't use the 8/16/32 system (like 36-bit ints with 9- bit char). 3. A list of all the median bit lengths between the values in [2], so I can test lengths between the exact ones. 4. A combined list of [2] and [3] 5. [4] extended with a length beyond all valid bit lengths In addition, I changed the implementation of the integer-selection classes. Before they could give a compiler error if a given input length was invalid. Now they're MPL-compatible and just lack a member "type" if the input is invalid. (The old templates now call these new ones and still cause a hard error.) Without this, tests using [5] aren't possible. Said tests use SFINAE to make decisions. (I think that MPL lists are limited to 20 elements total. So I made list [3] to be selective. Otherwise, I could have carpet-bombed with range_c or such.) All of these lists are compile-time, but based on a list I couldn't hard-code. It would have been harder without MPL manipulations. -- Daryle Walker Mac, Internet, and Video Game Junkie darylew AT hotmail DOT com

On Jul 20, 2008, at 9:04 PM, David Abrahams wrote:

on Sun Jul 20 2008, Daryle Walker wrote:

...

I think that MPL lists are limited to 20 elements total.

MPL lists are infinitely extensible (at least up to the limitations of your compiler.) There are numbered forms for lists of any practical length longer than 20 elements, and you can always construct longer lists by catenating them. However, it's nearly always more efficient to use MPL vector than MPL list.

I meant "list" in a generic sense. (I actually did start with a mpl::list, but switched to a mpl::vector because some operations weren't supported otherwise.) I think my needs are "impractical"; the numbered-form lists stop at around 50, and I'll need to go to nearly 70 (until 128-bit types come forth). How do you concatenate lists, hopefully so I wouldn't have to change the code that expects a "normal" sized list? -- Daryle Walker Mac, Internet, and Video Game Junkie darylew AT hotmail DOT com

Zeljko Vrba wrote:

So fusion is a tuple library that is supposed to ease manipulation of tuples. If the above code would be a bad use case for applying fusion, what would be a GOOD use case?

Sure: See spirit2, see Phoenix, see Proto. These are early adopters of fusion. Well, these are probably not what you mean by "traditional", but we couldn't have written these libraries without fusion (and MPL). Dave (Abrahams) has a very nice motivating example, I recall now. In one of his slides, he showed the zip iterator (an iterator adapter) implemented with and without fusion. The fusion version collapsed the code to a fraction of the size without it. Lemme see what I can do. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net

Istvan Buki wrote:

One of the next component I will probably develop is a something to load XML messages from a file. I'm not yet decided how I'll represent the XML data in the application. I saw that there is a fusion tree in the ext directory but I'm not sure how stable it is. Steven already helped me a while ago to understand how to work with nested fusion maps. If you have suggestions regarding this subject I would like to hear them.

BTW, you might want to check out the Spirit2 example directory. Spirit2 will be an experimental branch in the next 1.36 release and you can find it in both the boost SVN trunk and release branch. There you'll see a toy XML parser that loads its data to a struct adapted as a fusion sequence: struct mini_xml; typedef boost::variant< boost::recursive_wrapper , std::string > mini_xml_node; struct mini_xml { std::string name; // tag name std::vector children; // children }; //] // We need to tell fusion about our mini_xml struct // to make it a first-class fusion citizen BOOST_FUSION_ADAPT_STRUCT( mini_xml, (std::string, name) (std::vector, children) ) After adapting the mini_xml struct, I was able to work on it just like any fusion sequence. There's some docs (tutorials) in there detailing this. Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net

On Sat, Jul 19, 2008 at 06:16:53PM +0200, Istvan Buki wrote:

Hi,

Hi, thanks for the answer!

I do not know for sure if what I'm working on is a good example for using fusion. It started as an experiment to learn more about some of the boost libraries. The application is a very small ETL tool which allows me to load and save data from CSV files or database tables. Each record is represented by a fusion map. Between the load and save components I can insert all kind of filters to transform the data contained in the map.

This looks like a nice examples, yes. But why do you use maps instead of vectors?

On Sun, Jul 20, 2008 at 02:45:04PM +0200, Istvan Buki wrote:

For example, I use it to implement a join component. It works more or less like a join between two database table but here the table records are represented by fusion maps and in each map there is a field used to simulate the where clause. Using the fusion::at_key<> function on a map makes that very easy.

This "where" simulation - what does the field contain? A function object? In that case, which arguments does the function object take? ("Join" is actually a cartesian product, and "where a.x = b.y" over primary keys is just a special case of a predicate that creates "natural join".)

It also makes it very easy to filter out a column in a table with the fusion::erase_key<> function.

What is erasing keys useful for? Which concrete task does it solve? The structure already exists in its fullness in memory after it has been loaded from the file (or am I mistaken there?), so I doubt that it is for runtime efficiency. I mean, isn't "filtering out" more expensive than just ignoring the column? Why do you list this as an advantage? == BTW, I'm not challenging your choices in any way -- I'm just trying to learn something on a concrete example.

On Mon, Jul 21, 2008 at 03:27:39PM +0200, Istvan Buki wrote:

InMetaData contain, among other things, the type of the input data. More specifically, InMetaData::type is a fusion map describing all the fields of the input data.

Given that you're using a map, I guess that the user has to define "dummy" types as field names (unless he's satisfied with exactly one value for each primitive type)? Hm, a crazy question: what would happen if I decided to declare map, pair, pair > (note the repeated use of int key).. (That was a rhetoric question, I can try that myself :-))

typename result_of::join< InMetaData, CacheMetaData, IndexType

...

::type apply( const typename InMetaData::type &r ) { typename IndexType::value_type key_value = fusion::at_key< IndexType >( r ) ; return fusion::as_map( fusion::join( r, fusion::erase_key< IndexType >( cache_.lookup( key_value ) ) ) ) ; }

Given your explanation, I have several questions about this code: - the name apply -- is this function a part of some larger class, so the name is prescribed by some protocol? - You have CacheMetaData as template parameter, yet you do not use it -- you have an instance variable cache_ instead. Is this just for illustration purposes or some other reason? - the return statement prejudices "map" as implementation, i.e., the code does not look generic enough :-) is there some metafunction that could replace fusion::as_map with something more generic, in case the user would like to use e.g. vectors? ditto for erase_key? And now a perceived inconsistency within fusion itself -- vector is clearly a map from a set of integers [0..n-1] to types, yet it is not a model of associative sequence. I.e. after having read the documentation for erase_key, it turns out that for vectors one has to use erase(), but erase_key() for maps.. i'm not saying that it's good or bad, it just strikes me as weird. Could you please also write a simple example of composition of two elements? (I.e. how it would be used by the 'end-user' programmer?)

On the other hand if I work by ignoring what is not needed I need explicitly to tell what I'll copy to the output file which requires more work. In fact

Yes, the same reason occurred to me a bit later too.

something concrete which gives me a clear goal. I'll try to use fusion and MPL as much as possible and see how far I can go with them. At this point I

nice attitude :)

I hope this can be of some help.

Yes, it was helpful. Thank you.

On Tue, Jul 22, 2008 at 12:08:24AM +0800, Joel de Guzman wrote:

Then you might also perceive an inconsistency within STL too when a std::vector is not a model of an associative container and a std::vector is not a map? ;-) This is not the place to complain about that. Try the C++ committee :-P

To be honest, I've never given it a second thought in the case of STL. I guess that's because nobody tried to "sell" me STL as something abstract and generic, so I've never critically judged it in that light :-P Well, whether this is not the place to complain about that is questionable -- the future needs not repeat the mistakes of the past.. (of course, whether this particular issue is a "mistake" is also debatable, so best to drop it, things are as they are, for better or for worse :))

On Tue, Jul 22, 2008 at 06:57:59AM +0800, Joel de Guzman wrote:

Really? Seems that you know very little of STL. It's been sold by countless people, in books, in mags, as "something abstract and generic". It's the very essence of generic programming in C++.

Yes, I'm aware that it's been sold as "something abstract and generic". Despite this, it boils down to a small set of easy to understand and manually implemented (to me, at this level -- this is an advantage because I understand what's going on, and compiler errors are still understandable to a human!) conventions that enable writing of reusable algorithms for general and special cases. And I don't need a big imagination to see what std::map is good for or when to use std::find. And now, the last sentence brings us back to square one: me trying to understand what MPL and fusion are / could be good for and with you jumping right into the middle of my discussion with Istvan, and you becoming (seemingly at least) frustrated with trying to help me when I never even explicitly asked for help -- it was a question about real-world usage! == Anyway, Dave has already recommended Stepanov's slides. Can you please (yes, *now* I'm asking for help :-)) recommend something else that explains generic programming in C++ [*] and which is not so verbose that it uses several pages and examples to explain why does an adaptable unary function need the "result_type" typedef. (This was just an example, and *not* a reference to Stepanov's slides.. just to give you an idea about what kind of text I'm after.) There's a bunch of papers at Stepanov's site, but what to read and in which order to read it? [*] My basic mistake is, I guess, drawing parallels with ML. Generic programming in C++ is (obviously?) sufficiently different / harder that analogies do not work well.

On Mon, Jul 21, 2008 at 5:02 PM, Zeljko Vrba wrote:

On Mon, Jul 21, 2008 at 03:27:39PM +0200, Istvan Buki wrote:

...

InMetaData contain, among other things, the type of the input data. More specifically, InMetaData::type is a fusion map describing all the fields

of

...

the input data.

Given that you're using a map, I guess that the user has to define "dummy" types as field names (unless he's satisfied with exactly one value for each primitive type)?

Yes, Indeed. But they are not as dummy as one could first think, I use them to store the database field name in case I use one. For example: struct id { typedef fields::id type ; typedef int value_type ; static std::string value_name( void ) { return "id" ; } // return the database field name } ;

...

typename result_of::join< InMetaData, CacheMetaData, IndexType

...

::type apply( const typename InMetaData::type &r ) { typename IndexType::value_type key_value = fusion::at_key< IndexType >( r ) ; return fusion::as_map( fusion::join( r, fusion::erase_key< IndexType ( cache_.lookup( key_value ) ) ) ) ; }

Given your explanation, I have several questions about this code:

- the name apply -- is this function a part of some larger class, so the name is prescribed by some protocol?

Yes, the join component derives from a class whose responsibility is to manage input and output connections. In the base class I do: void exec( void ) { output_connection_->value() = static_cast< FuncType *>( this )->apply( input_connection_.value() ) ; }

- You have CacheMetaData as template parameter, yet you do not use it -- you have an instance variable cache_ instead. Is this just for illustration purposes or some other reason?

It is used to create the cache_ object which is a reference to a cache object . Here is the code of the join class that shows more clearly how it works: template < typename InMetaData, typename CacheMetaData, typename IndexType

struct join { typedef typename data_cache< CacheMetaData, IndexType >::impl join_cache ; struct impl : public one2one_component< InMetaData, result_of::join< InMetaData, CacheMetaData, IndexType >, impl > { impl( join_cache &c ) : cache_( c ) { } typename result_of::join< InMetaData, CacheMetaData, IndexType >::type apply( const typename InMetaData::type &r ) { typename IndexType::value_type key_value = fusion::at_key< IndexType >( r ) ; return fusion::as_map( fusion::join( r, fusion::erase_key< IndexType >( cache_.lookup( key_value ) ) ) ) ; } private: join_cache &cache_ ; } ; } ;

- the return statement prejudices "map" as implementation, i.e., the code does not look generic enough :-) is there some metafunction that could replace fusion::as_map with something more generic, in case the user would like to use e.g. vectors? ditto for erase_key?

I did not think about it. Currently my design is to pass map only between components. However, I use some transformation between map and vector in the database component (see beginning of this thread for more detail). This can perhaps be used as the starting point to achieve exactly that. [...]

Could you please also write a simple example of composition of two elements?

(I.e. how it would be used by the 'end-user' programmer?)

Here is a small example that read data from a CSV file, filter out a column and write the result to another CSV file. namespace fields { struct first_name { typedef fields::first_name type ; typedef std::string value_type ; } ; struct last_name { typedef fields::last_name type ; typedef std::string value_type ; } ; struct age { typedef fields::age type ; typedef int value_type ; } ; struct address { typedef fields::address type ; typedef std::string value_type ; } ; struct postal_code { typedef fields::postal_code type ; typedef std::string value_type ; } ; } struct AddressBook { typedef fusion::map < fusion::pair< fields::first_name::type, fields::first_name::value_type

, fusion::pair< fields::last_name::type, fields::last_name::value_type >, fusion::pair< fields::age::type, fields::age::value_type >, fusion::pair< fields::address::type, fields::address::value_type >, fusion::pair< fields::postal_code::type, fields::postal_code::value_type

type ; } ;

int main() { metl::csv_istream< AddressBook >::impl in_file( "addrbook.csv" ) ; typedef metl::filter< AddressBook, fields::age >::impl filter_t ; // remove the 'age' column typedef metl::result_of::filter< AddressBook, fields::age > filter_result_t ; filter_t my_filter ; metl::csv_ostream< filter_result_t >::impl out_file( "out.csv" ) ; // connect the components. in_file.output( my_filter.input() ) ; my_filter.output( out_file.input() ) ; while ( in_file.good() ) { in_file.exec() ; my_filter.exec() ; out_file.exec() ; } return 0 ;

On Tue, Jul 22, 2008 at 8:45 AM, Zeljko Vrba wrote:

On Mon, Jul 21, 2008 at 07:08:15PM +0200, Istvan Buki wrote:

...

Here is a small example that read data from a CSV file, filter out a

column

...

and write the result to another CSV file.

...

struct AddressBook { typedef fusion::map < fusion::pair< fields::first_name::type,

fields::first_name::value_type

...

...

, fusion::pair< fields::last_name::type, fields::last_name::value_type , fusion::pair< fields::age::type, fields::age::value_type >, fusion::pair< fields::address::type, fields::address::value_type >, fusion::pair< fields::postal_code::type, fields::postal_code::value_type

type ; } ;

Hm, this looks like a lot of duplication and grunt work. I wonder whether MPL or fusion could be used to generate the above given just the type list:

(because the pattern is very regular.. each type expands to two types (base type suffixed by ::type and ::value_type), wrapped in fusion::pair, concatenated into a list handed over to fusion::map>)

Yes, indeed, it is lot of "not so interesting" code to write and MPL could probably come to the rescue here. However, at this point I did not want to work on that because I don't know in which direction my tool will evolve. One of the ideas I have in mind is to create a small GUI that would allow creating the data flow by placing symbols on the window and connecting them with arrows. Then the GUI would generate the corresponding fusion maps, the components, the connections,... and finally call the compiler to create an executable. I'm far from that goal but as you can see I've plenty of crazy ideas ;-)

...

{ metl::csv_istream< AddressBook >::impl in_file( "addrbook.csv" ) ;

typedef metl::filter< AddressBook, fields::age >::impl filter_t ; // remove the 'age' column

typedef metl::result_of::filter< AddressBook, fields::age > filter_result_t ;

Why do you have to use metl::result_of? Wouldn't it be simpler to have a result typedef inside the filter? ('typedef filter_t::result filter_result_t')

Again, this started as a fusion learning experiment and I decided to use the same convention as fusion to construct the result types. I admit it is not clear to me what reason decided the fusion developers to use the result_of namespace instead of a typedef inside the algorithm classes.

Thanks for the example, a while back I did something similar: http://zvrba.net/software/data_stream_toolkit.html

The code was used to process huge amounts of data, but it was at the same time my own experiment with "declarative programming" in C++. I wanted to extend the toolkit to be able to reuse the same "declarative computation" at several places without actually redoing the computation several times, but that required storing the functors into variables of unwieldy types.. I also tried experimenting with Phoenix-2 at that time, but it was too difficult (my first experience with megabyte-sized messages from the compiler.) So I ended up with my own conventions for argument and result-types and coding something less general that fullfiled my needs at that time.

I'd like to extend the above idea to something like a "subset of SQL in C++", but.. that'll have to wait better times :-(

Thanks for the link. It looks very interesting and shows that there are many ways to solve the same kind of problems. I hope you will find the time to extend it as you wish.

On Tue, Jul 22, 2008 at 10:38 AM, Joel de Guzman wrote:

Istvan Buki wrote:

Why do you have to use metl::result_of? Wouldn't it be simpler to

...

have a result typedef inside the filter? ('typedef filter_t::result filter_result_t')

Again, this started as a fusion learning experiment and I decided to use the same convention as fusion to construct the result types. I admit it is not clear to me what reason decided the fusion developers to use the result_of namespace instead of a typedef inside the algorithm classes.

Various reasons: 1) A typedef is not good enough[x]. It has to be a template because the result-type is potentially dependent on the argument types. IOTW, the actual result type can change depending on what you pass.

2) The result_of namespace contains all the metafunction counterparts of all the functions. They mimic, as much as possible, what you have in MPL. So, if you want to do pure type computations, you can use the result_of namespace.

[x]: Take a simple function:

template <typename T> T square (T val) { return val * val; }

- What is the result type of square? Can it be a typedef? - How can you write a C++ utility that will give you this type? E.g. I want to call this function with an argument of int, but I want to know the result type beforehand so I can put the result in a vector.

It's obvious what the result is with our example above. But say I have another function with a more complicated result type. How do you automate this task for the user?

Thanks for explaining. It makes perfect sense now.

Stjepan Rajko wrote:

On Fri, Jul 18, 2008 at 11:36 PM, Zeljko Vrba wrote:

...

[snip]

The fusion manual is severely lacking in motivation, esp. when it comes to transformations, so any examples from the real world are welcome.

Hi Zeljko,

I have been using Fusion extensively in the Dataflow library (http://www.dancinghacker.com/code/dataflow/), and for many of the uses I can't think of a good replacement for using fusion. Here are some examples:

[...]

Stjepan, these are all very nice real world examples. The uses I have Spirit/Phoenix/Proto doesn't seem to connect well with the "traditional" world, but your examples do. Can I use those as part of the "motivation" section of fusion? Regards, -- Joel de Guzman http://www.boostpro.com http://spirit.sf.net