-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Oliver Kullmann
What is a "plain example"?
An example which shows one thing, without any additional fuss, so that it immediately clear what the example is doing, and how it is achieved.
Those are the ones that you (and I) call contrived. There are many of those.
If I enter the current documentation, then first I get to the short introduction page with a link to the chapter from the mpl book, which is based on earlier chapter in the book, and thus cannot be used as an introduction.
It isn't really based on an earlier chapter.
So then I continue to Topics (the next item on the list), where the natural choice seems to look for "Motivation": But here again there is no plain example, but we need to know at this point what the "is_function<> metafunction in Boost" is (no attempt is made to explain this thing). So the solution presented down the page doesn't motivate anything for anybody who is not familiar with all of Boost (it is not even said where we could find this function in Boost, i.e., to what sub-library it belongs).
Where does the documentation refer to an "is_function<> metafunction in Boost"? (The documentation uses some similar things as examples.) Incidentally, I didn't write the first three of the documents in the "Topics" section, but I did write the rest of them.
Such examples I call "non-plain". A plain example would consider for example the problem of how to create an enumeration out of a variable list of identifiers. The point is here that the example is *simple*, that is, atomic --- not much reference to anything else is needed.
And I'd call that a bad example that promotes a bad use of the library. Taking this one apart, you don't gain anything by defining a simple enum like this. Instead, you might gain something if you define not only the enum, but also surrounding code that makes the enum more useful than a built in enum (e.g. I/O support, etc.). But when you do that, the example is suddenly much more complicated. As I said before, this kind of thing does not boil down to simple use cases--which is exactly what this kind of example is. Instead, when the abstractions provided are brought all the way down to concrete use cases, those use cases are either bad uses or are complex. This is almost invariably true. If it wasn't, you wouldn't need preprocessor metaprogramming to do it.
and thus I skipped that and went to Reference, where I found BOOST_PP_SEQ_FOR_EACH, making some still mysterious remarks about "The next available BOOST_PP_FOR repetition." etc. But at various places it somehow says that "reentry problems" whatever this means are solved now.
There is an article in the topics section about this (titled "Reentrancy").
How does the reader of the documentation get there? After I found that the Motivation does not motivate much for me, I went on to "Known problems" --- alright, I understand that, but it doesn't give me much. So well, going to techniques : Again the very first example doesn't say what it is supposed to do, and so it continues; all examples are part of some discussion within an expert round of users (who don't bother to even spell out the problems they are discussing).
Expert users are the primary targets of the documentation. This is a generative _metaprogramming_ library. There are all kinds of things that you can do with metaprogramming--one of them is write code that is total crap. The target of the library is advanced programmers, because only advanced programmers have accumulated the design experience necessary to evaluate the tradeoffs involved. Furthermore, many Boost authors can readily identify with the underlying problem that something like 'is_function' takes in stride (and what it really is an example of). That's dealing with variable arity--a very common area in which the library is used.
So let's skip techniques. Next comes "Incompatibilities". It speaks about the previous Boost release 1.28, so it seems not of relevance anymore (especially since I'm a new user).
No, that one isn't relevant to you.
So let's skip this, going to "reentrancy". From the headline it's not clear what could be meant by it (if you don't know it). Alright, the first few paragraphs there name the problem. But actually not explicitely, it doesn't say that C or C++ in version soandso has this "feature", but it speaks of "the preprocessor", which might mean a specific compiler, some special features or this very library itself.
The preprocessor is defined by the standard, and that is what the documentation is referring to. Many real world preprocessors approximate it closely, and some are horribly broken.
It also speaks of recursion, which in the usual sense of the word (that is, in mathematics) is not what is really meant.
I'm not sure I follow this. "Recursion", as I'm using the term in the documentation and as it is commonly used in computer science, is the ability for an interface to use itself.
I want to make the point here, that as a documentation author one has similar responsibilities to a car driver: One must anticipate problems. Readers always have a certain special angle how the view it, and one must add many hints and redundancies to help them becoming aware of the incongruence of their point of view with the authors point of view.
I don't disagree with that, but I also can't accommodate every possible point of view.
My main assumption was, that C and C++ respects the programmer, and thus does not disallow potentially "dangerous constructions". So in my mind I glanced over the problem with "recursion", and basically thought that this is no problem for C and C++ (the whole process might go wrong, but we are in control). Coming from this point of view, I then see "recursion", guess they mean something different, and moreover I have already seen somewhere that "reentrancy" (whatever this means) is solved by the new version of the library anyway,
The documentation doesn't say that *anywhere*.
so I don't need to bother (not to forget that when I come to this point in the documentation, then I already "learned" that apparently most parts of the documentation have to be skipped).
Or read with an attempt to understand what they are saying...
Only when looking it up in the rationale of C99 I found a clear statement about the issue (while the C++ standard somehow supposes you know already the issue).
It is pretty clear in both standards (6.10.3.4/2 in C and 16.3.4/2 in C++).
I'm not aware of *any* book on C or C++ discussing this issue. I have quite a collection of them, read (or have read) CUJ, CVu and Overload, but the preprocessor seems to be a tabu. This I think one should keep in mind for the documentation.
So, what you're saying is: 1) there is no 20+ years of existing literature as there is for C and C++, so 2) this library should provide the equivalent of 20+ years of literature. The bottom line is that the responsibility of the documentation stops after documenting itself. Anything beyond that is a freebie--the documentation is going above an beyond the call, so-to-speak. I'm not against that, but I'm hearing nothing but "everything is poorly done".
Alright, leaves the standard(s). I worked through all examples there, but none of them shows the "recursion problem" (at least in the C++ standard). Sure, finally one gets to the point where you understand what they are talking about, but it takes (too) long. You start reading this chapter
I agree that the standard is not elucidating. I disagree that it is my _responsibility_ (via the documentation) to elucidate it, regardless of how useful it might be to do so. Frankly, the lack of recursion is an fundamental aspect of the way macro expansion works. If you don't know that (and I don't mean all the details down to the subatomic level), you probably aren't *ready* to use this library.
And so on. Among all of that you have to find the right few sentences.
Yep. That's because it is a technical specification, not an explanation. Yes, it is difficult to follow and get all the details. Yes, the elucidation that you want would be useful. No, it is _not_ my responsibility to do so--even though I'm already in the process of doing exactly that for macro expansion. On top of all this, I'm doing it in my spare time for free. Just like I'm spending hours on this conversation (and putting off other things that I'm working on).
First, text read by humans is different from text read by computers. Due to the inherent imprecision of natural language, and the necessity for the reader to anticipate meaning (a well-know psychological fact that without anticipation no understanding), the reader constantly goes into wrong directions, and only redundancies (saying important things at many places (with different words)) can help him.
Technical documentation needs to be concise. I shouldn't have to say the same thing twice. If the reader doesn't understand something later, they can go back and re-read it.
And second, if the documentation would be organised like software, this might actually help: Every file(!) (not just every translation unit) is supposed to include for example <algorithm> again and again --- accordingly one would at least expect at every reference for some construction (corresponding to a source code file) a *link* to the relevant general sources here ("including" that what needs to be known to understand the reference).
I agree with this--as long as it is about library-defined issues. E.g. I don't have a problem with providing cross-references to material on algorithm states. I do have a problem providing cross-references to basic preprocessor functionality every time it might possibly be relevant (which is just about everywhere).
Second of all it seems the current facilities in the library should be enhanced (though I don't understand how FOR and FOR_r is supposed to be used): BOOST_PP_SEQ_FOR_EACH is such a natural and easy thing, the use of it should be promoted, while FOR and its derivates look ugly.
Hmm. FOR is far more general. SEQ_FOR_EACH can be implemented with FOR, but not vice versa.
sure --- in other words, FOR_EACH is easier to use!
Sure, as are a whole lot of other things that are just as easy to use as FOR_EACH. IOW, it is nothing special; it is one of many, etc., etc..
It will only break if SEQ_FOR_EACH or LIST_FOR_EACH arbitrarily decides to use the other--which won't happen. I design the algorithms in the library, and I'm well aware of vertical dependencies and their implications.
I was referring to the documentation, which seems not to speak about this issue.
The documentation says that certain constructs are reentrant. It doesn't say for every non-reentrant macro that it is non-rentrant--because that is the status quo, the "normal" situation which is not defined by the library and thus is not the responsibility of the library to document.
According to what I understand from the documentation, the natural solution, offering for example BOOST_PP_SEQ_FOR_EACH_2, which uses macros guaranteed not to occur in BOOST_PP_SEQ_FOR_EACH, has been deprecated in favour of a more complicated solution.
You don't understand the implications of what you're saying.
sure
I'm not saying that to be derogatory, but I am saying that out of frustration because you immediately started pronouncing what should be and what I should do without knowing any of the implications of your pronouncements nor any of the rationale behind the design of the library or what is and isn't in the documentation. You didn't ask *why* something is the way it is. Instead, you ran into problems, and then said that everything should be better (so that you wouldn't have run into problems). I'm also not belittling the problems that you ran into. Preprocessor metaprogramming can be tricky, is very foreign to typical programming models, and has a steep learning curve. Sure, the documentation isn't perfect, but sometimes learning curves are inherently steep.
In order to do that, I'd need to define hundreds of macros to implement SEQ_FOR_EACH alone. It isn't as simple as just replicating the interface macro a few times. Instead, I'd have to reimplement the entire algorithm from scratch--intentionally avoiding the use of any other higher-order algorithm.
the whole thing looks quite complicated; why not adding a FAQ page to the documentation (I was looking for one, but didn't find one) answering for example my above request in this way? I think it would have helped me.
Because I don't want to implement the library in the documentation. These are implementation details and if I introduced those into the documentation, the documentation would be orders of magnitude more complex.
I'm not sure what you're referring to by "more complicated solution". If you specify what kind of thing your referring to, I'll expound.
The documentation says
In particular, the library must provide reentrance for BOOST_PP_FOR, BOOST_PP_REPEAT, and BOOST_PP_WHILE. There are two mechanisms that are used to accomplish this: state parameters (like the above concatenation example) and automatic recursion.
Thus it seems only for these loops you have a special mechanism for reentrancy, and this special, "generic" mechanism" is what I mean with "more complicated" : for the user it's easy to use BOOST_PP_SEQ_FOR_EACH and BOOST_PP_SEQ_FOR_EACH_2, but it's complicated to use the generic method.
Except that you're wrong. You run into massive dependency nightmares if you do that. It isn't simpler--it creates problems that are very difficult for users to track down.
The documentation has many examples, but you call them either complicated or contrived.
Let's go to the example section:
First we have "array_arithmetic.c": It says "This example implements over 2200 functions". Sounds impressive, but is not an example (sounds more like an extension of the library).
No, it's an example. The library is a code generator, not a library of generated code.
Among those six examples only "catch_builtin.cpp" seems to be close to an example (which is something you look at and you understand).
I disagree with your definition of an example. Any code that uses an interface is an example of using that interface. Whether or not it is a good example is another issue. However, the library includes many very simple examples--such as the example for SEQ_FOR_EACH: #define SEQ (w)(x)(y)(z) #define MACRO(r, data, elem) BOOST_PP_CAT(elem, data) BOOST_PP_SEQ_FOR_EACH(MACRO, _, SEQ) // w_ x_ y_ z_ That is a simple example that clearly shows what SEQ_FOR_EACH does without focusing on an overarching (complicated) problem to be solved. The library is littered with such examples.
In the reference section there are actually simple examples, but as far as I can see always only one example: I believe a systematical declination of the elementary cases would be very helpful (always staying simple, just handling simple sequences, but with example 5 to BOOST_PP_SEQ_FOR_EACH for example on would show a nested loop ("without added meaning", that is, no template stuff etc.)).
Why would I show a nested loop in an example for SEQ_FOR_EACH when SEQ_FOR_EACH cannot be called from the repeated macro invoked by SEQ_FOR_EACH?
What exactly to you want? This kind of stuff doesn't boil down to _simple_ use cases.
My understanding is that use cases are not very helpful. What we need is *understanding*.
Yes.
??? What does this want to say to us? What are "graphs" etc. ??? But shouldn't one know all these things!! And it's a very nice fundamental property of graphs, just *using* addition in two ways.
The difference between this mock example and the library's examples is significant. The templates (or anything else) that appear in the examples aren't what the examples are about. They are *output* of the example--not part of the functionality of the example.
sum_{i=1}^n i = n (n+1) /2.
Hope you get what I mean.
I get what you mean, but I think the simple examples are already there. I just don't think you invested much effort. It basically seems that if you saw "template", you just immediately ignored the example.
Rather, typical use cases are parts of overarching designs that are complex--which is precisely the reason that preprocessor metaprogramming enters the picture at all.
As I said, I don't want use cases, I just want to use the preprocessing library (for my own purposes).
Okay, but your example of using the library to generate an enum is a use-case. Sure, it's a simple one, but it is only simple because it is naive to the point of being destructive.
Furthermore, I have a responsibility (as does any library author) not to promote naive or poor uses of the library--which is almost always the case with examples that implement something simple.
So the documentation is the guard, meant to be as a kind of torture, and only those worthy souls which get through shall use the library?
This actually makes me smile. No, the documentation is not meant to be that, but the learning curve is inherently steep, and such learning requires the learner to expend a certain amount of effort--moreso than when learning something simpler. What I said was true. I have a responsibility not to promote naive or poor uses of the library. The user that is learning the library has not yet attained the degree of aptitude required to separate the way something is achieved (the point of the example) from what is being achieved (the medium of the example).
Speaking for myself, I never ever found any interest in those "real world scenario's" (for this library and others), since their "real world" is not mine. Of course, other might disagree, and having a healthy mix of simple and more complicated examples sounds like a good thing.
It is a good thing. What you seem to be suggesting, as far as examples are concerned, is somewhere in the middle. Something that I would call a trivial use case. But for the reasons that I already mentioned, such examples can be dangerous. Thus, I tried to either make them full-fledged and concrete or utterly simple and contrived. (There were a few times that I got bored of writing ultra-simple examples, so I made them a little more elaborate, but those are almost always in the interfaces targeted towards more advanced users.)
The library 1) shouldn't have to announce this problem
I meant the documentation
As did I.
and 2) already does announce this problem--it just doesn't do it hundreds of times in hundreds of different places. If you are writing code in the language defined by the preprocessor, you should *at least* know the basics of that language. The library is not the language, it is merely a systematic set of reusable constructs written in that language. The documentation's only responsibility is to document itself.
yes, but neither you are an angel (so that you could give a precise definition of the whole thing) nor am I (so that I could read it); instead we are human beings nd mst fnd r wy thrgh ntcptn (there are nicer examples; but hopefully this works a bit: just looking at it I hope you thought huh, but knowing that I left out the consonants it shouldn't be too hard; I have seen ^^^^^^^^^^ vowels?
a very nice series of examples in this style, where you were guided to read more and more strange stuff, and at the end you were presented with a plain English sentence --- and you couldn't read it anymore!).
You're right that documentation has to be more than just a formal definition. However, the point remains that it isn't the job of the documentation of a library written in language X to document language X.
Regarding examples in the documentation, I don't think there is a way to please you
perhaps there are ways: I believe actually in small steps, massaging something reasonable here and there so that it gets better. If some of the above thoughts would lead to some additions here and there (a few remarks, hints) in the documentation, that would mean progress.
Out of curiousity--do you believe there are certain things that have inherently steep learning curves? What you're suggesting here is basically that I flatten the learning curve by gradually introducing complexity. My argument is that there are times that complexity cannot be avoided or compartmentalized into small, easily understood, chunks. There are certain steps along the road that are transactional--all or nothing.
Regarding the technical implications:
I can understand the general point of view. But this general point of view might not necessarily be the point of a user (who in general will just occasionally use the library).
Perhaps it might be a good idea to add the above explanation somewhere to the documentation?
IOW, add implementation details to the documentation? As far as the reentry stuff is concerned, all that is necessary for users is to know is that macros cannot be used inside themselves (i.e. common knowledge). The places where that rule is broken (meaning that the library makes it look like it is breaking the rule) are explicitly noted. Users need not ever deal with state parameters--they only need to know that macros will not expand inside themselves. Thus, the macro passed to (e.g.) SEQ_FOR_EACH, that is invoked by SEQ_FOR_EACH, cannot use SEQ_FOR_EACH. Otherwise, the library makes it work automatically. You aren't required to use SEQ_FOR_EACH_R--or any interface that requires you to mess around with an algorithm state. Instead, you can ignore it completely.
So if I understand it correctly, you currently have three independent looping constructs BOOST_PP_FOR, BOOST_PP_REPEAT, BOOST_PP_WHILE, while all other loops are implemented in terms of these?
There are actually more than that, but the library makes it appear that there are only a few algorithmic states (e.g. 'r' for FOR, 'z' for REPEAT, and 'd' for WHILE), and therefore that there are only a few algorithms implemented this way. It used to be the case that, because SEQ_FOR_EACH was implemented in terms of FOR, you couldn't use SEQ_FOR_EACH inside FOR. You *had* to use SEQ_FOR_EACH_R. I removed that kind of limitation. You cannot use SEQ_FOR_EACH inside itself, but you need not deal with the algorithm states (like 'R') at all if you don't want to. This makes the library a great deal easier to use.
Then it would make sense to concentrate on the documentation of these three macros, giving more examples (showing also how to simulate nesting), and emphasising for e.g. FOR_EACH its special character.
It isn't FOR_EACH that is special, it is the ones that are explicitly designed to be reentrant that are special.
As far as I can see from the material provided on more powerful pp-libraries, the Boost pp-library won't be further developed, but at some time replaced?
Not necessarily. However, it is near what I can do with it either because of portability issues or efficiency concerns with various preprocessors. I have tried adding various things to it, but it is extremely difficult to keep the library stable on some very popular compilers. Things are improving. As far as broken preprocessors are concerned, there are only a few remaining roadblocks--the most significant being VC++. As far as efficiency is concerned, the major roadblock has always been EDG-based preprocessors (notoriously slow). I haven't seen proof of this, but *supposedly* the EDG preprocessor has been modified to use memory much more efficiently during macro expansion, and the previous way that it was using memory was *supposedly* the source of the major slowdown. (This is in an EDG version that is not yet available in any other compiler.) The speed issue is not as big an issue as the utter broken-ness of VC++ in particular. When and if VC++ and the other two broken preprocessors, Sun and IBM, fix their preprocessors, the library won't be replaced so much as heavily refactored. Some other refactoring is due anyway in preparation for variadics. As an immediate example, SEQ_FOR_EACH's interface needs to change from: SEQ_FOR_EACH(macro, data, seq) to: SEQ_FOR_EACH(macro, seq, data) in preparation for it becoming: SEQ_FOR_EACH(macro, seq, ...) Of course, when and if that happens--which depends on the vendors--a lot of the limitations that you've run into will be gone. SEQ_FOR_EACH would be just as reentrant as anything else. Regards, Paul Mensonides