-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Oliver Kullmann
First of all I think here the documentation of the library doesn't reach its full potential yet: I tried first to read through "Topics", but found all examples contrived and complicated (couldn't find a single plain example),
What is a "plain example"?
and thus I skipped that and went to Reference, where I found BOOST_PP_SEQ_FOR_EACH, making some still mysterious remarks about "The next available BOOST_PP_FOR repetition." etc. But at various places it somehow says that "reentry problems" whatever this means are solved now.
There is an article in the topics section about this (titled "Reentrancy").
Only when looking it up in the rationale of C99 I found a clear statement about the issue (while the C++ standard somehow supposes you know already the issue).
It is pretty clear in both standards (6.10.3.4/2 in C and 16.3.4/2 in C++).
So a clear statement about the limitations of the looping constructions *at each place* would help I would guess quite a few people trying to use this library.
It is unnecessary clutter. It is not the documentation's responsibility to constantly tell you the way that the language works. Unless the documentation says otherwise, it should be assumed that a macro cannot expand inside itself. That is simply the way that the language works. The library makes it *appear* that a few macros are recursive, but then it explicitly says that the macro can be used as if it was recursive.
(I believe documentation must be written so that it can be entered in many ways, and not assuming you are reading it from the start to the end --- redundancy is needed.)
I disagree on all counts. It is impractical to make every page of the documentation a complete discourse on both how the preprocessor works and how preprocessor metaprogramming works. Beyond that, redundancy in both code and documentation is a maintenance nightmare.
Second of all it seems the current facilities in the library should be enhanced (though I don't understand how FOR and FOR_r is supposed to be used): BOOST_PP_SEQ_FOR_EACH is such a natural and easy thing, the use of it should be promoted, while FOR and its derivates look ugly.
Hmm. FOR is far more general. SEQ_FOR_EACH can be implemented with FOR, but not vice versa.
But it seems that the library favours FOR, and seems to envisage a main loop using FOR, and only at the second level using FOR_EACH.
The library doesn't favor it, it simply is more general purpose. In order to allow the kind of reentrance that FOR allows, hundreds of macros are required for *each* higher-order algorithm. SEQ_FOR_EACH is only one of many.
My solution was to convert the second sequence B into a list, and use BOOST_PP_LIST_FOR_EACH for the inner loop. But it seems to me that this works is pure chance, and with any change in the implementation of BOOST_PP_SEQ_FOR_EACH or BOOST_PP_LIST_FOR_EACH (introducing a common macro) this solution will break.
It will only break if SEQ_FOR_EACH or LIST_FOR_EACH arbitrarily decides to use the other--which won't happen. I design the algorithms in the library, and I'm well aware of vertical dependencies and their implications.
According to what I understand from the documentation, the natural solution, offering for example BOOST_PP_SEQ_FOR_EACH_2, which uses macros guaranteed not to occur in BOOST_PP_SEQ_FOR_EACH, has been deprecated in favour of a more complicated solution.
You don't understand the implications of what you're saying. In order to do that, I'd need to define hundreds of macros to implement SEQ_FOR_EACH alone. It isn't as simple as just replicating the interface macro a few times. Instead, I'd have to reimplement the entire algorithm from scratch--intentionally avoiding the use of any other higher-order algorithm.
But it seems this more complicated solution works only for one kind of loop, and not for others, and the documentation
I'm not sure what you're referring to by "more complicated solution". If you specify what kind of thing your referring to, I'll expound.
seems so arcane that I don't have a clue how it could work (the typical phenomenon, that the documentation wants to show off by presenting only complicated examples, which show all sorts of templates, etc., and this without saying what actually should by achieved --- good code solves one problem at a time, and good documentation discusses one problem at a time).
The documentation has many examples, but you call them either complicated or contrived. What exactly to you want? This kind of stuff doesn't boil down to _simple_ use cases. Rather, typical use cases are parts of overarching designs that are complex--which is precisely the reason that preprocessor metaprogramming enters the picture at all. Furthermore, I have a responsibility (as does any library author) not to promote naive or poor uses of the library--which is almost always the case with examples that implement something simple. So, I can do two things--I can provide simple contrived examples that demonstrate what various things do, and I can provide complex examples that implement real world scenarios. The documentation does both already.
So if the library offers a natural way of nesting BOOST_PP_SEQ_FOR_EACH, then documenting this (directly, without solving other problems) would be good, or otherwise adding BOOST_PP_SEQ_FOR_EACH_2 would help some users (actually, just the presence of this second form would announce the underlying problem).
The library 1) shouldn't have to announce this problem and 2) already does announce this problem--it just doesn't do it hundreds of times in hundres of different places. If you are writing code in the language defined by the preprocessor, you should *at least* know the basics of that language. The library is not the language, it is merely a systematic set of reusable contructs written in that language. The documentation's only responsibility is to document itself.
Sometimes less general but simpler solutions are preferable over more general but complicated solutions.
Again, I'm not sure what you mean by "more complicated solutions". ----- I'm trying very hard not be harsh, but what you suggest has far-reaching implications that you don't understand. Regarding examples in the documentation, I don't think there is a way to please you--see posts by me in this thread: http://thread.gmane.org/gmane.comp.lib.boost.user/16132 ----- Regarding the technical implications: (Please make the effort to follow this.) As it stands now, the library emulates recursion by replicating macros. For example, a WHILE loop can be implemented such as: #define WHILE_1(pred, op, state) \ IF(pred(2, state), WHILE_2, state TUPLE_EAT(3))(pred, op, op(2, state)) \ /**/ #define WHILE_2(pred, op, state) \ IF(pred(3, state), WHILE_3, state TUPLE_EAT(3))(pred, op, op(3, state)) \ /**/ // etc. (i.e. hundreds...) A predicate is passed that determines when the algorithm should stop, and an operation is used to iterate the state on each step of the algorithm. There are a couple of things here must be understood. First, each WHILE_xyz represents both an entry point into the algorithm (i.e. an interface to the algorithm) *and* a single algorithmic step taken by the algorithm. So, WHILE_2 does not mean "separate nested WHILE loop" or "second looping dimension". Instead, it means "the step after WHILE_1". Now, when the algorithm (meaning the entire collection of macros) invokes the predicate and the operation, it passes along the number of the next available WHILE_xyz macro. This value is called the "state" of the algorithm. (Note that this is not the 'state' parameter. That is the (arbitrary) state the a particular *use* of the algorithm is iterating.) The predicate and the operation can use this algorithm state to reenter the algorithm, but before the algorithm itself actually continues into that macro. As the algorithm continues, the algorithm state values passed to the predicate and the operation change (i.e. they increase). This is important. They do not remain fixed, which means that the predicate, for example, cannot just arbitrarily use WHILE_2. It has to use the WHILE_xyz indicated by the algorithm state value passed to it. Obviously it is possible to provide several totally distinct implementations of the entire loop, but that implies several hundred macros per loop, instead of sharing a single set of several hundred macros for all loop uses. Say you did this anyway, and you get WHILE_A, WHILE_B, WHILE_C (and so on, each with their own WHILE_A_1, WHILE_A_2, etc.). This doesn't really buy you anything, because you'll still ultimately end up passing A, B, C (etc.) as a reentry point. E.g. suppose you write some macro that uses WHILE_A, and then you change another macro so that it uses this macro, but this other macro was already using WHILE_A. Fine, you just change this other macro to use WHILE_B. But then, there is another macro that uses this second macro, and this third macro is already using WHILE_B, so you have to change that to use WHILE_C. This is a significant bubbling of implementation detail, and it is *way* worse when all three macros are written by different people. Ultimately, you arrive at the conclusion that it doesn't matter *which* WHILE the first macro uses, so you parametize the first macro with the WHILE to be used, Similarly with the second (and when the second calls the first, it uses the next WHILE from the one it is currently using). Similarly again with the third. So, instead of having the most nested use of WHILE be WHILE_A, you have the outermost use be WHILE_A, and each inner use adjusts, so that the first nested use uses WHILE_B, the next nested use uses WHILE_C, and so on. This is the only way to do it that scales, and this is ultimately no different than just having one WHILE with one set of implementation macros. This is exactly (minus a lot of workarounds) the way that the library emulates recursion (particularly: in FOR). The problem comes when you write a higher-order algorithm on top of another higher-order algorithm. For example, SEQ_FOR_EACH on top of FOR. (Higher-order algorithms are fundamentally different because the implementation of the algorithm cannot completely control what macros are called inside of them.) It doesn't take hundreds of macros to implement SEQ_FOR_EACH because it can reuse the more general purpose algorithmic steps provided by FOR. In order to make SEQ_FOR_EACH reentrant, you need, at minimum, a distinct interface macro (i.e. SEQ_FOR_EACH_1, SEQ_FOR_EACH_2, etc.) and distinct macro to be repeated by FOR for each possible entry point into SEQ_FOR_EACH. That isn't too bad in terms of number of macros, but you introduce yet another algorithm state by doing so. If each higher-order algorithm did so, the user would be inundated by the states of various algorithms. E.g. just to invoke SEQ_FOR_EACH, you'd need to supply both the state of SEQ_FOR_EACH and the state of FOR. Worse, if another algorithm (with multiple entry points) uses SEQ_FOR_EACH, you'd need the state of that algorithm, the state of SEQ_FOR_EACH, and the state of FOR to use it. You resolve this in one of two ways: 1) you simply don't provide reentrance in higher-order algorithms that are implemented using other higher-order algorithms, or 2) you reimplement the algorithm from scratch to avoid using other higher-order algorithms. The former is what the library does, the latter requires hundreds of macros to implement each higher-order algorithm. ----- All of this is the status quo of the library. There are two other models that are superior in terms of ease of use and avoidance of these types of issues. The first of these, I use in a different (far more powerful) preprocessor library. The problem with this model is that it isn't portable to broken preprocessors (which are heavily used by users of Boost--e.g. VC++). The second alternative model revolves around automatically deducing all algorithm states whenever they are needed. IOW, completely hiding them from users altogether. The problem with this model is that it isn't portable because of efficiency concerns. Some preprocessors would handle it without any significant problem. Others would slow to a crawl--e.g. EDG-based preprocessors. Of these two, the first is much more extensible. It allows for the definition of new algorithms without *any* macro replication. The second would still be better (as far as ease-of-use is concerned) than what the library provides now, but there is no way that I could get away with it. The second can be improved beyond what I've described here (e.g. by separating the notions of "entry-point" and "algorithmic step"--200+ loop iterations may be necessary, 200+ nested loops is massive overkill), but it all boils down to the same thing--always deducing the state when needed. Unfortunately, I cannot do either of these in Boost because Boost cannot simply drop support for a compiler just because it has a broken or slow preprocessor (of all things).
P.S. The situation with the preprocessor is complicated by the general silence on this issue in the books. And quite amazing, how the library achieves to squeeze out nearly a general purpose machine (but not with an infinite tape :-)) from the preprocessor machinery, which seems to have been castrated by the designers on purpose.
Incidentally (and this isn't exactly relevant to this conversion), the model used by my other library is capable of using a set of macros exponentially, such that (e.g.) the "end of the tape" is trillions of iterations away--effectively removing it as a limitation. Given unlimited system resources (like memory), any algorithm that needed that many steps would literally take centuries to execute even on the fastest preprocessor. IOW, it removes the limitation for all intents and purposes. Also, attached is a draft of a document that I'm writing (for both libraries) that describes the macro expansion process in detail. You might find it useful. If you or others read this, feedback is welcome. I'm _not_ interested in feedback related to formatting or layout. I want to avoid an argument about an issue that 1) nobody will ever agree on and 2) I don't think is as important as it is sometimes made out to be. I _am_ interested in feedback relating to structure and content. Regards, Paul Mensonides