Paul Mensonides wrote:
I'm trying very hard not be harsh, but what you suggest has far-reaching implications that you don't understand.
There should be some sort of disclaimer in the documentation of Boost.PP that mentions that the design is most unfortunately influenced by the bad quality of some widely used preprocessors. Otherwise there will be false conclusions that pp-metaprogramming is either ugly by definition or that there is something wrong with the design of the library... Here are my comments on the text. Note that I use imperative for the sake of simplicity.
The process of macro expansion is best viewed as the process of =canning for macro expansion (rather than the process of a single macro =xpansion alone). When the preprocessor encounters a sequence of preprocessing tokens =nd whitespace separations that needs to be scanned for macro expansion, it has to perform a number of steps. These steps are examined in this document in detail.
Strike that paragraph. It uses terms not yet defined and doesn't say much more than the title (assuming it's still "how macro expansion works").
Conventions
Several conventions are used in this document for the sake of =implicity. First, except in contexts where the distinction matters, this =ocument uses the terms "token" and "preprocessing token" =nterchangeably. Where the distinction does matter, this document will explicitly =ifferentiate between them by using the terms "regular token" and =preprocessing token." From a technical standpoint, the primary difference between the two =oken types relates to permissiveness. Preprocessing tokens are much more permissive than regular tokens. Ignoring classifications (e.g. keyword,
How can tokens be permissive? Their definiton is probably more permissive (if you can say that).
identifier), all regular =okens are trivially converted to preprocessing tokens. Not all preprocessing tokens can be converted to regular tokens. For example, 0.0.0.0 is a single, valid preprocessing =oken, but it cannot be converted to a regular token. Thus, preprocessing tokens can be thought of as a superset of regular =okens.
Second, when this document uses the phrase "sequence of tokens" (i.e. "sequence of preprocessing tokens"), the possibility of whitespace separations is implied. In other words, "sequence of tokens" really means "sequence of tokens =nd whitespace separations." Both the C and C++ standards do this as well, though they don't =xplicitly say they are doing it. Rather, it is implied by the phases of translation and by certain parts of the text such as §16.3.2/2 of the C++ standard and §6.10.3.2/2 of the C standard.
Third, this document uses the term "scanning" to mean "scanning for =acro expansion." There are other kinds of scanning (in the general sense) that are =performed during the macro expansion process. For example, a replacement list is scanned for instances of formal =arameters to substitute. Other kinds of scanning besides scanning for macro expansion are =eferred to explicitly.
Lastly, this document uses a graphical notation to illustrate various parts of the process in examples. For clarity, all whitespace separations are ignored in these =iagrams. Given a sequence of tokens, the current location of the scanner is =ndicated with a circumflex (^). For example,
+ + + ^
A range of tokens is indicated by using a frame such as:
+ + + |^ | |_____| | LABEL
These frames are labeled with the meaning of the frame or the =equence of tokens bounded by the frame. An identifier token that has been painted is
The reader probably has no idea what "painted" means at this point. Indicate the forward-declaration by "see below" or something like that.
marked with an =postrophe (').
A' token ^
This document will discuss what it means for an identifier token to =e "painted." It also doesn't use character literals ('a') to avoid ambiguity. It is important to note that the apostrophe is not literally added to =he sequence of tokens. Rather, it is only a notation that indicates a property of the =dentifier to which it is "attached." _________________________________________________________________
Locations
There are several points where the preprocessor must scan a sequence =f tokens looking for macro invocations to expand. The most obvious of these is between preprocessing directives =subject to conditional compilation). For example,
I had to read this sentence multiple times for me to make sense... VVVVV
#include "file.h"
// ...
#define A()
// ...
#undef A
Both of the sections containing comments must be scanned for macro expansion.
The preprocessor must also scan a few other sequences of tokens. These include the expression that controls an #if or =elif directive. Subject to certain conditions, the operands of the =include directive and the #line directive are =lso scanned for macro expansion.
The places where a sequence of tokens is scanned for macro expansion =n a source file are not really the subject of this document. As such, the remainder of this document concentrates on just =equences of tokens that must be scanned, though sometimes the =define directive is used.
^^^^^ simplify this section by using negative logic (in other words: enumerate the contexts where /no/ macro expansion is done).
Before continuing, however, note that an argument to a macro that =ontains what could be a preprocessing directive results in undefined =ehavior. For example,
#define MACRO(x) x
MACRO( #include "file.h" )
Indicate more clearly that this code is not OK.
This is undefined behavior because it gives a preprocessor =mplementation latitude. Specifically, it allows for a preprocessor to parse an entire file =nto directives and sequences of tokens between directives prior to semantic evaluation. It also allows for a preprocessor to semantically evaluate the result =f preprocessing in a single pass.
Put the implementation details in parentheses or use a footnote.
_________________________________________________________________
Scanning
Consider the following macro definitions and the subsequent sequence =f tokens to be scanned:
#define OBJECT OBJ ## ECT F() #define F() G(G(G)) #define G(x) x
+ X OBJECT +
These definitions and the subsequent sequence of tokens will be used =s a running example in describing the steps of the expansion process.
Given a sequence of tokens to be scanned, the preprocessor =uccessively looks at each token looking for identifiers. If a given token is not an identifier, the preprocessor outputs the =oken and moves on to the next.
+ X OBJECT + ^
+ X OBJECT + ^
VVVVV
The term "output" is being used generally. Sometimes it means that the preprocessor is finished with the token =nd that the token can be handed off to the underlying language parser =r output to a text stream (in the case of a compiler's preprocess-only =ode). Other times, it means that it is output to a buffer that is later =eused by the preprocessor for another purpose. In all cases, however, it does mean that this scan for macro expansion is finished with the token and that the token results from = this scan. ^^^^^ shouldn't this be part of the "Conventions" section?
If the token is an identifier, the preprocessor must check =o see if the identifier corresponds to the name of a macro that is =urrently defined. If it is not, the preprocessor outputs the token and moves on to the =ext. In the running example, the current token is the identifier =, which does not correspond to a macro name.
+ X OBJECT + ^
+ X OBJECT + ^ _________________________________________________________________
"Blue Paint"
If the current token is an identifier that refers to a macro, the preprocessor must check to see if the token is painted. If it is painted, it outputs the token and moves on to the next.
When an identifier token is painted, it means that the preprocessor =ill not attempt to expand it as a macro (which is why it outputs it and =oves on). In other words, the token itself is flagged as disabled, and it behaves like an identifier that does not corresponds to a macro. This disabled flag is commonly referred to as "blue paint," and if =he disabled flag is set on a particular token, that token is called =painted." (The means by which an identifier token can become painted is =escribed below.)
Remove redundancy in the two paragraphs above. I like the "behaves like an identifier that does not correspond to a macro name"-part.
In the running example, the current token is the identifier =BJECT, which does correspond to a macro name. It is not painted, however, so the preprocessor moves on to the next =tep. _________________________________________________________________
Disabling Contexts
If the current token is an identifier token that corresponds to a =acro name, and the token is not painted, the preprocessor must =heck to see if a disabling context that corresponds to the macro =eferred to by the identifier is active. If a corresponding disabling context is active, the preprocessor =aints the identifier token, outputs it, and moves on to the next token.
A "disabling context" corresponds to a specific macro and exists over = range of tokens during a single scan. If an identifier that refers to a macro is found inside a disabling =ontext that corresponds to the same macro, it is painted.
Disabling contexts apply to macros themselves over a given geographic sequence of tokens, while blue paint applies to particular identifier tokens. The former causes the latter, and the latter is what prevents "recursion" in macro expansion. (The means by which a disabling cotnext comes into existence is =iscussed below.)
In the running example, the current token is still the identifier =BJECT. It is not painted, and there is no active disabling context that =ould cause it to be painted. Therefore, the preprocessor moves on to the next step.
The introductions of these terms feels structurally too aprupt to me. Introduce these terms along the way, continuing with the example. ______________________________________________________________
Object-like Macro Invocations
If an identifier token that corresponds to a macro name is not =ainted, and there is not active disabling context that would cause it =o be painted, the preprocessor must check to see what type of macro is =eing referenced--object-like or function-like. If the macro is defined as an object-like macro, which is the case =BJECT in the running example, the identifier alone forms =n invocation of the macro, and the preprocessor begins the replacement =rocess of that invocation.
+ X OBJECT + |^ | |______| | OBJECT invocation (INV)
For an object-like macro, the first step in the replacement process =s the retrieval of the replacement list of the macro from the symbol =able. In the running example, the OBJECT macro's replacement =ist is
OBJ ## ECT F()
The preprocessor then performs any token-pasting operations in the replacement list. (Note that there is no stringizing operator in object-like macros.) The result of token-pasting in OBJECT's replacement list =s
OBJECT F()
After all token-pasting has been performed, the resulting sequence of tokens is used to replace the invocation of the macro. At the same time, a disabling context that corresponds to the macro =eing replaced is activated. This disabling context surrounds the tokens that came from the replacement list.
+ X OBJECT F() + | | |__________| | OBJECT disabling context (DC)
<-- explain what a disabling context and then what blue paint is is here
Finally, scanning resumes beginning with the first token that came =rom the replacement list (or the next token after the invocation if the replacement list was empty). In the running example, scanning resumes at OBJECT.
+ X OBJECT F() + |^ | |__________| | OBJECT DC
Note that this OBJECT identifier is a different =dentifier token than the one that formed an invocation of the =BJECT macro. It came from the replacement list of the OBJECT macro =indirectly by way of token-pasting).
<snip>
Nullary Invocations
If scanning finds that an identifier (followed by a left-parenthesis) =hat refers to a nullary function-like macro (as is the case with = in the running example) the preprocessor must find the =orresponding right-parenthesis. The preprocessor must find the right-parenthesis. If it cannot (e.g. it finds EOF instead), the result is an =incomplete invocation" error.
While it does so, it must ensure that there are no tokens between the left-parenthesis and the right-parenthesis. Specifically, between the two parentheses, there must be either one =r more whitespace separations or nothing at all. If any tokens are present between them, the result is a "too many =rguments" error.
Try to remove some detail.
The identifier and the left- and right-parentheses form an invocation =f the macro, and the preprocessor begins the replacement process of =hat invocation.
+ X OBJECT' F() + | |^ || | |___|| | | | | F INV |____________| | OBJECT DC
As with an object-like macro, the first step in the replacement =rocess is the retrieval of the replacement list of the macro from the =ymbol table. In the running example, the F macro's replacement list =s
G(G(G))
The preprocessor then performs any token-pasting operations in the replacement list. (Note that a nullary macro is a function-like macro, so the =tringizing operator exists. However, the stringizing operator must be applied to an instance of a =ormal parameter in the replacement list. A
<-- add missing "nullary"
function-like macro has no formal parameters, and therefore any use =f the stringizing operator is automatically an error.) The result of token-pasting
It's not clear to me why the stringizing operator leads to an error rather than a '#' character. Probably too much of a sidenote, anyway.
in F's replacement list is
G(G(G))
In this case, there are no token-pasting operations to perform, so =he result is a no-op.
The sequence of tokens resulting from token-pasting is used to =eplace the invocation of the macro. At the same time, a disabling context that corresponds to the macro =eing replaced is activated. This disabling context surrounds the tokens that came from the =eplacement list.
+ X OBJECT' G(G(G)) + | | || | |_______|| | | | | F DC | |________________| | OBJECT DC
Finally, scanning resumes beginning with the first token that came =rom the replacement list (or the next token after the invocation if the replacement list was empty).
+ X OBJECT' G(G(G)) + | |^ || | |_______|| | | | | F DC | |________________| | OBJECT DC
Nullary function-like macro invocations are nearly identical to object-like macro invocations. The primary difference is that an invocation of a function-like macro =equires multiple tokens. _________________________________________________________________
Interleaved Invocations
It is important to note that disabling contexts only exist during a =ingle scan. Moreover, when scanning passes the end of a disabling context, that disabling context no longer exists. In other words, the output of a scan results only in tokens and =hitespace separations. Some of those tokens might be painted (and they remain painted), but =isabling contexts are not part of the result of scanning. (If they were, there would be no need for blue paint.)
Misses (at least) a reference to 16.3.4-1 (the wording "with the remaining tokens of the source" (or so) is quite nice there, so consider using something similar). I believe I wouldn't really understand what you are talking about here without knowing that part of the standard. "A single scan" -- the concept of rescanning was introduced too periphicially to make much sense to someone unfamiliar with the topic.
In the diagrams used in this document, the tokens that have been =utput by a scan are left in the diagram to provide context for the =eader, such as:
<snip>
Note that interleaved invocations do not allow for infinite =xpansion. More tokens must physically be present after the replacement list to complete an interleaved invocation, and this sequence of tokens is ultimately limited to the finite sequence of tokens contained in the source file. _________________________________________________________________
The above part seems very good to me.
Non-nullary Invocations
If scanning finds an identifier (followed by a left-parenthesis) that refers to a non-nullary function-like macro (as is the case with = G in the running example) the preprocessor must find the corresponding right-parenthesis. The preprocessor must find the right-parenthesis. If it cannot (e.g. it finds EOF instead), the result is an =incomplete invocation" error.
While it does so, it must delineate the actual arguments between the =eft- and right-parentheses. This delineation process is a distinct step that separates each =argument into a separate argument before any other processing of the =rguments is done. Each argument is separated by a comma token (,), but =ommas between matched pairs of left- and right-parentheses do not =eparate arguments, and the right-parenthesis of such matched pairs is =ot used as the right-parenthesis that terminates the argument list. For example, in
MACRO(a, (b, c), d)
the argument list to MACRO is delineated into three =rguments.
After the arguments have been delineated, the preprocessor compares =he number of actual arguments to the arity of the macro. If there are more or less, than the result is a "too many arguments" =r "too few arguments" error. (If the macro is variadic, the number of arguments must be at least one greater than the number of named formal parameters. This implies that a macro defined as:
#define V(x, y, ...) // ...
Mention that variadic macros are not a C++ feature.
has a minimum arity of three--not two.)
In C++, if any argument is empty or contains only whitespace =eparations, the behavior is undefined. In C, an empty argument is allowed, but gets special treatment. (That special treatment is described below.)
It requires at least C99, right? If so, say it (it's likely there are C compilers that don't support that version of the language). <snip>
+ X OBJECT' G(G(G)) + | ||^ ||| | ||_______||| | | | || | | G INV | | |_________|| | | | | F DC | |__________________| | OBJECT DC
// recursive scan of argument #1 to G:
G(G) |^ | |____| | G INV _ _ ______ _ _ | F DC _ _ ______ _ _ | OBJECT DC
// recursive scan of argument #1 to G:
G ^ _ _ _ _ _ | F DC _ _ _ _ _ | OBJECT DC
G ^ _ _ _ _ _ | F DC _ _ _ _ _ | OBJECT DC
// recursive scan results in: G
G |^| |_| | G DC _ _ ___ _ _ | F DC _ _ ___ _ _ | OBJECT DC
G' ^ _ _ ___ _ _ | F DC _ _ ___ _ _ | OBJECT DC
// recursive scan results in: G'
+ X OBJECT' G' + | ||^ ||| | ||__||| | | | || | | G DC | |____|| | | | | F DC |_____________| | OBJECT DC
+ X OBJECT' G' + ^
+ X OBJECT' G' + ^
Thus, the running example finally results in
+ X OBJECT' G' +
Note that the disabling context that is created when a macro =nvocation is replaced is not created until after all arguments =hat need to scanned for macro expansion have been. This is why both G's expand in a the example.
You should repeat the "painting rules" here. You already said that paint applies to single tokens but it's a good idea to more explicitly point out to the reader that no parentheses are needed for G to end up painted.
_________________________________________________________________
Arguments to Interleaved Invocations
Because arguments not used as operands of token-pasting or =tringizing are scanned separately but within any active disabling =ontexts, the possibility exists for an argument (or part of an =rgument) to be scanned in inside a disabling context that no longer =xists after the macro
An "in" too much?
invocation has been replaced. Consider,
#define A(m) m( B(f) #define B(x) A(x)
#define C(x) [ x ]
A(C) ) *
The following diagrams the expansion of this sequence of tokens:
A(C) ) * |^ | |____| | A INV
// recursive scan of argument #1 to A results in:
C( B(f) ) * ||^ | | ||_______|_| | | | | C INV |________| | A DC
// recursive scan of argument #1 to C:
B(f) |^ || |____|| | | B INV _ _ ______| | A DC
// recursive scan of argument #1 to B results in: f
A(f) |^ || |____|| | | B DC _ _ ______| | A DC
A'(f) | ^ || |_____|| | | B DC _ _ _______| | A DC
// recursive scan results in: A'(f)
[ A'(f) ] * |^ | |_________| | C DC
The result of scanning for expansion is
[ A'(f) ] *
Note that the argument to C was scanned inside the =isabling context corresponding to A, but that disabling =ontext is no longer active when scanning resumes after the replacement =f C's invocation. (Scenarios such as this one are tricky, and preprocessors rarely =andle this kind of thing correctly.)
Point out explicitly that A remains painted.
_________________________________________________________________
Virtual Tokens
[...]
Interesting! Is it implemented somewhere? BTW. I'm done with the comments, so far. I believe there is more that can be done to enhance the structure. Anyway, all in all it reads quite nice. I hope it's of any use.
© Copyright [1]Paul Mensonides = 2003-2006
Regards, Tobias