42 // operation to at least update the L1 cache. *** Note: This 43 // concern is specific to the particular application at which 44 // we're targeting the test. ***
that seems quite important but a little opaque out of context.
It means the application we were going to use this technique on was going to update a long series of accumulators, which will certainly not fit in a few registers. We wanted to force the test to reflect that reality.
Ah, I did some hunting and found things for the iterative calculation of the mean - presumably this is an embodiment of the original application. How various caches interact is not something I've given a great deal of thought to. In practice it can presumably make or break the real 'speed' of something. I can see that there's a great difference in timing a large bunch of iterations of the same thing one after another and the actual typical execution characteristics of the same function in a different context. This kind of thing is something I have to shrug my shoulders at - I can see it's a potential issue but don't know enough to comment.
One thing I take exception to is the (effective) use of the mean as a measurement of central tendency
Why?
The mean is not a robust measure of central tendency. Robust, in this context, means that a single arbitrarily large observation can shift the estimator (the mean in this case) arbitrarily far. The median on the other hand is robust, a single arbitrarily large observation may not move the median at all. Technically the 'breakdown point' of the mean is zero. For the obligatory Wikipedia overview see: http://tinyurl.com/l4vldr
- perhaps their trickery has eliminated the heavy tail.
Why would one assume there is a heavy tail in the first place?
Informally speaking one may expect the OS to very-very occasionally interrupt a running function that nominally takes (say) 100 cycles and run of to do some disk IO. That delay may will run into hundreds of millions of clock cycles. Consequently that observation may differ from all the others by (say) six orders of magnitude. Most probably of course that will not occur; that's a long tail - very rare events of extreme magnitude. A bit like returns in stock market crashes only less frequent. ;-) Less informally I've measured it! On a log-log histogram plot of ~10^7 measurements even trivial functions display a definite hint of Pareto-like [ http://tinyurl.com/ngnyk2 ] behaviour in the wings. -ed ------------------------------------------------ "No more boom and bust." -- Dr. J. G. Brown, 1997