Boost Test Library: general floating point comparisons
Hi folks, This is probably a question aimed at Gennadiy Rozental, but it may be of interest to the community in general. We've been using the boost test library for about a year and a half now, and are happy customers. I'm revamping our unit test framework (e.g. revisiting the way we use boost test), and am specifically looking at the floating point comparisons. One thing that I'm changing / hacking is the use of percentages to specify tolerance. We usually want answers to agree to (say) 1 part in 10^-12, and remembering to multiply or divide that by 100 is annoying. To this end, I'm defining our own set of comparison functions and macros, still using the stuff in ttdetail (e.g. fpt_abs and save_fpt_division). Fine. Now, we want to be able to compare more complicated mathematical objects, such as vectors and matrices. After some thought, we decided the responsibility for providing such functions is the responsibility of the mathematical object, not the unit test framework. That is to say, a function "relative difference between matrices" should be in Matrix.h, not some unit test thing. Which leads to the idea of Matrix.h #including "boost/test/floating_point_comparison.hpp", which strikes me as rather odd. My first impulse was to copy/pull the relevent code out of floating_point_comparison.hpp and stick it in some more convenient place, where mathematical objects could use those functions to implement their own comparision functions. Might this be a common idiom? Might we consider factoring these floating point comparison functions (not the BOOST_CHECK_CLOSE macro, but the functions it uses) out into a separate library? Any comments on my approach? Any and all thoughts, comments, or complaints welcome. Thanks! ---------------------------------------------------------------------- Dave Steffen, Ph.D. Disobey this command! Software Engineer IV - Douglas Hofstadter Numerica Corporation dg@steffen a@t numerica d@ot us (remove @'s to email me)
"Dave Steffen" <dgsteffen@numerica.us> wrote in message news:17874.12117.50555.431498@yttrium.numerica.us...
Hi folks,
This is probably a question aimed at Gennadiy Rozental, but it may be of interest to the community in general.
We've been using the boost test library for about a year and a half now, and are happy customers. I'm revamping our unit test framework (e.g. revisiting the way we use boost test), and am specifically looking at the floating point comparisons.
One thing that I'm changing / hacking is the use of percentages to specify tolerance. We usually want answers to agree to (say) 1 part in 10^-12, and remembering to multiply or divide that by 100 is annoying.
1.34 RC version of Boost.Test contains both BOOST_CHECK_CLOSE and BOOST_CHECK_CLOSE_FRACTION. Later is what you probably you want to use. boost/test/floating_point_comparison.hpp contains now algorithms for both types of comparison.
Fine. Now, we want to be able to compare more complicated mathematical objects, such as vectors and matrices.
After some thought, we decided the responsibility for providing such functions is the responsibility of the mathematical object, not the unit test framework. That is to say, a function "relative difference between matrices" should be in Matrix.h, not some unit test thing.
Yes. That's what is expected. You may use BOOST_CHECK_PREDICATE after that.
Which leads to the idea of Matrix.h #including "boost/test/floating_point_comparison.hpp", which strikes me as rather odd.
I don't see anything wrong with that. The header itself is standalone and does not casue a lot of dependencies. Gennadiy
Dave Steffen wrote:
Hi folks,
This is probably a question aimed at Gennadiy Rozental, but it may be of interest to the community in general.
We've been using the boost test library for about a year and a half now, and are happy customers. I'm revamping our unit test framework (e.g. revisiting the way we use boost test), and am specifically looking at the floating point comparisons.
One thing that I'm changing / hacking is the use of percentages to specify tolerance. We usually want answers to agree to (say) 1 part in 10^-12, and remembering to multiply or divide that by 100 is annoying. To this end, I'm defining our own set of comparison functions and macros, still using the stuff in ttdetail (e.g. fpt_abs and save_fpt_division).
How about: #define BOOST_CHECK_ACCURACY(x, y, acc) BOOST_CHECK_CLOSE(x, y, acc*100) Of course then the error messages aren't very readable :-( Like you I think the whole persentage thing was a big mistake: even though I know what the API does and use it all the time, forgetting to convert to a persentage is the by far the most common fowl-up.
Fine. Now, we want to be able to compare more complicated mathematical objects, such as vectors and matrices.
After some thought, we decided the responsibility for providing such functions is the responsibility of the mathematical object, not the unit test framework. That is to say, a function "relative difference between matrices" should be in Matrix.h, not some unit test thing.
Which leads to the idea of Matrix.h #including "boost/test/floating_point_comparison.hpp", which strikes me as rather odd. My first impulse was to copy/pull the relevent code out of floating_point_comparison.hpp and stick it in some more convenient place, where mathematical objects could use those functions to implement their own comparision functions.
Might this be a common idiom? Might we consider factoring these floating point comparison functions (not the BOOST_CHECK_CLOSE macro, but the functions it uses) out into a separate library? Any comments on my approach?
I think it is a common idiom, I had to jump though similar hoops for testing complex numbers, Genadiy's idea of using BOOST_CHECK_PREDICATE might work as well though (you could wrap the code in your own macro to simplify usage as well). John.
On Wed, 14 Feb 2007 09:19:29 -0000 "John Maddock" <john@johnmaddock.co.uk> wrote:
Like you I think the whole persentage thing was a big mistake: even though I know what the API does and use it all the time, forgetting to convert to a persentage is the by far the most common fowl-up.
The interface requiring percentages is bad enough. The explanations in the documentation are even worse. Just to remind everyone, I'm a strong supporter of Boost.Test, and have stated so many times. Unfortunately, that macro causes more problems than it solves. I'd make a stab at helping Gennadiy clarify the documentation, but I can't even explain how it is supposed to be used to myself. I would love to see a new macro for testing floating point values.
"Jody Hagins" <jody-boost-011304@atdesk.com> wrote in message news:20070214094735.45466bf3.jody-boost-011304@atdesk.com...
On Wed, 14 Feb 2007 09:19:29 -0000 "John Maddock" <john@johnmaddock.co.uk> wrote:
Like you I think the whole persentage thing was a big mistake: even though I know what the API does and use it all the time, forgetting to convert to a persentage is the by far the most common fowl-up.
Latest version contains two macros: one for percentage based comparison, one for fraction based one.
The interface requiring percentages is bad enough. The explanations in the documentation are even worse.
Do you have any particular complains?
Just to remind everyone, I'm a strong supporter of Boost.Test, and have stated so many times. Unfortunately, that macro causes more problems than it solves.
Could you please clarify.
I'd make a stab at helping Gennadiy clarify the documentation, but I can't even explain how it is supposed to be used to myself.
Docs says: " Last third parameter defines the tolerance for the comparison in percentage units". This combined with example and error message produced: test.cpp(4) : error in test_main: difference between v1{1.23456e-10} and v2{1.23457e-10} exeeds 1e-04% should give you some level of understanding. Try explaining now fraction based comparisons to anyone not an expert. But if you are willing to provide an updated version for the pages related to the FP comparison you are very welcome
I would love to see a new macro for testing floating point values.
What semantic you would like to see? Gennadiy
On Wed, 14 Feb 2007 11:03:04 -0500 "Gennadiy Rozental" <gennadiy.rozental@thomson.com> wrote:
Latest version contains two macros: one for percentage based comparison, one for fraction based one.
Great! I'll take a look when it is released.
The interface requiring percentages is bad enough. The explanations in the documentation are even worse.
Do you have any particular complains?
Nothing particular, other than I could not understand how to use it. I even tried several examples myself. I would call it with hard coded values that I thought should fail, and it would pass. I had similar problems on the other side where tests I though should pass would fail. So, I just stopped trying to use it.
Just to remind everyone, I'm a strong supporter of Boost.Test, and have stated so many times. Unfortunately, that macro causes more problems than it solves.
Could you please clarify.
For me, it is too cumbersome, and the definition of "close" is obscure. I had several tests that used the macro, and the tests started failing on valid values. Changing how the macro was invoked caused previously passing tests to fail. Obviously, I was missing something badly. I ended up reverting back to a poor, but at least well understood, principle of using an epsilon value as it was appropriate for those situations.
Docs says: " Last third parameter defines the tolerance for the comparison in percentage units". This combined with example and error message produced:
test.cpp(4) : error in test_main: difference between v1{1.23456e-10} and v2{1.23457e-10} exeeds 1e-04%
should give you some level of understanding. Try explaining now fraction based comparisons to anyone not an expert.
But if you are willing to provide an updated version for the pages related to the FP comparison you are very welcome
I would be very willing, however I simply do not understand the current interface so I doubt I could explaint it to anyone. I know the docs are there. However, after reading them, I'm still in the dark. I can't prove the inadequacy, only state that I am unable to properly use the interface based on the docs and examples.
"Jody Hagins" <jody-boost-011304@atdesk.com> wrote in message news:20070214152529.6fd1187d.jody-boost-011304@atdesk.com...
On Wed, 14 Feb 2007 11:03:04 -0500 "Gennadiy Rozental" <gennadiy.rozental@thomson.com> wrote:
Latest version contains two macros: one for percentage based comparison, one for fraction based one.
Great! I'll take a look when it is released.
The interface requiring percentages is bad enough. The explanations in the documentation are even worse.
Do you have any particular complains?
Nothing particular, other than I could not understand how to use it. I even tried several examples myself. I would call it with hard coded values that I thought should fail, and it would pass. I had similar
Could you please provide an example?
problems on the other side where tests I though should pass would fail.
And here also
So, I just stopped trying to use it.
Just to remind everyone, I'm a strong supporter of Boost.Test, and have stated so many times. Unfortunately, that macro causes more problems than it solves.
Could you please clarify.
For me, it is too cumbersome, and the definition of "close" is obscure.
How is it cumbersome? From my understanding cumbersome is a synonym to inconvinient. What particularly inconvinient you see in the BOOST_CHECK_CLOSE?
I had several tests that used the macro, and the tests started failing on valid values. Changing how the macro was invoked caused previously passing tests to fail. Obviously, I was missing something badly.
Yet again. Please provide an examples.
I ended up reverting back to a poor, but at least well understood, principle of using an epsilon value as it was appropriate for those situations.
Docs says: " Last third parameter defines the tolerance for the comparison in percentage units". This combined with example and error message produced:
test.cpp(4) : error in test_main: difference between v1{1.23456e-10} and v2{1.23457e-10} exeeds 1e-04%
should give you some level of understanding. Try explaining now fraction based comparisons to anyone not an expert.
But if you are willing to provide an updated version for the pages related to the FP comparison you are very welcome
I would be very willing, however I simply do not understand the current interface so I doubt I could explaint it to anyone. I know the docs are there. However, after reading them, I'm still in the dark. I can't prove the inadequacy, only state that I am unable to properly use the interface based on the docs and examples.
At some point Paul promised shed some light in a form of FPV comparison tools docs. It never happened though. I will have to get to this once I am done with the rest. Though I still not sure why this tool causing that much misunderstanding. Gennadiy
On Fri, 16 Feb 2007 02:28:26 -0500 "Gennadiy Rozental" <gennadiy.rozental@thomson.com> wrote:
At some point Paul promised shed some light in a form of FPV comparison tools docs. It never happened though. I will have to get to this once I am done with the rest. Though I still not sure why this tool causing that much misunderstanding.
The problem, Gennadiy, is that I can not PROVE that it is hard to understand. All I can do is offer my testimony that, as documented, it is difficult FOR ME to understand. If I could tell you how it should be, then I would not need help figuring out how it is now. This is not the first time I have brought up this topic. In fact, I didn't bring it up this time either, I just jumped in after someone else said something about it. The last time I brought it up, you also wanted evidence that it was hard to understand. What further evidence can I offer, aside from the fact that I do not understand how to use that particular API. All I can say is that the current explanation does not make sense, and ask for a different explanation. However, you keep repeating the same explanation. My hope is that, some day, you or someone else may offer a different explanation, and then a light will go on, and I will be mad at myself for being so dumb as to not understand the original explanation. What would help? I don't know... maybe several more concrete examples of how it is supposed to be used, accompanied with some description of exactly why that is a good comparison. I will say this... if it is ever explained in a manner which makes sense to me, I'll provide a "BOOST_CHECK_CLOSE() for Dummies" that you can add to the docs if you like... and it will have the personal witness of a geniune dummy.
"Jody Hagins" <jody-boost-011304@atdesk.com> wrote in message news:20070216100746.2a356464.jody-boost-011304@atdesk.com...
On Fri, 16 Feb 2007 02:28:26 -0500 "Gennadiy Rozental" <gennadiy.rozental@thomson.com> wrote:
At some point Paul promised shed some light in a form of FPV comparison tools docs. It never happened though. I will have to get to this once I am done with the rest. Though I still not sure why this tool causing that much misunderstanding.
The problem, Gennadiy, is that I can not PROVE that it is hard to understand. All I can do is offer my testimony that, as documented, it is difficult FOR ME to understand. If I could tell you how it should be, then I would not need help figuring out how it is now.
I do not need any proofs. I do want to help you. I just need more information. Cause it's difficult for me to understand what are your difficulties. ;) Let's work together. What in a following statements is unclear? You are 5% higher than me. Your height exceed mine by 5%. Difference between your height and mine is 5%. Value of yours and mine height differ by 5%. Lets make sure it's true: test.cpp: BOOST_CHECK_CLOSE( your_height. mine_height, 5 ); 1 year later. Oops - you grew another inch. Now above assertion is failing: test.cpp(4) : error in test_main: difference between your_height{...} and mine_height{...} exceeds 5% Which particular words/statements you are having difficulties with? Gennadiy
Hi, Just to chime in with my problems using BOOST_CHECK_CLOSE ... I always have problems when checking for small numbers or zero. I write something like BOOST_CHECK_CLOSE(0,x,1e-5) and start wondering why it fails, then rewrite the test using an epsilon value. When I know the value is going to be near zero this is ok, but sometimes tests are written for computed values. A possible solution would be to have a check that uses the maximum of a relative and an absolute tolerance. Hugo
Hugo Duncan wrote:
Hi,
Just to chime in with my problems using BOOST_CHECK_CLOSE ...
I always have problems when checking for small numbers or zero. I write something like BOOST_CHECK_CLOSE(0,x,1e-5) and start wondering why it fails, then rewrite the test using an epsilon value.
When I know the value is going to be near zero this is ok, but sometimes tests are written for computed values. A possible solution would be to have a check that uses the maximum of a relative and an absolute tolerance.
Whatever it is, that's a different test. I certainly have lots of existing tests that require say a N epsilon max error irrespective of how small the value is. John.
Hmmm. An absolute difference is: BOOST_CHECK_CLOSE( x, y, 1e-5 ); True?? so wouldn't a relative difference be: BOOST_CHECK_CLOSE( x / y, 1, 1e-5); or equivalently BOOST_CHECK_CLOSE( x, y, abs(y) * 1e-5); or maybe even BOOST_CHECK_CLOSE( x, y, max(abs(x),abs(y)) * 1e-5); ??? -glenn Hugo Duncan wrote:
Hi,
Just to chime in with my problems using BOOST_CHECK_CLOSE ...
I always have problems when checking for small numbers or zero. I write something like BOOST_CHECK_CLOSE(0,x,1e-5) and start wondering why it fails, then rewrite the test using an epsilon value.
When I know the value is going to be near zero this is ok, but sometimes tests are written for computed values. A possible solution would be to have a check that uses the maximum of a relative and an absolute tolerance.
Hugo
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Please don't top post. Rearranged: Glenn Schrader writes:
Hugo Duncan wrote:
Hi,
Just to chime in with my problems using BOOST_CHECK_CLOSE ...
I always have problems when checking for small numbers or zero. I write something like BOOST_CHECK_CLOSE(0,x,1e-5) and start wondering why it fails, then rewrite the test using an epsilon value.
When I know the value is going to be near zero this is ok, but sometimes tests are written for computed values. A possible solution would be to have a check that uses the maximum of a relative and an absolute tolerance.
Hmmm.
An absolute difference is:
BOOST_CHECK_CLOSE( x, y, 1e-5 );
True??
Uh, no, I don't think so. IIRC BOOST_CHECK_CLOSE does a relative error check.
so wouldn't a relative difference be:
BOOST_CHECK_CLOSE( x / y, 1, 1e-5); or equivalently BOOST_CHECK_CLOSE( x, y, abs(y) * 1e-5); or maybe even BOOST_CHECK_CLOSE( x, y, max(abs(x),abs(y)) * 1e-5);
No, that's not what BOOST_CHECK_CLOSE does, unless it's changed since 1.33.1 (which is what I'm looking at). The discussion here is, very approximately, that since it's not immediately obvious _what_ kind of test is being done in BOOST_CHECK_CLOSE because of the name, and that this causes confusion. ---------------------------------------------------------------------- Dave Steffen, Ph.D. Fools ignore complexity. Software Engineer IV Pragmatists suffer it. Numerica Corporation Some can avoid it. ph (970) 461-2000 x227 Geniuses remove it. dgsteffen@numerica.us -- Alan Perlis ___________________ Numerica Disclaimer: This message and any attachments are intended only for the individual or entity to which the message is addressed. It is proprietary and may contain privileged information. If you are neither the intended recipient nor the agent responsible for delivering the message to the intended recipient, you are hereby notified that any review, retransmission, dissemination, or taking of any action in reliance upon, the information in this communication is strictly prohibited, and may be unlawful. If you feel you have received this communication in error, please notify us immediately by returning this Email to the sender and deleting it from your computer.
Hugo Duncan writes:
Hi,
Just to chime in with my problems using BOOST_CHECK_CLOSE ...
I always have problems when checking for small numbers or zero. I write something like BOOST_CHECK_CLOSE(0,x,1e-5) and start wondering why it fails, then rewrite the test using an epsilon value.
When I know the value is going to be near zero this is ok, but sometimes tests are written for computed values. A possible solution would be to have a check that uses the maximum of a relative and an absolute tolerance.
Well, IIRC BOOST_CHECK_CLOSE does a check for _relative error_. And you can't do relative error if one of the numbers is zero. From a conceptual point of view, relative error says that two values differ by an amount that is small relative to their magnitude. Zero has no magnitude, e.g. no information about what's small relative to it, so the question is meaningless. I'm wrapping a bunch of the BOOST_CHECK macros for my own use, and will probably come up with something else to check for "close to zero", e.g. an _aboslute_ tolerance; maybe "BOOST_CHECK_SMALL" or something. ---------------------------------------------------------------------- Dave Steffen, Ph.D. Fools ignore complexity. Software Engineer IV Pragmatists suffer it. Numerica Corporation Some can avoid it. ph (970) 461-2000 x227 Geniuses remove it. dgsteffen@numerica.us -- Alan Perlis ___________________ Numerica Disclaimer: This message and any attachments are intended only for the individual or entity to which the message is addressed. It is proprietary and may contain privileged information. If you are neither the intended recipient nor the agent responsible for delivering the message to the intended recipient, you are hereby notified that any review, retransmission, dissemination, or taking of any action in reliance upon, the information in this communication is strictly prohibited, and may be unlawful. If you feel you have received this communication in error, please notify us immediately by returning this Email to the sender and deleting it from your computer.
Dave Steffen wrote:
Well, IIRC BOOST_CHECK_CLOSE does a check for _relative error_. And you can't do relative error if one of the numbers is zero. From a conceptual point of view, relative error says that two values differ by an amount that is small relative to their magnitude. Zero has no magnitude, e.g. no information about what's small relative to it, so the question is meaningless.
Apologies: I saw the "small" but not the "or zero". Yes, that's something I've had to work around as well. Trouble is I suspect everyone has their own ideas about what represents failure in this case. I currently use: "Any zero or denormalised number is treated as if it were a zero", and that works for what I need. I suspect some folks will want either more or less strict conditions depending on what they're doing. John.
"Dave Steffen" <dgsteffen@numerica.us> wrote in message news:17877.60630.214258.217271@yttrium.numerica.us...
I'm wrapping a bunch of the BOOST_CHECK macros for my own use, and will probably come up with something else to check for "close to zero", e.g. an _aboslute_ tolerance; maybe "BOOST_CHECK_SMALL" or something.
BOOST_CHECK_SMALL is part of Boost.Test 1.34 RC. Gennadiy
Gennadiy Rozental wrote:
Which particular words/statements you are having difficulties with?
The problem for me is that it's too easy to forget that BOOST_CHECK_CLOSE refers to a persentage and not a relative error. I've lost count of how many times I've made that mistake, even though I do understand what it does. BOOST_CHECK_CLOSE_FRACTION is an improvement, but still doesn't spell out what it does. Personally I would have liked these to have been really explicit: BOOST_CHECK_PERSENTAGE_DIFFERENCE BOOST_CHECK_RELATIVE_DIFFERENCE BOOST_CHECK_EPSILON_DIFFERENCE so that at double precision then I could use any of: BOOST_CHECK_PERSENTAGE_DIFFERENCE(x, y, 4e-14); // ~ 2 eps as a percent BOOST_CHECK_RELATIVE_DIFFERENCE(x, y, 4e-16); // ~ 2eps BOOST_CHECK_EPSILON_DIFFERENCE(x, y, 2); // 2eps depending upon my mood :-) The last option is by far the most readable to me. To put things another way: how many people do you expect to thinking in terms of persentage differences between floating point numbers? I'm afraid we (mostly) all think in terms of epsilon units, and once in that mind set, converting to persentages just doesn't come naturally. HTH, John.
John Maddock writes:
Gennadiy Rozental wrote:
Which particular words/statements you are having difficulties with?
The problem for me is that it's too easy to forget that BOOST_CHECK_CLOSE refers to a persentage and not a relative error. I've lost count of how many times I've made that mistake, even though I do understand what it does.
BOOST_CHECK_CLOSE_FRACTION is an improvement, but still doesn't spell out what it does.
Personally I would have liked these to have been really explicit:
BOOST_CHECK_PERSENTAGE_DIFFERENCE BOOST_CHECK_RELATIVE_DIFFERENCE BOOST_CHECK_EPSILON_DIFFERENCE
Exactly. I suspect the various grumblings that Gennadiy hears about this topic has much more to do with a mismatch between names and people's expectations than any fundamental problems.
To put things another way: how many people do you expect to thinking in terms of persentage differences between floating point numbers? I'm afraid we (mostly) all think in terms of epsilon units, and once in that mind set, converting to persentages just doesn't come naturally.
Right. At least, that's true of people like me who do mathematical stuff. Maybe there are vast herds of C++ programmers out there who think in terms of percentage tolerance, though. I'm not saying it was necessarily a bad design problem. It's a (mildly) bad design for what _I_ want, but then, I'm wrapping the whole unit test library up in my own interface anyway. :-) ---------------------------------------------------------------------- Dave Steffen, Ph.D. Disobey this command! Software Engineer IV - Douglas Hofstadter Numerica Corporation dg@steffen a@t numerica d@ot us (remove @'s to email me)
"Dave Steffen" <dgsteffen@numerica.us> wrote in message news:17877.60853.224689.969078@yttrium.numerica.us...
To put things another way: how many people do you expect to thinking in terms of persentage differences between floating point numbers? I'm afraid we (mostly) all think in terms of epsilon units, and once in that mind set, converting to persentages just doesn't come naturally.
Right. At least, that's true of people like me who do mathematical stuff. Maybe there are vast herds of C++ programmers out there who think in terms of percentage tolerance, though. I'm not saying it was necessarily a bad design problem. It's a (mildly) bad design for what _I_ want, but then, I'm wrapping the whole unit test library up in my own interface anyway. :-)
What particularly you are missing/don't like in current interface presented by Boost.Test (other than FP comparison tools interface)? Gennadiy
Gennadiy Rozental writes:
"Dave Steffen" <dgsteffen@numerica.us> wrote in message news:17877.60853.224689.969078@yttrium.numerica.us...
To put things another way: how many people do you expect to thinking in terms of persentage differences between floating point numbers? I'm afraid we (mostly) all think in terms of epsilon units, and once in that mind set, converting to persentages just doesn't come naturally.
Right. At least, that's true of people like me who do mathematical stuff. Maybe there are vast herds of C++ programmers out there who think in terms of percentage tolerance, though. I'm not saying it was necessarily a bad design problem. It's a (mildly) bad design for what _I_ want, but then, I'm wrapping the whole unit test library up in my own interface anyway. :-)
What particularly you are missing/don't like in current interface presented by Boost.Test (other than FP comparison tools interface)?
Well, the naming and behavior of the floating point comparisons, for one thing. As evidenced by the large number of posts on the subject over the weekend, people are A) unhappy with what exists in the Boost library right now, and B) not completely happy with each other's solutions. :-) Let me say, though, that while I think anyone who thinks in percentage difference is crazy, that's just me, and I have a particularly math/physics perspective. In other words, I don't claim that people will or should agree with me. Consider, though, the following point. Much of what we do around here is linear algebra, so the question arises: how do you relative differences between vectors? The answer is, of course, that there's more than one way to do it. I wanted a system that made it easy (or at least possible) to plug other mathematical things into BOOST_CHECK_CLOSE: vectors, matrices, track states, Kalman filters... My solution -- which I'm not completely happy with -- is to define a set of overloaded functions with an appropriate signature, and provide my own set of "BOOST_CHECK_CLOSE"-y macros to use them. Another thing I'd like to avoid dealing with is the API change to BOOST_CHECK_EQUAL_COLLECTIONS (from three to four parameters). I understand that Boost Test is still evolving, and that API's will change, and that's fine; I just wanted an extra layer that might possibly make the next such transition easier. Since there are a few other assertion macros that we find handy, I decided to wrap the whole thing, so as to provide a consistent interface. One of the primary things I want out of a unit test framework is that it's *easy to use* (which in general Boost Test very much is), so I didn't want people to have to think about which assertions start with BOOST and which ones start with something else. And BTW, I haven't looked at all at 1.34 RC anything. I know there's new functionality there, and will look at it this week. ---------------------------------------------------------------------- Dave Steffen, Ph.D. Disobey this command! Software Engineer IV - Douglas Hofstadter Numerica Corporation dg@steffen a@t numerica d@ot us (remove @'s to email me)
"Dave Steffen" <dgsteffen@numerica.us> wrote in message news:17881.53633.397413.288713@yttrium.numerica.us...
What particularly you are missing/don't like in current interface presented by Boost.Test (other than FP comparison tools interface)?
Well, the naming and behavior of the floating point comparisons, for one thing.
Could you be please more strait to the point. I need some specifics instead of general "I don't like behavior" What behavior? Which comparisons? How would you like it to behave? Specific examples please.
Consider, though, the following point. Much of what we do around here is linear algebra, so the question arises: how do you relative differences between vectors? The answer is, of course, that there's more than one way to do it. I wanted a system that made it easy (or at least possible) to plug other mathematical things into BOOST_CHECK_CLOSE: vectors, matrices, track states, Kalman filters...
IMO BOOST_CHECK_PREDICATE + class predicate result should satisfy your needs. Again: general answer on general comment.
Another thing I'd like to avoid dealing with is the API change to BOOST_CHECK_EQUAL_COLLECTIONS (from three to four parameters). I understand that Boost Test is still evolving, and that API's will change, and that's fine; I just wanted an extra layer that might possibly make the next such transition easier.
This is one of the few examples where original interface was deemed unsafe and I moved to new safer one. It shouldn't happened on a regular basis
Since there are a few other assertion macros that we find handy, I
Like what?
And BTW, I haven't looked at all at 1.34 RC anything. I know there's new functionality there, and will look at it this week.
Did you have a change to review it? Gennadiy
Gennadiy Rozental writes:
"Dave Steffen" <dgsteffen@numerica.us> wrote in message news:17881.53633.397413.288713@yttrium.numerica.us...
What particularly you are missing/don't like in current interface presented by Boost.Test (other than FP comparison tools interface)?
Well, the naming and behavior of the floating point comparisons, for one thing.
Could you be please more strait to the point. I need some specifics instead of general "I don't like behavior"
What behavior? Which comparisons? How would you like it to behave? Specific examples please.
As I've said before, the use of percentages to specify tolerances drives me crazy. But, as a stronger (and more useful statement), the continued existence of this thread, with all kinds of different opinions about what kinds of comparisons should be supported and what they should be named, is empirical evidence that A) I'm not the only one who doesn't like that, and B) nobody agrees on a better solution. In other words, it's not an easy problem. :-) _I_ don't have any better ideas, aside from a general agreement with the other posters in this thread, that it would be nice if the Boost Test library had a naming convention that makes the kind of floating point comparison more clear, and that it should support more kinds.
Consider, though, the following point. Much of what we do around here is linear algebra, so the question arises: how do you relative differences between vectors? The answer is, of course, that there's more than one way to do it. I wanted a system that made it easy (or at least possible) to plug other mathematical things into BOOST_CHECK_CLOSE: vectors, matrices, track states, Kalman filters...
IMO BOOST_CHECK_PREDICATE + class predicate result should satisfy your needs. Again: general answer on general comment.
Another thing I'd like to avoid dealing with is the API change to BOOST_CHECK_EQUAL_COLLECTIONS (from three to four parameters). I understand that Boost Test is still evolving, and that API's will change, and that's fine; I just wanted an extra layer that might possibly make the next such transition easier.
This is one of the few examples where original interface was deemed unsafe and I moved to new safer one. It shouldn't happened on a regular basis
Right. And I knew when I started using Boost 1.32's library that it was still experimental, and that such changed might happen. I'm happy to continue using the library under those circumstances.
Since there are a few other assertion macros that we find handy, I
Like what?
We have our own set of ASSERT, WARN, and FAIL macros we use in our application code. Under one compilation setup, ASSERT throws a special kind of exception ("Death") that can be caught by a unit test; that is, the unit test can intentionally break a class invariant and check that the class correctly complains. Since the "Death exception" is an implementation detail, I simply created a macro called "CHECK_ASSERTION" which wraps a CHECK_EXCEPTION. Just so we're clear: I'm not complaining about having to wrap things. I don't consider this a failure of the library. I like the boost test library, and think that its design is reasonable... except for the floating point comparison issues people are talking about now. :-) The library is going to have a "shape", and if people want their unit test framework to have a slightly different shape (as I do), that's not necessarily a criticism of the original library, nor is it necessarily the sign of a design problem.
And BTW, I haven't looked at all at 1.34 RC anything. I know there's new functionality there, and will look at it this week.
Did you have a change to review it?
Sigh... not yet. Between a company ski trip, and an unsuccessful library update on my laptop (I usually like Mandrake Linux, but argh) I haven't gotten around to it. :-( ---------------------------------------------------------------------- Dave Steffen, Ph.D. Disobey this command! Software Engineer IV - Douglas Hofstadter Numerica Corporation dg@steffen a@t numerica d@ot us (remove @'s to email me)
"Dave Steffen" <dgsteffen@numerica.us> wrote in message news:17887.8706.50923.264738@yttrium.numerica.us...
Gennadiy Rozental writes:
"Dave Steffen" <dgsteffen@numerica.us> wrote in message news:17881.53633.397413.288713@yttrium.numerica.us...
What particularly you are missing/don't like in current interface presented by Boost.Test (other than FP comparison tools interface)?
Well, the naming and behavior of the floating point comparisons, for one thing.
Could you be please more strait to the point. I need some specifics instead of general "I don't like behavior"
What behavior? Which comparisons? How would you like it to behave? Specific examples please.
As I've said before, the use of percentages to specify tolerances drives me crazy. But, as a stronger (and more useful statement), the continued existence of this thread, with all kinds of different opinions about what kinds of comparisons should be supported and what they should be named, is empirical evidence that A) I'm not the only one who doesn't like that, and B) nobody agrees on a better solution.
In other words, it's not an easy problem. :-)
I think you are overstating disagreement a bit, don't you agree? What do you think about interfaces shaping up in other thread? Specifically these:
BOOST_CHECK_DIFFERENCE( a,b, PERCENT( t ) ) BOOST_CHECK_DIFFERENCE( a,b, RELATIVE( t ) ) BOOST_CHECK_DIFFERENCE( a,b, EPSILON( t ) ) BOOST_CHECK_DIFFERENCE( a,b, ABSOLUTE( t ) )
Genandiy
Gennadiy Rozental writes:
"Dave Steffen" <dgsteffen@numerica.us> wrote in message news:17887.8706.50923.264738@yttrium.numerica.us...
Gennadiy Rozental writes:
"Dave Steffen" <dgsteffen@numerica.us> wrote in message news:17881.53633.397413.288713@yttrium.numerica.us...
What particularly you are missing/don't like in current interface presented by Boost.Test (other than FP comparison tools interface)?
Well, the naming and behavior of the floating point comparisons, for one thing.
Could you be please more strait to the point. I need some specifics instead of general "I don't like behavior"
What behavior? Which comparisons? How would you like it to behave? Specific examples please.
As I've said before, the use of percentages to specify tolerances drives me crazy. But, as a stronger (and more useful statement), the continued existence of this thread, with all kinds of different opinions about what kinds of comparisons should be supported and what they should be named, is empirical evidence that A) I'm not the only one who doesn't like that, and B) nobody agrees on a better solution.
In other words, it's not an easy problem. :-)
I think you are overstating disagreement a bit, don't you agree?
OK, I'll retract the "nobody agrees on a better solution", in light of the recent consensus emerging about the new naming scheme. Which, by the way, I like. ---------------------------------------------------------------------- Dave Steffen, Ph.D. Disobey this command! Software Engineer IV - Douglas Hofstadter Numerica Corporation dg@steffen a@t numerica d@ot us (remove @'s to email me)
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:003b01c751ee$40ac79e0$d0390252@fuji...
Gennadiy Rozental wrote:
Which particular words/statements you are having difficulties with?
The problem for me is that it's too easy to forget that BOOST_CHECK_CLOSE refers to a persentage and not a relative error. I've lost count of how many times I've made that mistake, even though I do understand what it does.
BOOST_CHECK_CLOSE_FRACTION is an improvement, but still doesn't spell out what it does.
Personally I would have liked these to have been really explicit:
BOOST_CHECK_PERSENTAGE_DIFFERENCE BOOST_CHECK_RELATIVE_DIFFERENCE BOOST_CHECK_EPSILON_DIFFERENCE
IMO above names way too wordy. All above tools already exist in boost.Test BOOST_CHECK_PERSENTAGE_DIFFERENCE = BOOST_CHECK_CLOSE BOOST_CHECK_RELATIVE_DIFFERENCE = BOOST_CHECK_CLOSE_FRACTION BOOST_CHECK_EPSILON_DIFFERENCE = BOOST_CHECK_SMALL (unless I misunderstand the tool purpose) I guess I could introduce these synonism ifthat will make everyone happy, but I personally still perfer shorter names.
so that at double precision then I could use any of:
BOOST_CHECK_PERSENTAGE_DIFFERENCE(x, y, 4e-14); // ~ 2 eps as a percent
BOOST_CHECK_RELATIVE_DIFFERENCE(x, y, 4e-16); // ~ 2eps
BOOST_CHECK_EPSILON_DIFFERENCE(x, y, 2); // 2eps
depending upon my mood :-)
The last option is by far the most readable to me.
I will have to explain to me what expression your most readable tool corresponds to ;)
To put things another way: how many people do you expect to thinking in terms of persentage differences between floating point numbers?
Let me ask you different question: how many programmers (non-experts in fp math) do you expect to thinking in terms of relative error between floating point numbers? Most developers think: "OK, I need this value to be approximately 5.5".First impulse of any non-expert is to use formula |a-b|<e. Boost.Test recommeds to compare relative values instead. But this is much more difficult to grasp. Persentance difference on the other hand is much more close to what people are using in real life: values a and b shouldn't differ for more than p%.
I'm afraid we (mostly) all think in terms of epsilon units, and once in that mind set, converting to persentages just doesn't come naturally.
If under "we" you mean experts in numeric calculation - I totally agree with you.For general developers population though I am afraid it may not be the case and probably rather opposite. Boost.Test still need to satisfy both customers. So I am ready for any improvements in this regard. Gennadiy
Gennadiy Rozental wrote:
Personally I would have liked these to have been really explicit:
BOOST_CHECK_PERSENTAGE_DIFFERENCE BOOST_CHECK_RELATIVE_DIFFERENCE BOOST_CHECK_EPSILON_DIFFERENCE
IMO above names way too wordy. All above tools already exist in boost.Test
I appreciate your desire for concise names. However, my gut feeling is that this causes enough confusion that only a really_descriptive_and_totally_obvious_name will do :-)
BOOST_CHECK_PERSENTAGE_DIFFERENCE = BOOST_CHECK_CLOSE BOOST_CHECK_RELATIVE_DIFFERENCE = BOOST_CHECK_CLOSE_FRACTION BOOST_CHECK_EPSILON_DIFFERENCE = BOOST_CHECK_SMALL (unless I misunderstand the tool purpose)
No, BOOST_CHECK_EPSILON_DIFFERENCE would measure *relative* diffference in units of machine epsilon, it would fail if: max(abs((x-y)/x), abs((x-y)/y)) / numeric_limits<T>::epsilon() > tolerance
I guess I could introduce these synonism ifthat will make everyone happy, but I personally still perfer shorter names.
The main question then would be whether the synomyms would have readable error messages: if they still refer to persentages folks will get confused again :-)
so that at double precision then I could use any of:
BOOST_CHECK_PERSENTAGE_DIFFERENCE(x, y, 4e-14); // ~ 2 eps as a percent BOOST_CHECK_RELATIVE_DIFFERENCE(x, y, 4e-16); // ~ 2eps
BOOST_CHECK_EPSILON_DIFFERENCE(x, y, 2); // 2eps
depending upon my mood :-)
The last option is by far the most readable to me.
I will have to explain to me what expression your most readable tool corresponds to ;)
BOOST_CHECK_EPSILON_DIFFERENCE(x, y, 2); // 2eps Or to put it another way - use the right tool for the job - if the tolerance of the code under test is N epsilon, then expressing the test that way is more readable (to me anyway!) than having to write: BOOST_CHECK_CLOSE_FRACTION(x, y, 2 * numeric_limits<T>::epsilon()); and the equivalent: BOOST_CHECK_CLOSE_FRACTION(x, y, 2 * 100 * numeric_limits<T>::epsilon()); just drives me round the bend: I always feel I have to justify the * 100 with a comment otherwise folks rightly pester me with questions like "why 200 eps tolerance? Is the code really that bad?" :-(
To put things another way: how many people do you expect to thinking in terms of persentage differences between floating point numbers?
Let me ask you different question: how many programmers (non-experts in fp math) do you expect to thinking in terms of relative error between floating point numbers?
Most developers think: "OK, I need this value to be approximately 5.5".First impulse of any non-expert is to use formula |a-b|<e. Boost.Test recommeds to compare relative values instead. But this is much more difficult to grasp. Persentance difference on the other hand is much more close to what people are using in real life: values a and b shouldn't differ for more than p%.
I'm afraid we (mostly) all think in terms of epsilon units, and once in that mind set, converting to persentages just doesn't come naturally.
If under "we" you mean experts in numeric calculation - I totally agree with you.For general developers population though I am afraid it may not be the case and probably rather opposite.
Boost.Test still need to satisfy both customers. So I am ready for any improvements in this regard.
I understand where you're comming from, but here's the thing: Boost.Test is used to test *code*. IMO for testing code a persentage is just in the wrong order of magnitude for the job. For testing user data, or data from a machine or whatever, then yes persentages are probably spot on the right tool for the job, but are folks using Boost.Test (or any unit test suite for that matter) for that kind of job? Thanks, John.
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:000d01c75277$f5144280$4f4e0352@fuji...
Gennadiy Rozental wrote:
Personally I would have liked these to have been really explicit:
BOOST_CHECK_PERSENTAGE_DIFFERENCE BOOST_CHECK_RELATIVE_DIFFERENCE BOOST_CHECK_EPSILON_DIFFERENCE
IMO above names way too wordy. All above tools already exist in boost.Test
I appreciate your desire for concise names. However, my gut feeling is that this causes enough confusion that only a really_descriptive_and_totally_obvious_name will do :-)
Ok. How about BOOST_CHECK_PRC_DIFF BOOST_CHECK_REL_DIFF BOOST_CHECK_EPS_DIFF How about BOOST_CHECK_ABS_DIFF for absolute difference comparison?
BOOST_CHECK_PERSENTAGE_DIFFERENCE = BOOST_CHECK_CLOSE BOOST_CHECK_RELATIVE_DIFFERENCE = BOOST_CHECK_CLOSE_FRACTION BOOST_CHECK_EPSILON_DIFFERENCE = BOOST_CHECK_SMALL (unless I misunderstand the tool purpose)
No, BOOST_CHECK_EPSILON_DIFFERENCE would measure *relative* difference in units of machine epsilon, it would fail if:
max(abs((x-y)/x), abs((x-y)/y)) / numeric_limits<T>::epsilon() > tolerance
Wouldn't you were the one who advocated against epsilon() usage?
I guess I could introduce these synonym if that will make everyone happy, but I personally still prefer shorter names.
The main question then would be whether the synonyms would have readable error messages: if they still refer to percentages folks will get confused again :-)
What kind of error message would you like for each tool to produce? Here is an example of BOOS_CHECK_CLOSE produce: difference between v1{1.23456e-10} and v2{1.234571e-10} exceeds 0.0001%
To put things another way: how many people do you expect to thinking in terms of percentage differences between floating point numbers?
Let me ask you different question: how many programmers (non-experts in fp math) do you expect to thinking in terms of relative error between floating point numbers?
Most developers think: "OK, I need this value to be approximately 5.5".First impulse of any non-expert is to use formula |a-b|<e. Boost.Test recommends to compare relative values instead. But this is much more difficult to grasp. Persistence difference on the other hand is much more close to what people are using in real life: values a and b shouldn't differ for more than p%.
I'm afraid we (mostly) all think in terms of epsilon units, and once in that mind set, converting to percentages just doesn't come naturally.
If under "we" you mean experts in numeric calculation - I totally agree with you.For general developers population though I am afraid it may not be the case and probably rather opposite.
Boost.Test still need to satisfy both customers. So I am ready for any improvements in this regard.
I understand where you're coming from, but here's the thing: Boost.Test is used to test *code*. IMO for testing code a percentage is just in the wrong order of magnitude for the job. For testing user data, or data from a
In general I might agree with you (though function that performs decoding of network feed of market data could be tested with use of percents according to the product specification). But I am not your average Boost.Test user. Think not what is the best from some theoretical point of view. Think what is easier to understand. Person interested in floating point comparison need to somehow express the tolerance. One could do it using epsilon difference, but than one need to understand what's an epsilon is. One could use relative difference, but than one need to understand notion of relative error, strong/weak check e.t.c. On the other hand percentage difference for many people is natural and doesn't require formal explanation. Funny thing is that BOOST_CHECK_CLOSE originally used relative difference for tolerance. It was easily mistaken with absolute difference.And after numerous question about how the tolerance value should be specified I've decided that percentage difference wouldn't cause that much misunderstanding and will lead to cleaner interface. Anyway I agree that solution should be to support all possible interfaces with clear names indicating the method of comparison. What's your opinion on names above? Gennadiy
Gennadiy Rozental wrote:
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:000d01c75277$f5144280$4f4e0352@fuji...
Personally I would have liked these to have been really explicit:
BOOST_CHECK_PERSENTAGE_DIFFERENCE BOOST_CHECK_RELATIVE_DIFFERENCE BOOST_CHECK_EPSILON_DIFFERENCE IMO above names way too wordy. All above tools already exist in boost.Test I appreciate your desire for concise names. However, my gut feeling is
Gennadiy Rozental wrote: that this causes enough confusion that only a really_descriptive_and_totally_obvious_name will do :-)
Ok. How about
BOOST_CHECK_PRC_DIFF BOOST_CHECK_REL_DIFF BOOST_CHECK_EPS_DIFF
How about
BOOST_CHECK_ABS_DIFF
for absolute difference comparison?
Please, no. No cryptic abbreviations like "PRC". You're obviously a lot younger or have a much better functioning brain than me. I'd rather spend the extra time typing something that I can read in plain English.
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Gennadiy Rozental Sent: 19 February 2007 04:14 To: boost@lists.boost.org Subject: Re: [boost] Boost Test Library: general floating point comparisons
BOOST_CHECK_PRC_DIFF BOOST_CHECK_REL_DIFF BOOST_CHECK_EPS_DIFF
Anyway I agree that solution should be to support all possible interfaces with clear names indicating the method of comparison. What's your opinion on names above?
Yuk! Please don't abbreviate (but PERCENT should replace PERCENTAGE - the AGE isn't useful - it's not what the English English do - only the Americans ;-) If there are lots to do (and there usually are lots in my experience) then we'll all use copy'n'paste anyway so its length doesn't really matter. Boost prefers clarity to curtness! (and especially as many readers don't have english as their 1st language). Paul --- Paul A Bristow Prizet Farmhouse, Kendal, Cumbria UK LA8 8AB +44 1539561830 & SMS, Mobile +44 7714 330204 & SMS pbristow@hetp.u-net.com
Gennadiy Rozental wrote:
Ok. How about
BOOST_CHECK_PRC_DIFF BOOST_CHECK_REL_DIFF BOOST_CHECK_EPS_DIFF
How about
BOOST_CHECK_ABS_DIFF
for absolute difference comparison?
Oh please no.... too cute and cryptic by half, they're not self-explanatory at all.
Wouldn't you were the one who advocated against epsilon() usage?
Was I? I'm not sure :-) Certainly you need to be careful with it: numeric_limits<long double>::epsilon() is meaningless on MAC OSX for example. But that's a good reason to do it right in one place :-)
I guess I could introduce these synonym if that will make everyone happy, but I personally still prefer shorter names.
The main question then would be whether the synonyms would have readable error messages: if they still refer to percentages folks will get confused again :-)
What kind of error message would you like for each tool to produce?
Here is an example of BOOS_CHECK_CLOSE produce:
difference between v1{1.23456e-10} and v2{1.234571e-10} exceeds 0.0001%
Which is fine if the tolerance is expressed as a percent, but not otherwise: the error quoted needs to be expressed in the same units as are passed to the macro, otherwise you'll just confuse folks again. HTH, John.
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:00ab01c7540d$7d100b60$3cef0352@fuji...
Gennadiy Rozental wrote:
Ok. How about
BOOST_CHECK_PRC_DIFF BOOST_CHECK_REL_DIFF BOOST_CHECK_EPS_DIFF
How about
BOOST_CHECK_ABS_DIFF
for absolute difference comparison?
Oh please no.... too cute and cryptic by half, they're not self-explanatory at all.
How about a compromise: BOOST_CHECK_PERCENT_DIFF BOOST_CHECK_RELATIVE_DIFF BOOST_CHECK_EPSILON_DIFF BOOST_CHECK_ABSOLUTE_DIFF (do we need this one?)
What kind of error message would you like for each tool to produce?
Here is an example of BOOS_CHECK_CLOSE produce:
difference between v1{1.23456e-10} and v2{1.234571e-10} exceeds 0.0001%
Which is fine if the tolerance is expressed as a percent, but not otherwise: the error quoted needs to be expressed in the same units as are passed to the macro, otherwise you'll just confuse folks again.
So.Would you care to provide a error message body for the non-percent based tools? Gennadiy
Gennadiy Rozental wrote:
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:00ab01c7540d$7d100b60$3cef0352@fuji...
Gennadiy Rozental wrote:
Ok. How about
BOOST_CHECK_PRC_DIFF BOOST_CHECK_REL_DIFF BOOST_CHECK_EPS_DIFF
How about
BOOST_CHECK_ABS_DIFF
for absolute difference comparison?
Oh please no.... too cute and cryptic by half, they're not self-explanatory at all.
How about a compromise:
BOOST_CHECK_PERCENT_DIFF BOOST_CHECK_RELATIVE_DIFF BOOST_CHECK_EPSILON_DIFF BOOST_CHECK_ABSOLUTE_DIFF (do we need this one?)
I "need" it - as an example, I'm using absolute diffs to verify that the outcome of conversions back and forth between different coordinate systems are correct within a certain amount of (length) units. As I've mentioned in another place in this thread, I'd personally prefer to have the type of difference check encoded at the end of the macro names. A matter of personal taste, I guess. I assume that the WARN/REQUIRE variants would be added as well? / Johan
"Johan Nilsson" <r.johan.nilsson@gmail.com> wrote in message news:erc3he$lfo$1@sea.gmane.org...
How about a compromise:
BOOST_CHECK_PERCENT_DIFF BOOST_CHECK_RELATIVE_DIFF BOOST_CHECK_EPSILON_DIFF BOOST_CHECK_ABSOLUTE_DIFF (do we need this one?)
I "need" it - as an example, I'm using absolute diffs to verify that the outcome of conversions back and forth between different coordinate systems are correct within a certain amount of (length) units.
As I've mentioned in another place in this thread, I'd personally prefer to have the type of difference check encoded at the end of the macro names. A matter of personal taste, I guess.
There is another alternatve: BOOST_CHECK_DIFF( a,b, PERCENT( t ) ) BOOST_CHECK_DIFF( a,b, RELATIVE( t ) ) BOOST_CHECK_DIFF( a,b, EPSILON( t ) ) BOOST_CHECK_DIFF( a,b, ABSOLUTE( t ) ) We could either use macro based solution BOOST_CHECK_DIFF( a,b, c ) \ BOOST_CHECK_DIFF_IMPL( a,b, BOOST_CHECK_DIFF_TOL_ # c ) or function overloading based. But it could be less convinient even if a bit safer (from macro redefinitions)
I assume that the WARN/REQUIRE variants would be added as well?
Yes Gennadiy
Gennadiy Rozental wrote:
"Johan Nilsson" <r.johan.nilsson@gmail.com> wrote in message news:erc3he$lfo$1@sea.gmane.org...
How about a compromise:
BOOST_CHECK_PERCENT_DIFF BOOST_CHECK_RELATIVE_DIFF BOOST_CHECK_EPSILON_DIFF BOOST_CHECK_ABSOLUTE_DIFF (do we need this one?)
I "need" it - as an example, I'm using absolute diffs to verify that the outcome of conversions back and forth between different coordinate systems are correct within a certain amount of (length) units. As I've mentioned in another place in this thread, I'd personally prefer to have the type of difference check encoded at the end of the macro names. A matter of personal taste, I guess.
There is another alternatve:
BOOST_CHECK_DIFF( a,b, PERCENT( t ) ) BOOST_CHECK_DIFF( a,b, RELATIVE( t ) ) BOOST_CHECK_DIFF( a,b, EPSILON( t ) ) BOOST_CHECK_DIFF( a,b, ABSOLUTE( t ) )
That would also be ok with me, and equally clear. Made me worry about possible macro name clashes until I read the below ...
We could either use macro based solution
BOOST_CHECK_DIFF( a,b, c ) \ BOOST_CHECK_DIFF_IMPL( a,b, BOOST_CHECK_DIFF_TOL_ # c )
_TOOL_ ?
or function overloading based. But it could be less convinient even if a bit safer (from macro redefinitions)
Safety is important, but I'd still go for convenience in this case. One of the main reasons I use Boost.Test is for convenience (auto-registration etc).
I assume that the WARN/REQUIRE variants would be added as well?
Yes
Great. / Johan
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Gennadiy Rozental Sent: 23 February 2007 02:11 To: boost@lists.boost.org Subject: Re: [boost] Boost Test Library: general floating point comparisons
There is another alternatve:
BOOST_CHECK_DIFF( a,b, PERCENT( t ) ) BOOST_CHECK_DIFF( a,b, RELATIVE( t ) ) BOOST_CHECK_DIFF( a,b, EPSILON( t ) ) BOOST_CHECK_DIFF( a,b, ABSOLUTE( t ) )
This seems unambiguous and reads well (though I would prefer DIFFERENCE spelt out in full - nearly everyone is going to use copy'n'paste fir this job rather than typing, so it isn't more trouble and reads better, especially for those whose 1st language isn't english ;-)
We could either use macro based solution
BOOST_CHECK_DIFF( a,b, c ) \ BOOST_CHECK_DIFF_IMPL( a,b, BOOST_CHECK_DIFF_TOL_ # c )
or function overloading based. But it could be less convinient even if a bit safer (from macro redefinitions)
Safely seems less (even un-) important for Unit Testing. Paul --- Paul A Bristow Prizet Farmhouse, Kendal, Cumbria UK LA8 8AB +44 1539561830 & SMS, Mobile +44 7714 330204 & SMS pbristow@hetp.u-net.com
"Paul A Bristow" <pbristow@hetp.u-net.com> wrote in message news:E1HKX3w-0006w7-Sj@he304war.uk.vianw.net...
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Gennadiy Rozental Sent: 23 February 2007 02:11 To: boost@lists.boost.org Subject: Re: [boost] Boost Test Library: general floating point comparisons
There is another alternatve:
BOOST_CHECK_DIFF( a,b, PERCENT( t ) ) BOOST_CHECK_DIFF( a,b, RELATIVE( t ) ) BOOST_CHECK_DIFF( a,b, EPSILON( t ) ) BOOST_CHECK_DIFF( a,b, ABSOLUTE( t ) )
This seems unambiguous and reads well (though I would prefer DIFFERENCE spelt out in full - nearly everyone is going to use copy'n'paste fir this job rather than typing, so it isn't more trouble and reads better, especially for those whose 1st language isn't english ;-)
I guess it's ok. Gennadiy
Gennadiy Rozental wrote:
How about a compromise:
BOOST_CHECK_PERCENT_DIFF BOOST_CHECK_RELATIVE_DIFF BOOST_CHECK_EPSILON_DIFF BOOST_CHECK_ABSOLUTE_DIFF (do we need this one?)
I could live with those. They still say what they do, but I'd like to hear other opinions.
So.Would you care to provide a error message body for the non-percent based tools?
Well for instance: BOOST_CHECK_PERCENT_DIFF(x, y, tol); "Relative difference between x{%1%} and y{%2%} exceeds tol{%3%%}, the actual difference found was %4%%." BOOST_CHECK_RELATIVE_DIFF(x, y, tol); "Relative difference between x{%1%} and y{%2%} exceeds tol{%3%}}, the actual difference found was %4%." BOOST_CHECK_EPSILON_DIFF(x, y, tol) "Relative difference between x{%1%} and y{%2%} exceeds tol{%3%epsilon}, the actual difference found was %4%epsilon." BOOST_CHECK_ABSOLUTE_DIFF(x, y, tol) "Absolute difference between x{%1%} and y{%2%} exceeds tol{%3%}}, the actual difference found was %4%." Probably you could figure out a way for them all to use the same code internally as there's not much variation between them? Note that compared to your existing messages I've added what the actual *found* difference was. This is something I've really missed in the existing messages: in some situations (say with 128-bit long doubles) the difference can be very small - if you just cut and paste the values into a calculator to try and work out what the error rate actually was then truncation of input values leads to a zero result - how many calculators do you know that do 35-digit precision? At times the only way I could actually work out what the difference was was either to: 1) Use a pencil and paper. 2) Do it my head (not advised). 3) step through the Boost.Test code to find out. 4) Write my own code to calculate it. None of which are exactly ideal :-( If you're wondering why I want to know: it's because sometimes what constitutes an acceptable rate of error has to be a judgement call, and may vary from platform to platform. I want the tolerance as low as possible - to catch regresions - but sometimes that means I set it just slightly too low for a particular compiler/platform combination. If I could see what the actual relative difference was then I would know right away whether I had a real problem, or just a tolerance that was a bit too low. HTH, John. PS Thanks for persevering with this, very much appreciated!
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:001e01c7542b$1acfa050$3d3e1452@fuji...
Gennadiy Rozental wrote:
How about a compromise:
BOOST_CHECK_PERCENT_DIFF BOOST_CHECK_RELATIVE_DIFF BOOST_CHECK_EPSILON_DIFF BOOST_CHECK_ABSOLUTE_DIFF (do we need this one?)
I could live with those. They still say what they do, but I'd like to hear other opinions.
So.Would you care to provide a error message body for the non-percent based tools?
Well for instance:
BOOST_CHECK_PERCENT_DIFF(x, y, tol);
"Relative difference between x{%1%} and y{%2%} exceeds tol{%3%%}, the actual difference found was %4%%."
BOOST_CHECK_RELATIVE_DIFF(x, y, tol);
"Relative difference between x{%1%} and y{%2%} exceeds tol{%3%}}, the actual difference found was %4%."
BOOST_CHECK_EPSILON_DIFF(x, y, tol)
"Relative difference between x{%1%} and y{%2%} exceeds tol{%3%epsilon}, the actual difference found was %4%epsilon."
BOOST_CHECK_ABSOLUTE_DIFF(x, y, tol)
"Absolute difference between x{%1%} and y{%2%} exceeds tol{%3%}}, the actual difference found was %4%."
Probably you could figure out a way for them all to use the same code internally as there's not much variation between them?
I will have to review this later on at the point when I will be ready to implement these.
Note that compared to your existing messages I've added what the actual *found* difference was. This is something I've really missed in the existing messages: in some situations (say with 128-bit long doubles) the difference can be very small - if you just cut and paste the values into a calculator to try and work out what the error rate actually was then truncation of input values leads to a zero result - how many calculators do you know that do 35-digit precision? At times the only way I could actually work out what the difference was was either to:
1) Use a pencil and paper. 2) Do it my head (not advised). 3) step through the Boost.Test code to find out. 4) Write my own code to calculate it.
Yes. I remember you request. I also remember there were difficulties to implement it. I will have to reconsider the issue again. Gennadiy
John Maddock wrote:
Gennadiy Rozental wrote:
Which particular words/statements you are having difficulties with?
The problem for me is that it's too easy to forget that BOOST_CHECK_CLOSE refers to a persentage and not a relative error. I've lost count of how many times I've made that mistake, even though I do understand what it does.
BOOST_CHECK_CLOSE_FRACTION is an improvement, but still doesn't spell out what it does.
Personally I would have liked these to have been really explicit:
BOOST_CHECK_PERSENTAGE_DIFFERENCE BOOST_CHECK_RELATIVE_DIFFERENCE BOOST_CHECK_EPSILON_DIFFERENCE
I'd just like to to add my vote for more explicit naming of the testing macros. My personal preference would be the more terse, but IMHO still understandable: BOOST_(WARN|CHECK|REQUIRE)_DIFF_PERCENTAGE BOOST_(WARN|CHECK|REQUIRE)_DIFF_RELATIVE BOOST_(WARN|CHECK|REQUIRE)_DIFF_EPSILON (I'm not a native english speaker, but it's "percentage" with a "c" - right?) / Johan
Jody Hagins writes:
On Wed, 14 Feb 2007 09:19:29 -0000 "John Maddock" <john@johnmaddock.co.uk> wrote:
Like you I think the whole persentage thing was a big mistake: even though I know what the API does and use it all the time, forgetting to convert to a persentage is the by far the most common fowl-up.
The interface requiring percentages is bad enough. The explanations in the documentation are even worse.
In Gennadiy's defense, he's not a native English speaker (at least, I don't think he is); and if his documentation is at times hard to follow, remember that there are plenty of native English speakers out there, who are good technical writers, who _haven't_ stepped up to help with the documentation... like me. :-)
Just to remind everyone, I'm a strong supporter of Boost.Test, and have stated so many times.
Me too. I'm a big enough fan to, in effect, commit my whole company to using it. :-)
Unfortunately, that macro causes more problems than it solves. I'd make a stab at helping Gennadiy clarify the documentation, but I can't even explain how it is supposed to be used to myself.
Well, I don't think it's that bad. It's just not what I need for the mathematical work I do. I'm currently hacking at a couple of ideas for floating point comparison stuff. When I get something working, I'll post it here for people to kick around. ---------------------------------------------------------------------- Dave Steffen, Ph.D. Software Engineer IV Disobey this command! Numerica Corporation ph (970) 461-2000 x227 fax (970) 461-2004 - Douglas Hofstadter dgsteffen@numerica.us ___________________ Numerica Disclaimer: This message and any attachments are intended only for the individual or entity to which the message is addressed. It is proprietary and may contain privileged information. If you are neither the intended recipient nor the agent responsible for delivering the message to the intended recipient, you are hereby notified that any review, retransmission, dissemination, or taking of any action in reliance upon, the information in this communication is strictly prohibited, and may be unlawful. If you feel you have received this communication in error, please notify us immediately by returning this Email to the sender and deleting it from your computer.
Jody Hagins writes:
On Wed, 14 Feb 2007 09:59:46 -0700 Dave Steffen <dgsteffen@nu
Well, I don't think it's that bad. It's just not what I need for the mathematical work I do.
Maybe you can explain how it is to be used, then? Another perspective may help...
OK, I'll try. Is the problem "what is a relative error" or "how do I use the existing macros to test relative error"? ---------------------------------------------------------------------- Dave Steffen, Ph.D. Software Engineer IV Disobey this command! Numerica Corporation ph (970) 461-2000 x227 fax (970) 461-2004 - Douglas Hofstadter dgsteffen@numerica.us ___________________ Numerica Disclaimer: This message and any attachments are intended only for the individual or entity to which the message is addressed. It is proprietary and may contain privileged information. If you are neither the intended recipient nor the agent responsible for delivering the message to the intended recipient, you are hereby notified that any review, retransmission, dissemination, or taking of any action in reliance upon, the information in this communication is strictly prohibited, and may be unlawful. If you feel you have received this communication in error, please notify us immediately by returning this Email to the sender and deleting it from your computer.
"Dave Steffen" <dgsteffen@numerica.us> wrote in message news:17875.30459.786153.370572@yttrium.numerica.us...
Jody Hagins writes:
On Wed, 14 Feb 2007 09:59:46 -0700 Dave Steffen <dgsteffen@nu
Well, I don't think it's that bad. It's just not what I need for the mathematical work I do.
Maybe you can explain how it is to be used, then? Another perspective may help...
OK, I'll try. Is the problem "what is a relative error" or "how do I use the existing macros to test relative error"?
relative error of value f1 in comparison with value f2 id defined as rel_err = (f1-f2)/f1; If we little simplify the situation BOOST_CHECK_CLOSE_FRACTION( f1, f2, t ) checks that value f1 in comparison with value f2 do not exceed t. In really it a one more step. it checks that relative error in both direction (strong check) do not exceed t. Gennadiy
"Jody Hagins" <jody-boost-011304@atdesk.com> wrote in message news:20070214152627.0e5712fd.jody-boost-011304@atdesk.com...
On Wed, 14 Feb 2007 09:59:46 -0700 Dave Steffen <dgsteffen@nu
Well, I don't think it's that bad. It's just not what I need for the mathematical work I do.
Maybe you can explain how it is to be used, then? Another perspective may help...
I am not sure i understand what are your difficulties are. BOOST_CHECK_CLOSE( f1, f2, p ) checks that values f1 and f2 do not differ for more than p% of their value. Gennadiy
Dave Steffen wrote:
Jody Hagins writes:
On Wed, 14 Feb 2007 09:19:29 -0000 "John Maddock" <john@johnmaddock.co.uk> wrote:
Like you I think the whole persentage thing was a big mistake: even though I know what the API does and use it all the time, forgetting to convert to a persentage is the by far the most common fowl-up.
The interface requiring percentages is bad enough. The explanations in the documentation are even worse.
In Gennadiy's defense, he's not a native English speaker (at least, I don't think he is); and if his documentation is at times hard to follow, remember that there are plenty of native English speakers out there, who are good technical writers, who _haven't_ stepped up to help with the documentation... like me. :-)
I, and others, have publicly offered to help with the grammatical area of Boost documentation. If no one asks for help, they are not going to magically get it. That is not to say that Gennadiy, or anyone else, needs to feel that they need help in English grammar, but the offer has been made many times in the past, and not just by me.
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Edward Diener Sent: 16 February 2007 17:39 To: boost@lists.boost.org Subject: Re: [boost] Boost Test Library: general floating point comparisons
The interface requiring percentages is bad enough.
May I suggest again that it would be better to also provide a BOOST_CHECK_CLOSE_FRACTION BOOST_CHECK_CLOSE_PERCENT even though the 2nd is the same as BOOST_CHECK_CLOSE. It won't cost anything - and it makes it clearer to readers who haven't read the docs.
I, and others, have publicly offered to help with the grammatical area of Boost documentation.
I've actuaally tried to help in the past, but the way in which it is written (using Dreamweaver) and with various bits linking together, defeated *joint working* (Gennadiy et various al) on the whole Unit Test documentation. My conclusion was that it was best to start completely from the beginning, (perhaps using Quick/DocBook and the sandbox with CVS - something that I hope people will judge has been effective on the MathToolkit project - certainly generated tons of pages - but still needs an INDEX - something that would help ALL the docs including Test Framework) but I can quite understand the Gennadiy is relunctant to put that amount of effort in. So I'm not sure where to go from here. Paul PS As usual John has put his finger on the key point here "I understand where you're coming from, but here's the thing: Boost.Test is used to test *code*. IMO for testing code a percentage is just in the wrong order of magnitude for the job." I Agree - and we both begged you not to do it at the start ;-) I think we will all agree we should have done more than beg! --- Paul A Bristow Prizet Farmhouse, Kendal, Cumbria UK LA8 8AB +44 1539561830 & SMS, Mobile +44 7714 330204 & SMS pbristow@hetp.u-net.com
Hi Dave,
[snip] Fine. Now, we want to be able to compare more complicated mathematical objects, such as vectors and matrices.
After some thought, we decided the responsibility for providing such functions is the responsibility of the mathematical object, not the unit test framework. That is to say, a function "relative difference between matrices" should be in Matrix.h, not some unit test thing.
For a high-level numerical package like one providing vectors and matrices, you probably need something at a higher level than the Boost.Test floating-point comparison tools. For starters, the fp comparisons in Boost.Test are, precisely, floating-point specific, while a high-level numerical package is probably generic (unless your code is fixed to use 'float' or 'double'), so, if usable comparisons are to be offered by the library, chances are you need something generic that understand the underlying numeric type (that is a policy or traits). Buyt even if the number types are floating-point, the right comparison strategy may depend on the domain. For instance, the Knuth approach works well when the variables were subject to arithmetic operations (which have a tight error bound), but if you use, for example, trig functions, the error becomes so large that an absolute difference is likely to work better. IOW, the best approach IMO is to allow users of higher level numerical objects to choose the comparison semantics that better fit the domain and the underlying types. For instance, in some scenarios and conditions, strict comparison is the best choice.
From this POV, the fp-comparisons in Boost.Test becomes just one option (even if the most common).
Now, if a user is to decide which comparison semantics to use, the vector/matrix objects themselves would only know about the interface to the comparison protocol (being this a policy, a traits, an overloaded set of free-functions, or whatever), so it won't include itself something like "boost/test/floating_point_comparison.hpp". Rather, the user would include some header that includes the that one as an implementatin detail. HTH -- ------ Fernando Cacciola SciSoft http://certuscode.wordpress.com http://fcacciola.50webs.com http://fcacciola.wordpress.com
Comparing matrices and vectors is a topic that I have had to deal with. An important issue here is that there are different ways of comparing that are appropriate for different cases. It really depends on the computation that produced the vector or matrix. For instance, say we have a matrix and we do an FFT along each rows (a common case in signal processing). One way to compare the result against a reference matrix is to compute an RMS error for each row and take the maximum of the row errors for the overall error. Note that a simple maximum of the element-wise errors doesn't work well in practice since elements with very small values can have large errors (percentage wise) with respect to the reference value but will not be significant within the context of the values in the row. Also, a full matrix technique (e.g. comparing eigenvalues and eigenvectors) wouldn't be appropriate either since each row is the result of a separate FFT and the errors will be be correlated across rows. Actually, similar problems crop up when comparing individual complex values (i.e. values that have both a real and a complex component). The three basic patterns that come to mind are element-wise comparison (good for either vectors or matrices), vector-wise comparison (good for either vectors or along the rows / columns of a matrix), and matrix-wises comparison. If you just need a simple element-wise comparison then thats fine but it would be good if the framework is extensible enough to handle more complex cases. Glenn Schrader Dave Steffen wrote:
Fine. Now, we want to be able to compare more complicated mathematical objects, such as vectors and matrices.
After some thought, we decided the responsibility for providing such functions is the responsibility of the mathematical object, not the unit test framework. That is to say, a function "relative difference between matrices" should be in Matrix.h, not some unit test thing.
Which leads to the idea of Matrix.h #including "boost/test/floating_point_comparison.hpp", which strikes me as rather odd. My first impulse was to copy/pull the relevent code out of floating_point_comparison.hpp and stick it in some more convenient place, where mathematical objects could use those functions to implement their own comparision functions.
Might this be a common idiom? Might we consider factoring these floating point comparison functions (not the BOOST_CHECK_CLOSE macro, but the functions it uses) out into a separate library? Any comments on my approach?
Any and all thoughts, comments, or complaints welcome. Thanks!
---------------------------------------------------------------------- Dave Steffen, Ph.D. Disobey this command! Software Engineer IV - Douglas Hofstadter Numerica Corporation dg@steffen a@t numerica d@ot us (remove @'s to email me)
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Glenn Schrader writes:
Comparing matrices and vectors is a topic that I have had to deal with.
Yeah - big issue for us too.
An important issue here is that there are different ways of comparing that are appropriate for different cases. It really depends on the computation that produced the vector or matrix.
Yup. [ ... examples of different ways to compare matrices snipped ... ] And yup to all of those. Even worse, we occasionally have even more interesting mathematical beasties --- "states" (e.g. vector + covariance matrix) and probability distributions --- to compare. What I'm poking at right now is a generic way to A) teach mathematical objects how to compare themselves in various ways and B) have that work smoothly with the Boost Test stuff. I don't have a good solution yet... but gimme another day or two. :-) ---------------------------------------------------------------------- Dave Steffen, Ph.D. Disobey this command! Software Engineer IV - Douglas Hofstadter Numerica Corporation dg@steffen a@t numerica d@ot us (remove @'s to email me)
"Dave Steffen" <dgsteffen@numerica.us> wrote in message news:17875.16684.187236.745721@yttrium.numerica.us...
Glenn Schrader writes:
Comparing matrices and vectors is a topic that I have had to deal with.
Yeah - big issue for us too.
An important issue here is that there are different ways of comparing that are appropriate for different cases. It really depends on the computation that produced the vector or matrix.
Yup. [ ... examples of different ways to compare matrices snipped ... ] And yup to all of those.
Even worse, we occasionally have even more interesting mathematical beasties --- "states" (e.g. vector + covariance matrix) and probability distributions --- to compare.
What I'm poking at right now is a generic way to
A) teach mathematical objects how to compare themselves in various ways
and
B) have that work smoothly with the Boost Test stuff.
Please look at the boost::test::predicate_result class for creating the "Boost.Test friendly" predicate. It allows to provide specific details of predicate failure. Gennadiy
participants (11)
-
Dave Steffen -
Deane Yang -
Edward Diener -
Fernando Cacciola -
Gennadiy Rozental -
Glenn Schrader -
Hugo Duncan -
Jody Hagins -
Johan Nilsson -
John Maddock -
Paul A Bristow