[Boost.Test] BOOST_FAIL vs BOOST_ERROR
Hi, I'm new to Boost testing and I am looking to distinguish whether boost.test has a certain facility. It is clear from the documentation that BOOST_FAIL reports a problem and then quits, whereas BOOST_ERROR reports a problem then continues. My chief concern right now is how problems are reported actually- not control flow. I am wondering if boost.test has a facility to signal that something is wrong with the testware itself (not the software under test) . For instance, a reporting entry point that says "this is an exceptional condition and not the detection of a software bug". I think that _both_ BOOST_FAIL (or BOOST_REQUIRE) and BOOST_ERROR (or BOOST_CHECK) do indicate that there is a software under test failure (bug properly detected by testware). I'd like to hear I'm wrong or that there is another way to signal a testware failure. I suppose that I can use fixture support to attempt to notice any exceptional problems during setup so that I don't even get to the test execution code when there is an exceptional condition or environment problem. However, I'd like to try to wrap my head around the capabilities of the system and understand them as best as I can before proceeding. Thanks in advance for any help. Greg Christopher
Greg Christopher
For instance, a reporting entry point that says "this is an exceptional condition and not the detection of a software bug".
Condition of what then?
I think that _both_ BOOST_FAIL (or BOOST_REQUIRE) and BOOST_ERROR (or BOOST_CHECK) do indicate that there is a software under test failure (bug
They can indicate whatever you want it to indicate. If you require some particular condition before proceeding with test you check for it using BOOST_REQUIRE on top of the test case. You can also use fixtures.
properly detected by testware). I'd like to hear I'm wrong or that there is another way to signal a testware failure.
I suppose that I can use fixture support to attempt to notice any exceptional problems during setup so that I don't even get to the test execution code when there is an exceptional condition or environment problem. However, I'd like to try to wrap my head around the capabilities of the system and understand them as best as I can before proceeding.
Please let me know if understood your problem. Gennadiy
Hi Gennadiy ; thanks very much for your response...
-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users- bounces@lists.boost.org] On Behalf Of Gennadiy Rozental Sent: Friday, May 08, 2009 1:04 AM To: boost-users@lists.boost.org Subject: Re: [Boost-users] [Boost.Test] BOOST_FAIL vs BOOST_ERROR
Greg Christopher
writes: For instance, a reporting entry point that says "this is an exceptional condition and not the detection of a software bug".
Condition of what then?
For example, network down.
I think that _both_ BOOST_FAIL (or BOOST_REQUIRE) and BOOST_ERROR (or BOOST_CHECK) do indicate that there is a software under test failure (bug
They can indicate whatever you want it to indicate. If you require some particular condition before proceeding with test you check for it using BOOST_REQUIRE on top of the test case. You can also use fixtures.
Right, but the point is there are two reasons why I might not be able to continue: 1)A feature of the software under test is broken, and therefore I won't be able to test further. Still a bug in the software I'm testing. 2)Something exceptional has happened outside the ability of my software under test to deal with. At this point, I want to abend without indicating a software "bug". I do think fixtures are a great way to go. If the fixture initialize fails, don't run that test and report a problem in the testware or environment. But sometimes I may want to indicate that during the flow of the testing. I think that test reporting frameworks need at least three states they indicate- fail(bug), success, or "other". At a higher level, I want the system to _only_ contact the team executing the tests if "other" comes up so they can look at the testing system. As such systems over-report "failures" or bugs, trust in the testing system erodes. I also feel strongly that this has to be done within the reporting APIs, and can't be done via string conventions. Eventually conventions fail and systems that parse the logs will improperly report bugs. APIs for reporting should handle formatting and indirectly, control flow at the triage level where people respond to problems. I think everyone is in agreement on that- I just wonder if you handle this third state as far as reporting. Thanks! Greg
properly detected by testware). I'd like to hear I'm wrong or that there is another way to signal a testware failure.
I suppose that I can use fixture support to attempt to notice any exceptional problems during setup so that I don't even get to the test execution code when there is an exceptional condition or environment problem. However, I'd like to try to wrap my head around the capabilities of the system and understand them as best as I can before proceeding.
Please let me know if understood your problem.
Gennadiy
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Greg Christopher wrote:
I think that _both_ BOOST_FAIL (or BOOST_REQUIRE) and BOOST_ERROR (or BOOST_CHECK) do indicate that there is a software under test failure (bug They can indicate whatever you want it to indicate. If you require some particular condition before proceeding with test you check for it using BOOST_REQUIRE on top of the test case. You can also use fixtures.
Right, but the point is there are two reasons why I might not be able to continue: 1)A feature of the software under test is broken, and therefore I won't be able to test further. Still a bug in the software I'm testing. 2)Something exceptional has happened outside the ability of my software under test to deal with. At this point, I want to abend without indicating a software "bug".
I do think fixtures are a great way to go. If the fixture initialize fails, don't run that test and report a problem in the testware or environment. But sometimes I may want to indicate that during the flow of the testing.
I believe, at the moment fixture execution is part of the test case and is treated accordingly.
I think that test reporting frameworks need at least three states they indicate- fail(bug), success, or "other". At a higher level, I want the system to _only_ contact the team executing the tests if "other" comes up so they can look at the testing system. As such systems over-report "failures" or bugs, trust in the testing system erodes.
Boost.Test tools already has 3 levels: CHECK, REQUIRE and WARN. You can use third level to report conditions which are not software bugs. You'll need to abort the test case manually though, since Boost.Test continues on warning. The warnings are reported accordingly in an output. Gennadiy
-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users- bounces@lists.boost.org] On Behalf Of Gennadiy Rozental Sent: Tuesday, May 12, 2009 8:54 PM To: boost-users@lists.boost.org Subject: Re: [Boost-users] [Boost.Test] BOOST_FAIL vs BOOST_ERROR
Greg Christopher wrote:
I think that _both_ BOOST_FAIL (or BOOST_REQUIRE) and BOOST_ERROR (or BOOST_CHECK) do indicate that there is a software under test failure (bug They can indicate whatever you want it to indicate. If you require some particular condition before proceeding with test you check for it using BOOST_REQUIRE on top of the test case. You can also use fixtures.
Right, but the point is there are two reasons why I might not be able to continue: 1)A feature of the software under test is broken, and therefore I won't be able to test further. Still a bug in the software I'm testing. 2)Something exceptional has happened outside the ability of my software under test to deal with. At this point, I want to abend without indicating a software "bug".
I do think fixtures are a great way to go. If the fixture initialize fails, don't run that test and report a problem in the testware or environment. But sometimes I may want to indicate that during the flow of the testing.
I believe, at the moment fixture execution is part of the test case and is treated accordingly.
I think that test reporting frameworks need at least three states they indicate- fail(bug), success, or "other". At a higher level, I want the system to _only_ contact the team executing the tests if "other" comes up so they can look at the testing system. As such systems over-report "failures" or bugs, trust in the testing system erodes.
Boost.Test tools already has 3 levels: CHECK, REQUIRE and WARN. You can use third level to report conditions which are not software bugs. You'll need to abort the test case manually though, since Boost.Test continues on warning. The warnings are reported accordingly in an output.
This is true- we can use warn to mean aborting test although it seems to go against what most "first readers" of the code would think it is. This actually has raised another question for me. I am looking at fixtures. I would like to abort if the setup step fails. Unfortunately, setup is a constructor. I take it that means to abort if setup fails, I need to throw? Is there a specific exception I can throw to make it abort the current test case vs abort the test framework? Maybe just using a BOOST_REQUIRE() statement in the constructor will work? Thanks!! Greg
Gennadiy
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Greg Christopher
This is true- we can use warn to mean aborting test although it seems to go against what most "first readers" of the code would think it is.
warning by itself does not cause test case to abort.
This actually has raised another question for me. I am looking at fixtures. I would like to abort if the setup step fails.
Unfortunately, setup is a constructor. I take it that means to abort if setup fails, I need to throw?
Yes.
Is there a specific exception I can throw to make it abort the current test case vs abort the test framework?
Any C++ exception
Maybe just using a BOOST_REQUIRE() statement in the constructor will work?
BOOST_REQUIRE should work just fine. Note that you'll have 1 warning and 1 fatal failure reported in an output. Gennadiy
Hi and thanks,
This is true- we can use warn to mean aborting test although it seems to go against what most "first readers" of the code would think it is.
warning by itself does not cause test case to abort.
That is clear- again my concern is with the workflow that is triggered after execution is complete and people are reading the test reports. A warn coupled with a logging message that says test was aborted should be clear enough. BOOST_REQUIRE won't work in the setup step because it does exactly what I want to avoid :) Thanks, I will throw a c++ exception. Greg
Greg Christopher
BOOST_REQUIRE won't work in the setup step because it does exactly what I want to avoid :) Thanks, I will throw a c++ exception.
Actually C++ exception will lead to the same result - test case being marked as failed. If I understand what you need correctly, you need to throw boost::execution_aborted(). This exception is treated specially by execution monitor and no error is reported. Gennadiy
-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users- bounces@lists.boost.org] On Behalf Of Gennadiy Rozental Sent: Thursday, May 14, 2009 2:15 PM To: boost-users@lists.boost.org Subject: Re: [Boost-users] [Boost.Test] BOOST_FAIL vs BOOST_ERROR
Greg Christopher
writes: BOOST_REQUIRE won't work in the setup step because it does exactly what I want to avoid :) Thanks, I will throw a c++ exception.
Actually C++ exception will lead to the same result - test case being marked as failed.
If I understand what you need correctly, you need to throw boost::execution_aborted(). This exception is treated specially by execution monitor and no error is reported.
As it turns out this throws off the monitor. If my setup step throws this, boost test thinks that my test was ill defined (no assertions): Entering test case "TEST_2" Test case TEST_2 doesn't include any assertions Leaving test case "TEST_2" Leaving test suite "BasicTest" Leaving test suite "BasicTest" If I don't abort the setup, this executes all the test cases and doesn't say it doesn't include any assertions. I suppose the error message is harmless, but incorrect. Greg
Greg Christopher
As it turns out this throws off the monitor. If my setup step throws this, boost test thinks that my test was ill defined (no assertions):
Entering test case "TEST_2" Test case TEST_2 doesn't include any assertions Leaving test case "TEST_2" Leaving test suite "BasicTest" Leaving test suite "BasicTest"
If I don't abort the setup, this executes all the test cases and doesn't say it doesn't include any assertions. I suppose the error message is harmless, but incorrect.
Why was it incorrect? Your test case indeed did not run any assertions. This warning you can ignore safely in your case. Gennadiy
-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users- bounces@lists.boost.org] On Behalf Of Gennadiy Rozental Sent: Friday, May 22, 2009 4:56 AM To: boost-users@lists.boost.org Subject: Re: [Boost-users] [Boost.Test] BOOST_FAIL vs BOOST_ERROR
As it turns out this throws off the monitor. If my setup step throws
Greg Christopher
writes: this, boost test thinks that my test was ill defined (no assertions):
Entering test case "TEST_2" Test case TEST_2 doesn't include any assertions Leaving test case "TEST_2" Leaving test suite "BasicTest" Leaving test suite "BasicTest"
If I don't abort the setup, this executes all the test cases and doesn't say it doesn't include any assertions. I suppose the error message is harmless, but incorrect.
Why was it incorrect? Your test case indeed did not run any assertions. This warning you can ignore safely in your case.
Well, the message says "include any assertions" not "did not run any assertions". However the bigger issue is: How do I grep the log to see if there was an aborted test run due to setup? I guess I can look for this string "doesn't include any assertions". That seems like I'm doing it by side effect less than a more direct message that explains that the test case chose to quit because setup failed. Greg
Gennadiy
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Greg Christopher wrote:
Why was it incorrect? Your test case indeed did not run any assertions. This warning you can ignore safely in your case.
Well, the message says "include any assertions" not "did not run any assertions".
Good point. I guess I can change the warning message.
However the bigger issue is: How do I grep the log to see if there was an aborted test run due to setup?
You can't have it both ways. Either you want your test case to be reported as aborted or as as passed.
I guess I can look for this string "doesn't include any assertions". That seems like I'm doing it by side effect less than a more direct message that explains that the test case chose to quit because setup failed.
You can 2 options: 1. You can fail some REQUIRE level check and thus abort the test case. 2. You can detect that the test case did not run any assertions as a condition that it was aborted by fixture setup In both case I do not recommend to parse plain text output, but to switch to XML report and it's process it. Gennadiy
Hi Gennadiy, and welcome back!
-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users- bounces@lists.boost.org] On Behalf Of Gennadiy Rozental Sent: Saturday, June 06, 2009 12:02 AM To: boost-users@lists.boost.org Subject: Re: [Boost-users] [Boost.Test] BOOST_FAIL vs BOOST_ERROR
Greg Christopher wrote:
Why was it incorrect? Your test case indeed did not run any assertions. This warning you can ignore safely in your case.
Well, the message says "include any assertions" not "did not run any assertions".
Good point. I guess I can change the warning message.
However the bigger issue is: How do I grep the log to see if there was an aborted test run due to setup?
You can't have it both ways. Either you want your test case to be reported as aborted or as as passed.
There are three valid states- aborted, passed, or failed.
I guess I can look for this string "doesn't include any assertions". That seems like I'm doing it by side effect less than a more direct message that explains that the test case chose to quit because setup failed.
You can 2 options:
1. You can fail some REQUIRE level check and thus abort the test case. 2. You can detect that the test case did not run any assertions as a condition that it was aborted by fixture setup
Right. And the log does show differently.. but the message "doesn't include any assertions" is more like a canary that setup failed. It's fine- still usable. I like your suggestion to parse the xml. I will begin looking at XML output- thanks! Greg
Greg Christopher
There are three valid states- aborted, passed, or failed.
From Boost.Test prospective abort is failure condition.
Right. And the log does show differently.. but the message "doesn't include any assertions" is more like a canary that setup failed. It's fine- still usable.
The warning body changed in trunk. Gennadiy
participants (2)
-
Gennadiy Rozental -
Greg Christopher