[Boost.Test] reporting tests with warnings

newer
boost 1.48, Qt and [Parse error at...

older
[lambda] Using Boost.Lambda with...

Olaf Meeuwissen

15 Nov 2011 15 Nov '11

10:52 a.m.

Hi all, For our CI setup I would like to be able to report the following results from our unit test suite: - number of tests aborted (started but did not complete) - number of tests failed - number of tests passed without warnings - number of tests passed with warnings The first two are supported just fine but the last two are collapsed into a single count in the reports. In the logs, I can make out the warning messages from the passing tests, but I can't seem to get at a count of the number of such tests in the (XML) reports. Any suggestions? Thanks in advance, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962

Show replies by date

Brian Allison

15 Nov 15 Nov

11:05 a.m.

I only use unit tests that either pass (complete or *the *expected exception is caught) or fail. In the same way that a 2-symbol language is Turing complete, a pass/fail system of unit tests is also complete. Adding N other levels (N >= 1) for warnings doesn't add anything and can make your type algebra unnecessarily complex. On Tue, Nov 15, 2011 at 5:52 AM, Olaf Meeuwissen wrote:

...

Hi all,

For our CI setup I would like to be able to report the following results from our unit test suite:

- number of tests aborted (started but did not complete) - number of tests failed - number of tests passed without warnings - number of tests passed with warnings

The first two are supported just fine but the last two are collapsed into a single count in the reports. In the logs, I can make out the warning messages from the passing tests, but I can't seem to get at a count of the number of such tests in the (XML) reports.

Any suggestions?

Thanks in advance, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962 _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Olaf Meeuwissen

11:32 p.m.

Brian Allison writes:

...

I only use unit tests that either pass (complete or the expected exception is caught) or fail.

Good for you, but I don't.

...

In the same way that a 2-symbol language is Turing complete, a pass/fail system of unit tests is also complete.

Complete though it may be, it doesn't address my needs.

...

Adding N other levels (N >= 1) for warnings doesn't add anything and can make your type algebra unnecessarily complex.

Boost.Test already support three "symbols": BOOST_REQUIRE_* BOOST_CHECK_* BOOST_WARN_* and catches system errors (which could be interpreted as a fourth but seem to get lumped in with any failing BOOST_REQUIRE_* assertions). What you're seems to be implying is that BOOST_WARN_* is a bad idea. I disagree, because there are "failure" modes that are not release critical. It is these that I want to flag through BOOST_WARN_* and report to our CI setup. We want to know how many we have and where they are so we can think about addressing them. So what I want to do is report the number of tests that passed all of their BOOST_REQUIRE_* and BOOST_CHECK_* assertions, but failed one or more BOOST_WARN_* assertions. So anyone have any suggestions? Thanks in advance, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962

Gennadiy Rozental

16 Nov 16 Nov

2:13 a.m.

Olaf Meeuwissen writes:

...

So what I want to do is report the number of tests that passed all of their BOOST_REQUIRE_* and BOOST_CHECK_* assertions, but failed one or more BOOST_WARN_* assertions.

So anyone have any suggestions?

There is no "legal" way to get what you want in Boost.Test now. I suggest you file a ticket and I'll consider it for next release. Meanwhile, the only component which gets any information about warnings is unit test log. You can "hack" around a bit to get the desired behavior by implementing custom unit_test_log_formatter. log_entry_start method receives log_entry_type, which will be equal to BOOST_UTL_ET_WARNING in case of warning. You can trap this event (don't forget to forward to the regular formatter after that) Using framework::current_test_case you can implement your own registry for the warnings information. Combine it with regular Boost.Test output and you'll get what you looking for. HTH, Gennadiy

Olaf Meeuwissen

11:57 p.m.

Gennadiy Rozental writes:

...

Olaf Meeuwissen writes:

...
So what I want to do is report the number of tests that passed all of their BOOST_REQUIRE_* and BOOST_CHECK_* assertions, but failed one or more BOOST_WARN_* assertions.

So anyone have any suggestions?

There is no "legal" way to get what you want in Boost.Test now. I suggest you file a ticket and I'll consider it for next release.

Will do.

...

Meanwhile, the only component which gets any information about warnings is unit test log. You can "hack" around a bit to get the desired behavior by implementing custom unit_test_log_formatter.

log_entry_start method receives log_entry_type, which will be equal to BOOST_UTL_ET_WARNING in case of warning. You can trap this event (don't forget to forward to the regular formatter after that)

Using framework::current_test_case you can implement your own registry for the warnings information. Combine it with regular Boost.Test output and you'll get what you looking for.

Thanks for the info. I'll have a look later. -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962

Olaf Meeuwissen

17 Nov 17 Nov

8:46 a.m.

Olaf Meeuwissen writes:

...

Gennadiy Rozental writes:

...
Olaf Meeuwissen writes:

...
So what I want to do is report the number of tests that passed all of their BOOST_REQUIRE_* and BOOST_CHECK_* assertions, but failed one or more BOOST_WARN_* assertions.

So anyone have any suggestions?

There is no "legal" way to get what you want in Boost.Test now. I suggest you file a ticket and I'll consider it for next release.

Will do.

Done. Ticket at https://svn.boost.org/trac/boost/ticket/6129 -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962

legalize+jeeves＠mail.xmission.com

16 Nov 16 Nov

6:35 p.m.

[Please do not mail me a copy of your followup] boost-users@lists.boost.org spake the secret code <87ipml2c8u.fsf@avasys.jp> thusly:

...

Boost.Test already support three "symbols":

BOOST_REQUIRE_* BOOST_CHECK_* BOOST_WARN_*

What you're seems to be implying is that BOOST_WARN_* is a bad idea.

Not just BOOST_WARN_*, but also BOOST_CHECK_* since they keep executing a failed test case. If you keep executing a test case that's already failed, it is highly likely in C++ that the test runner will simply crash.

...

I disagree, because there are "failure" modes that are not release critical.

Segregate them into a separate test project, change the warnings to failures and setup a separate CI build for them. -- "The Direct3D Graphics Pipeline" -- DirectX 9 version available for download http://legalizeadulthood.wordpress.com/the-direct3d-graphics-pipeline/ Legalize Adulthood! http://legalizeadulthood.wordpress.com

Gennadiy Rozental

17 Nov 17 Nov

5:31 a.m.

Richard writes:

...

...
Boost.Test already support three "symbols":

BOOST_REQUIRE_* BOOST_CHECK_* BOOST_WARN_*

What you're seems to be implying is that BOOST_WARN_* is a bad idea.

Not just BOOST_WARN_*, but also BOOST_CHECK_* since they keep executing a failed test case.

Why so radical view? This is TDD purist position and is not necessarily the only one.

...

If you keep executing a test case that's already failed, it is highly likely in C++ that the test runner will simply crash.

This is simply not true. Gennadiy

Olaf Meeuwissen

6:29 a.m.

Gennadiy Rozental writes:

...

Richard writes:

...
...
Boost.Test already support three "symbols":

BOOST_REQUIRE_* BOOST_CHECK_* BOOST_WARN_*

What you're seems to be implying is that BOOST_WARN_* is a bad idea.

Not just BOOST_WARN_*, but also BOOST_CHECK_* since they keep executing a failed test case.

Why so radical view? This is TDD purist position and is not necessarily the only one.

FTR, I use Boost.Test for TDD and want test cases to continue execution even though an assertion has failed already. It let's me see what part of the test case is already working (even if just by accident ;-) True purists stop the whole test suite at the first failed assertion ;-P Those same people abhor make's --keep-going option and have instructed their compilers to quit at the first warning ;-) Seriously, I like to get an overview of all the things that don't work yet (or have been broken) before I decide where to start fixing stuff. That's also why I like to have BOOST_WARN_* around so I can add tests *now* that are not targetted for the current milestone. When these tests become relevant to a milestone, I switch them to BOOST_CHECK_*. The warnings serve as a reminder of things to come.

...

...
If you keep executing a test case that's already failed, it is highly likely in C++ that the test runner will simply crash.

This is simply not true.

The only times Boost.Test runner has crashed for me was during test case registration and because of my stupidity. Hope this helps, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962

legalize+jeeves＠mail.xmission.com

19 Nov 19 Nov

12:27 a.m.

[Please do not mail me a copy of your followup] boost-users@lists.boost.org spake the secret code thusly:

...

Richard writes:

...
...
Boost.Test already support three "symbols":

BOOST_REQUIRE_* BOOST_CHECK_* BOOST_WARN_*

What you're seems to be implying is that BOOST_WARN_* is a bad idea.

Not just BOOST_WARN_*, but also BOOST_CHECK_* since they keep executing a failed test case.

Why so radical view?

I don't think its radical to want my tests to simply stop at the first point of failure.

...

This is TDD purist position and is not necessarily the only one.

I don't know if its "purist" or not and that's beside the point. I never said it was the "only" position.

...

...
If you keep executing a test case that's already failed, it is highly likely in C++ that the test runner will simply crash.

This is simply not true.

So if I do something like: BOOST_CHECK(p); BOOST_CHECK(p->accessor()); and p is a null pointer, what good does it do to continue after the first failed check? The pointer is null and anything else that is checked after that is useless. Dereferencing a null pointer is going to crash. Yes, on Windows that crash is turned into an exception that the test runner could intercept, but its still worthless. We don't even know if the whole heap has been corrupted at this point. The bottom line is that due to the low-level nature of C++, all kinds of things could go wrong in the test runner's environment that are outside the control of the test runner. Yes, you can guard against most kinds of controlled failures, but you simply can't guarantee that C++ test code (or the SUT) didn't horribly screw everything up. In managed languages like C# and Java, this simply isn't the case due to their different execution model. That's why for C++ code, I consider it a bad idea to use anything other than BOOS_REQUIRE. I want the test case to fail immediately and not continue any further at all once an expectation hasn't been met. Yes, its my opinion, but its not one that I grabbed onto randomly. Even in other environments (C#, for instance), I don't have any use for warnings or "mark a failure, but keep executing the test case". Whenever I've encountered this in other people's tests, its because the test is trying to test too many things instead of a single thing and so they are trying to report multiple different possible failures from a single test case. A single test should test one thing only. You may call this "TDD purism", but I simply call it good test design and it seems to be an opinion shared among other people I know that have been writing unit tests for a long time and have much more experience at it than I do. -- "The Direct3D Graphics Pipeline" -- DirectX 9 version available for download http://legalizeadulthood.wordpress.com/the-direct3d-graphics-pipeline/ Legalize Adulthood! http://legalizeadulthood.wordpress.com

Steven Watanabe

3:13 a.m.

AMDG On 11/18/2011 04:27 PM, Richard wrote:

...

[Please do not mail me a copy of your followup]

boost-users@lists.boost.org spake the secret code thusly:

...
Richard writes:

...
If you keep executing a test case that's already failed, it is highly likely in C++ that the test runner will simply crash.

This is simply not true.

So if I do something like:

BOOST_CHECK(p); BOOST_CHECK(p->accessor());

and p is a null pointer, what good does it do to continue after the first failed check? The pointer is null and anything else that is checked after that is useless.

In this case BOOST_REQUIRE is appropriate, but not all tests are like this. For example, with BOOST_AUTO_TEST_CASE(test_f) { BOOST_CHECK_EQUAL(f(0), 0); BOOST_CHECK_EQUAL(f(1), 1); BOOST_CHECK_EQUAL(f(2), 1); BOOST_CHECK_EQUAL(f(6), 8); BOOST_CHECK_EQUAL(f(15), 610); BOOST_CHECK_EQUAL(f(47), 2971215073); } BOOST_AUTO_TEST_CASE(test_C) { C c1(1, 2); C c2(3, 4); C c3 = c1 * c2; BOOST_CHECK_EQUAL(c3.real(), -5); BOOST_CHECK_EQUAL(c3.imag(), 10); } It's safe to continue executing even after a failure and I would want to continue, especially in the second case. I can't speak for anyone else, but the tests that I write tend to look a lot like this. In Christ, Steven Watanabe

legalize+jeeves＠mail.xmission.com

20 Nov 20 Nov

6:03 a.m.

[Please do not mail me a copy of your followup] boost-users@lists.boost.org spake the secret code <4EC71EEB.6000206@providere-consulting.com> thusly:

...

BOOST_AUTO_TEST_CASE(test_f) { BOOST_CHECK_EQUAL(f(0), 0); BOOST_CHECK_EQUAL(f(1), 1); BOOST_CHECK_EQUAL(f(2), 1); BOOST_CHECK_EQUAL(f(6), 8); BOOST_CHECK_EQUAL(f(15), 610); BOOST_CHECK_EQUAL(f(47), 2971215073); }

IMO, this is poor test design because its testing a whole bunch of different scenarios in a single test.

...

BOOST_AUTO_TEST_CASE(test_C) { C c1(1, 2); C c2(3, 4); C c3 = c1 * c2;

BOOST_CHECK_EQUAL(c3.real(), -5); BOOST_CHECK_EQUAL(c3.imag(), 10); }

In this case the problem is that you're testing equality by pieces instead of by the whole. Again, all my opinion, but I find the above tests poorly written. -- "The Direct3D Graphics Pipeline" -- DirectX 9 version available for download http://legalizeadulthood.wordpress.com/the-direct3d-graphics-pipeline/ Legalize Adulthood! http://legalizeadulthood.wordpress.com

4844

Age (days ago)

4849

Last active (days ago)

List overview

Download

11 comments

5 participants

participants (5)

Brian Allison
Gennadiy Rozental
legalize+jeeves＠mail.xmission.com
Olaf Meeuwissen
Steven Watanabe