[Boost.Test] C++ Unit Tests with Boost.Test, Part 1

[Please do not mail me a copy of your followup] http://legalizeadulthood.wordpress.com/2009/07/04/c-unit-tests-with-boost-te... In this post, I will describe how to start making unit tests in C++ with Boost.Test. -- "The Direct3D Graphics Pipeline" -- DirectX 9 draft available for download http://www.xmission.com/~legalize/book/download/index.html Legalize Adulthood! http://legalizeadulthood.wordpress.com

Richard wrote:
[Please do not mail me a copy of your followup]
http://legalizeadulthood.wordpress.com/2009/07/04/c-unit-tests-with-boost-te...
In this post, I will describe how to start making unit tests in C++ with Boost.Test.
I glanced across them, and at first glance they look like a nice set of tutorials. He has 5 parts posted. John

[This is actually mainly comments to Part 2] Richard wrote:
[Please do not mail me a copy of your followup]
http://legalizeadulthood.wordpress.com/2009/07/04/c-unit-tests-with-boost-te...
In this post, I will describe how to start making unit tests in C++ with Boost.Test.
I've only glanced through the parts, so I might have missed something. Also please note that my comments are very much from a TDD point of view. In Part 2, you recommend using the CHECK macros by default. Being a long-time Boost.Test user (as well as a long-time TDD:er) I strongly believe that by default one should always use the REQUIRE variety of the Boost.Test macros, for a couple of reasons: - They most closely correspond to xUnit assertions. TDD:ers are used to having their test cases aborted upon an assertion failure. - When you have multiple verifications per test[*], one of the early verifications are quite commonly some kind of precondition/guard statement which could lead to memory corruption (or simply useless further testing) if the test continues. I know you mention this as a special case in your article, but if you use REQUIRE by default you will always be safe. People will probably object to the above, saying that the want to test multiple things in the same test, and want to see the result of all "related verifications". Sorry, but I just won't buy that, because: - You really don't want to manually inspect a lot of numbers/results when TDD:ing. Seeing the name of the failing test case should be enough most of the time. - If you really need to see e.g. "x out of y" verifications fail together with their output, from a _single_ test case, this is a clear indicator that you should split that into multiple (more specific) test cases. Other than this specific thing, it looks like a good introduction to testing with Boost.Test. Just my 0.02EUR. / Johan [*] As you normally should attempt to stay out of.

[Please do not mail me a copy of your followup]
boost-users@lists.boost.org spake the secret code
In Part 2, you recommend using the CHECK macros by default. Being a long-time Boost.Test user (as well as a long-time TDD:er) I strongly believe that by default one should always use the REQUIRE variety of the Boost.Test macros, for a couple of reasons [...]
I'll see about getting that updated soon.
People will probably object to the above, saying that the want to test multiple things in the same test, and want to see the result of all "related verifications". [...]
If you read through the whole series, you'll see that each test case tests one thing and one thing only. I link to xUnit Test Patterns book online, which has quite a bit to say about test smells. There's so much to cover on this subject and the posts were already getting a bit lengthy (originally I thought it would be 1 blog post!). Thanks for the feedback. -- "The Direct3D Graphics Pipeline" -- DirectX 9 draft available for download http://www.xmission.com/~legalize/book/download/index.html Legalize Adulthood! http://legalizeadulthood.wordpress.com

Richard
There's so much to cover on this subject and the posts were already getting a bit lengthy (originally I thought it would be 1 blog post!).
Thanks for the feedback.
And you only touched the surface ;) These are very nice. I definitely need to collect references to the online tutorials somewhere inside Boost.Test docs. Couple notes: Didn't you mean to include single header variant of UTF in first tutorial? You probably should also mention BOOST_FIXTURE_TEST_SUITE, since it's frequently a better way to manage fixtures. I'll look a bit more later and post more comments if I have any. Regards, Gennadiy

[Please do not mail me a copy of your followup]
boost-users@lists.boost.org spake the secret code
Richard
writes: There's so much to cover on this subject and the posts were already getting a bit lengthy (originally I thought it would be 1 blog post!).
Thanks for the feedback.
And you only touched the surface ;)
Yes, there's quite a bit inside Boost.Test (thanks, BTW). I have had a hard time getting bootstrapped into writing tests using it, though. My first unit tests were in NUnit and I found it very easy to become productive right away. At my last employer, we evaluated a bunch of test frameworks for C++ and ended up picking Boost.Test as our official test framework for C++. I had been using it for a while, mostly with BOOST_AUTO_TEST_CASE, but recently I wanted to learn more about fixtures to extract common setup/teardown code. I had to work with the documentation for a while in order to figure out how to do this. I don't know exactly what it is about the documentation, but its difficult for me to find stuff in there quickly. The easier it is to get started with a library, the more people use it. I don't know why, but Boost.Test feels hard to get started. Once I got it figured out, I decided that some tutorials would help others.
These are very nice. I definitely need to collect references to the online tutorials somewhere inside Boost.Test docs.
Feel free. I think the permalinks should be good for a long time now that my blog is on wordpress.com.
Didn't you mean to include single header variant of UTF in first tutorial?
I didn't want to do that because I wanted the first tutorial to cover setting up a whole project and the way I've been doing that is to have the dedicated main go into a single source file. I know there are other ways to have a "standalone single .cpp test project", but I don't feel those cases are typical. I always end up having one test file per SUT (i.e. production class) with all the test cases for that SUT in that source file. I always end up with an EXE project and a single main source file and one source file for each class I want to test.
You probably should also mention BOOST_FIXTURE_TEST_SUITE, since it's frequently a better way to manage fixtures.
That's covered in part 5 after I motivate its use with some test-driven GUI development. -- "The Direct3D Graphics Pipeline" -- DirectX 9 draft available for download http://www.xmission.com/~legalize/book/download/index.html Legalize Adulthood! http://legalizeadulthood.wordpress.com

Richard
I don't know why, but Boost.Test feels hard to get started.
Can't fix something you can't even pinpoint ;) I've tried to make it friendly for the first time users. Could it be that you came from C#, Java experience and thus have some different expectations?
Once I got it figured out, I decided that some tutorials would help others.
Yes. Tutorials are definitely a good media to promote the library. Maybe some more to come? to cover some more advanced topics? Gennadiy

[Please do not mail me a copy of your followup]
boost-users@lists.boost.org spake the secret code
Richard
writes: [...]
I don't know why, but Boost.Test feels hard to get started.
Can't fix something you can't even pinpoint ;)
Yeah, I know. If I have specific suggestions for improving the docs, you'll hear about them. I know its not beceause I started with NUnit.
Yes. Tutorials are definitely a good media to promote the library.
Maybe some more to come? to cover some more advanced topics?
I think what's needed next (as far as my blog is concerned) is showing how to use a C++ mocking framework, particularly for COM objects. -- "The Direct3D Graphics Pipeline" -- DirectX 9 draft available for download http://www.xmission.com/~legalize/book/download/index.html Legalize Adulthood! http://legalizeadulthood.wordpress.com

Johan Nilsson
[This is actually mainly comments to Part 2]
Richard wrote:
[Please do not mail me a copy of your followup]
http://legalizeadulthood.wordpress.com/2009/07/04/c-unit-tests-with-boost-te...
In this post, I will describe how to start making unit tests in C++ with Boost.Test.
I've only glanced through the parts, so I might have missed something. Also please note that my comments are very much from a TDD point of view.
In Part 2, you recommend using the CHECK macros by default. Being a long-time Boost.Test user (as well as a long-time TDD:er) I strongly believe that by default one should always use the REQUIRE variety of the Boost.Test macros, for a couple of reasons:
This only effectively implements single assertion per test case policy, which is what you as hardcore TDDer probably doing anyway. For all us other folks CHECK should be default level. REQUIRE should be used if you actually require one assertion to pass before you proceed to next one. Gennadiy

Gennadiy Rozental wrote:
Johan Nilsson
writes: [This is actually mainly comments to Part 2]
Richard wrote:
[Please do not mail me a copy of your followup]
http://legalizeadulthood.wordpress.com/2009/07/04/c-unit-tests-with-boost-te...
In this post, I will describe how to start making unit tests in C++ with Boost.Test.
I've only glanced through the parts, so I might have missed something. Also please note that my comments are very much from a TDD point of view.
In Part 2, you recommend using the CHECK macros by default. Being a long-time Boost.Test user (as well as a long-time TDD:er) I strongly believe that by default one should always use the REQUIRE variety of the Boost.Test macros, for a couple of reasons:
This only effectively implements single assertion per test case policy, which is what you as hardcore TDDer probably doing anyway. For all us other folks CHECK should be default level. REQUIRE should be used if you actually require one assertion to pass before you proceed to next one.
Hmm, I wouldn't describe myself as "hard-core" TDD:er but, yes, having a single assertion per test case is definitely something I aim for. Multiple assertions are ok when verifying different aspects of the same logical state, but even then such a thing can be split into multiple test cases (or fixed by adding a custom assert). As for REQUIRE vs CHECK there's another point of view IMHO; either the SUT works or it does not. Why (in automated testing) bother checking whether something "half-works"? If you are into "manually" verifying stuff by looking at and interpreting large amounts of output on the screen IMHO you're not really doing automated unit/developer testing. (Ok, so maybe I'm a "medium-core" TDD:er then :) I still can appreciate that Boost.Test can be used in many testing scenarios and strategies, of which one TDD is only one application. The series of blog posts though seems to be mainly written as an introduction on how to do test-first development with Boost.Test, even though the title starts out with "C++ Unit Tests ..." - which could mean many different things. Hence my original comments. Regards, Johan

Johan Nilsson
As for REQUIRE vs CHECK there's another point of view IMHO; either the SUT works or it does not. Why (in automated testing) bother checking whether something "half-works"? If you are into "manually" verifying stuff by looking at and interpreting large amounts of output on the screen IMHO you're not really doing automated unit/developer testing.
I am not gonna argue virtues of of using single or multiple asserts in test cases. My only point is that you can't have it both ways. Either you have single assertion per test case policy, or you agree people to use CHECK to validate independent assertions within the same test case. REQUIRE still needs to be used in case of dependent assertions. Gennadiy
participants (4)
-
Gennadiy Rozental
-
Johan Nilsson
-
John Phillips
-
legalize+jeeves@mail.xmission.com