whitebox testing with boost unit test framework?

I'm using the boost unit test framework to implement unit tests for code. I'm wondering if there is a best-practice for doing white-box testing/internal state inspection of types under test. My hack right now is to strip out "protected" and "private" from the header, but I hold my nose as I do that. Thoughts/suggestions? -- Chris Cleeland

Chris Cleeland
I'm using the boost unit test framework to implement unit tests for code. I'm wondering if there is a best-practice for doing white-box testing/internal state inspection of types under test. My hack right now is to strip out "protected" and "private" from the header, but I hold my nose as I do that.
I don't use private members in the header file and put a subclass in the test implementation that makes the parent class' protected API public. So you get, in the header, something like: class A { public: // protected: int i; }; and then in the test implementation file #include "A.hpp" class A_test : A { public: using i; }; so that in the tests you can access i without any problem. Hope this helps, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962

[Please do not mail me a copy of your followup] boost-users@lists.boost.org spake the secret code <87pqfhijjd.fsf@avasys.jp> thusly:
Chris Cleeland
writes: I'm using the boost unit test framework to implement unit tests for code. I'm wondering if there is a best-practice for doing white-box testing/internal state inspection of types under test. My hack right now is to strip out "protected" and "private" from the header, but I hold my nose as I do that.
I don't use private members in the header file and put a subclass in the test implementation that makes the parent class' protected API public. [...]
I do this with legacy code, i.e. code that wasn't designed with testing in mind. However, I find that I don't need to do this when I practice TDD because the classes were designed to be testable in the first place. -- "The Direct3D Graphics Pipeline" -- DirectX 9 version available for download http://legalizeadulthood.wordpress.com/the-direct3d-graphics-pipeline/ Legalize Adulthood! http://legalizeadulthood.wordpress.com

On Wed, Dec 21, 2011 at 3:57 PM, Richard
[Please do not mail me a copy of your followup]
boost-users@lists.boost.org spake the secret code <87pqfhijjd.fsf@avasys.jp> thusly:
Chris Cleeland
writes: I'm using the boost unit test framework to implement unit tests for code. I'm wondering if there is a best-practice for doing white-box testing/internal state inspection of types under test. My hack right now is to strip out "protected" and "private" from the header, but I hold my nose as I do that.
That is what I've come to accept as probably the cleanest possible solution without hacking/redefining "private"/"protected" in production vs. test code (even though it's far from what I would call elegant). For a class I essentially create a friend in a completely different "test" namespace, the unit tests go in that namespace and are linked against the object files to test them and validate class internals/invariants. The shipping code (headers) always have the 'missing' friend declarations, but as I said it's a 1 line that I've gotten to accept as the least ugly solution until I come across something that's more elegant.
I don't use private members in the header file and put a subclass in the test implementation that makes the parent class' protected API public. [...]
I do this with legacy code, i.e. code that wasn't designed with testing in mind.
However, I find that I don't need to do this when I practice TDD because the classes were designed to be testable in the first place.
While I think that's a great idea and it does to an extent make you think about your classes and making sure that they are simple enough to be externally testable, sometimes there are complicated relationships between classes, to test them in isolation, you may have to mock a few classes and replace them with test classes, this eventually leads to having to virtualize methods in order to be able to call them in the test classes without the class under test being aware. I have found this to cause unnecessary methods having to be virtualized in order to replace concrete objects with test objects in a class being tested, which ended up getting quite messy at times, so I essentially had to settle for the "friend" solution. Regards, Ahmed

Thanks to all for the suggestions! You are correct that I'm testing
"legacy" code, though in this particular case the legacy code is
completely open for change. But, I need unit tests before I go
mucking about so I know that I haven't made things any worse than they
are.
On Wed, Dec 21, 2011 at 7:04 PM, Ahmed Badran
That is what I've come to accept as probably the cleanest possible solution without hacking/redefining "private"/"protected" in production vs. test code (even though it's far from what I would call elegant). For a class I essentially create a friend in a completely different "test" namespace, the unit tests go in that namespace and are linked against the object files to test them and validate class internals/invariants. The shipping code (headers) always have the 'missing' friend declarations, but as I said it's a 1 line that I've gotten to accept as the least ugly solution until I come across something that's more elegant.
For some reason I'm having difficulties implementing this suggestion. The "legacy" code lives in a namespace, e.g., // ClassToBeTested.hh namespace mycompany { namespace internal { class ClassToBeTested { ... }; } // namespace internal } // namespace mycompany If I understand the suggestion, I think I want to put the testing in a different namespace, e.g., // test_ClassToBeTested.cxx namespace mycompany { namespace internal { namespace test { // here lies the boost unit test stuff that I create struct ClassToBeTested_suite_fixture { ... }; BOOST_FIXTURE_TEST_SUITE( ClassToBeTested_suite, ClassToBeTested_suite_fixture ) ... here lie unit tests... BOOST_AUTO_TEST_SUITE_END() } } } So, in the declaration for ClassToBeTested, I should have something like class ClassToBeTested { public: // public interface protected: // protected members/methods friend class mycompany::internal::test::ClassToBeTested_suite_fixture; }; When I compile, the compiler complains that it does not know about namespace 'test' ]$ clang++ namespace_test.cxx namespace_test.cxx:4:46: error: no member named 'test' in namespace 'mycompany::internal' friend class ::mycompany::internal::test::ClassToBeTested_suite_fixture; ~~~~~~~~~~~~~~~~~~~~~~~^ namespace_test.cxx:6:8: error: expected external declaration } } } ^ 2 errors generated. The only way I've found around this is to forward declare ClassToBeTested_suite_fixture in the ClassToBeTested header, e.g., // ClassToBeTested.hh namespace mycompany { namespace internal { namespace test { class ClassToBeTested_suite_fixture; } class ClassToBeTested { ... }; } // namespace internal } // namespace mycompany Is this the technique that you use? I wonder, because it's more than one lane and still intrusive, and I'm wondering if I'm missing something. -- Chris Cleeland

On Thu, Dec 22, 2011 at 8:59 AM, Chris Cleeland
Thanks to all for the suggestions! You are correct that I'm testing "legacy" code, though in this particular case the legacy code is completely open for change. But, I need unit tests before I go mucking about so I know that I haven't made things any worse than they are.
On Wed, Dec 21, 2011 at 7:04 PM, Ahmed Badran
wrote: That is what I've come to accept as probably the cleanest possible solution without hacking/redefining "private"/"protected" in production vs. test code (even though it's far from what I would call elegant). For a class I essentially create a friend in a completely different "test" namespace, the unit tests go in that namespace and are linked against the object files to test them and validate class internals/invariants. The shipping code (headers) always have the 'missing' friend declarations, but as I said it's a 1 line that I've gotten to accept as the least ugly solution until I come across something that's more elegant.
For some reason I'm having difficulties implementing this suggestion.
The "legacy" code lives in a namespace, e.g.,
// ClassToBeTested.hh namespace mycompany { namespace internal { class ClassToBeTested { ... }; } // namespace internal } // namespace mycompany
If I understand the suggestion, I think I want to put the testing in a different namespace, e.g.,
// test_ClassToBeTested.cxx namespace mycompany { namespace internal { namespace test { // here lies the boost unit test stuff that I create struct ClassToBeTested_suite_fixture { ... }; BOOST_FIXTURE_TEST_SUITE( ClassToBeTested_suite, ClassToBeTested_suite_fixture ) ... here lie unit tests... BOOST_AUTO_TEST_SUITE_END() } } }
So, in the declaration for ClassToBeTested, I should have something like
class ClassToBeTested { public: // public interface protected: // protected members/methods friend class mycompany::internal::test::ClassToBeTested_suite_fixture; };
When I compile, the compiler complains that it does not know about namespace 'test'
]$ clang++ namespace_test.cxx namespace_test.cxx:4:46: error: no member named 'test' in namespace 'mycompany::internal' friend class ::mycompany::internal::test::ClassToBeTested_suite_fixture; ~~~~~~~~~~~~~~~~~~~~~~~^ namespace_test.cxx:6:8: error: expected external declaration } } } ^ 2 errors generated.
In your header (the class to be tested, you need to forward declare the test namespace and the tester class, e.g. // ClassToBeTested.h // You may be able to generate the forward declaration using a script and put it in an include file that you // include here. namespace test { class TesterOfClassToBeTested; } namespace myOtherNamespace { // optional class ClassToBeTested { friend class test::TesterOfClassToBeTested; // here's the one line :) public: private: }; } // TesterOfClassToBeTested.cpp #include "ClassToBeTested.h" #define foreach BOOST_FOREACH using namespace std; using namespace boost; using namespace stage; namespace test { class TesterOfClassToBeTested { public: TesterOfClassToBeTested() { } void copy_constructor() { ClassToBeTested loc2; ClassToBeTested loc(loc2); BOOST_REQUIRE(loc.locIdStruct == loc2.locIdStruct); } ~TesterOfClassToBeTested() { } private: }; } BOOST_AUTO_TEST_SUITE(TesterOfClassToBeTested) struct fixture { test::TesterOfClassToBeTested t; }; BOOST_FIXTURE_TEST_CASE(copy_constructor, fixture) { t.copy_constructor(); } BOOST_AUTO_TEST_SUITE_END()
The only way I've found around this is to forward declare ClassToBeTested_suite_fixture in the ClassToBeTested header, e.g.,
// ClassToBeTested.hh namespace mycompany { namespace internal { namespace test { class ClassToBeTested_suite_fixture; } class ClassToBeTested { ... }; } // namespace internal } // namespace mycompany
Is this the technique that you use? I wonder, because it's more than one lane and still intrusive, and I'm wondering if I'm missing something.
-- Chris Cleeland _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

On Thu, Dec 22, 2011 at 1:15 PM, Ahmed Badran
In your header (the class to be tested, you need to forward declare the test namespace and the tester class, e.g.
// ClassToBeTested.h
// You may be able to generate the forward declaration using a script and put it in an include file that you // include here. namespace test { class TesterOfClassToBeTested; }
namespace myOtherNamespace { // optional class ClassToBeTested { friend class test::TesterOfClassToBeTested; // here's the one line :) public: private: }; }
I had done that. But I think I've figured out my problem.
// TesterOfClassToBeTested.cpp
#include "ClassToBeTested.h"
#define foreach BOOST_FOREACH
using namespace std; using namespace boost; using namespace stage;
namespace test { class TesterOfClassToBeTested { public: TesterOfClassToBeTested() { } void copy_constructor() { ClassToBeTested loc2; ClassToBeTested loc(loc2); BOOST_REQUIRE(loc.locIdStruct == loc2.locIdStruct); } ~TesterOfClassToBeTested() { } private: }; }
BOOST_AUTO_TEST_SUITE(TesterOfClassToBeTested)
struct fixture { test::TesterOfClassToBeTested t; };
BOOST_FIXTURE_TEST_CASE(copy_constructor, fixture) { t.copy_constructor(); }
BOOST_AUTO_TEST_SUITE_END()
What you've got above is a different usage of Boost::Unit than what I have. In my code, I have something like struct ClassToBeTested_suite_fixture { ... }; BOOST_FIXTURE_TEST_SUITE( ClassToBeTested_suite, ClassToBeTested_suite_fixture ) BOOST_AUTO_TEST_CASE( case1 ) { // code to do testing } BOOST_AUTO_TEST_CASE( case2 ) { // code to test a different case } BOOST_AUTO_TEST_SUITE_END() I designated friendship to ClassToBeTested_suite_fixture, but was trying to access the private data members in 'case1', 'case2', etc. That doesn't work because even though the cases inherit from the fixture, friendship does not get inherited. Indeed, when I went back to the header for ClassToBeTested and added a friend designation for ClassToBeTested::case1, it compiled just fine. I am a n00b to boost's unit test framework. I stumbled upon some examples that used the construct that I have been using, and they've worked just fine up until this case. Now that I have a need, I will have to learn just a little more. "Extreme Learning" :) Thanks for showing me a way that works for you. I'm certain it would have taken me a lot longer to figure that out to the point where I may have simply reverted to preprocessor tricks of "#define (protected|private) public" in the test case code. -- Chris Cleeland

[Please do not mail me a copy of your followup]
boost-users@lists.boost.org spake the secret code
I am a n00b to boost's unit test framework. [...]
You may want to read through the 5-part tutorial I wrote on using Boost.Test: http://legalizeadulthood.wordpress.com/2009/07/04/c-unit-tests-with-boost-te... -- "The Direct3D Graphics Pipeline" -- DirectX 9 version available for download http://legalizeadulthood.wordpress.com/the-direct3d-graphics-pipeline/ Legalize Adulthood! http://legalizeadulthood.wordpress.com

on Thu Dec 22 2011, legalize+jeeves-AT-mail.xmission.com (Richard) wrote:
You may want to read through the 5-part tutorial I wrote on using Boost.Test:
http://legalizeadulthood.wordpress.com/2009/07/04/c-unit-tests-with-boost-te...
Gennadiy, if this tutorial is any good I suggest you link to it from the Boost.Test documentation. -- Dave Abrahams BoostPro Computing http://www.boostpro.com

Dave Abrahams
on Thu Dec 22 2011, legalize+jeeves-AT-mail.xmission.com (Richard) wrote:
You may want to read through the 5-part tutorial I wrote on using Boost.Test:
http://legalizeadulthood.wordpress.com/2009/07/04/c-unit-tests-with-boost-te...
Gennadiy, if this tutorial is any good I suggest you link to it from the Boost.Test documentation.
I am familiar with this one, there are couple more around. I am planning to add page with links. Gennadiy

[Please do not mail me a copy of your followup]
boost-users@lists.boost.org spake the secret code
on Thu Dec 22 2011, legalize+jeeves-AT-mail.xmission.com (Richard) wrote:
You may want to read through the 5-part tutorial I wrote on using Boost.Test:
http://legalizeadulthood.wordpress.com/2009/07/04/c-unit-tests-with-boost-te...
Gennadiy, if this tutorial is any good I suggest you link to it from the Boost.Test documentation.
When I published it, Gennadiy said he would link to it, but as near as I can tell, that never happened even though I wrote the tutorial 2+ years ago. -- "The Direct3D Graphics Pipeline" -- DirectX 9 version available for download http://legalizeadulthood.wordpress.com/the-direct3d-graphics-pipeline/ Legalize Adulthood! http://legalizeadulthood.wordpress.com

[Please do not mail me a copy of your followup]
boost-users@lists.boost.org spake the secret code
I have found this to cause unnecessary methods having to be virtualized in order to replace concrete objects with test objects in a class being tested, which ended up getting quite messy at times, so I essentially had to settle for the "friend" solution.
In C++ you can either use static polymorphism or dynamic polymorphism to get substitutability of test doubles for real collaborators. I've done both, depending on the situation. If I had to choose between which I like least: virtual functions or "friend", I choose friend. I generally consider "friend" declarations a code smell that indicates poor class design. -- "The Direct3D Graphics Pipeline" -- DirectX 9 version available for download http://legalizeadulthood.wordpress.com/the-direct3d-graphics-pipeline/ Legalize Adulthood! http://legalizeadulthood.wordpress.com

[Please do not mail me a copy of your followup]
boost-users@lists.boost.org spake the secret code
I'm using the boost unit test framework to implement unit tests for code. I'm wondering if there is a best-practice for doing white-box testing/internal state inspection of types under test. My hack right now is to strip out "protected" and "private" from the header, but I hold my nose as I do that.
Thoughts/suggestions?
Hard to suggest specific actions without seeing the particular code, but my gut tells me that your class isn't designed for testability and that's why you need to "stick your hands down its pants" in order to sense what it's doing. I've often "lifted" the visibility of something in a piece of legacy code(*) in order to write a unit test for something I was changing. I might lift something from private to protected and then use a derived class in my unit test to increase the visibility to public for the purposes of the test. Sometimes you can just derive your test fixture from the system under test and that's good enough because in Boost.Test, every test case derives from the test fixture, so it can see protected items in the fixture or in the SUT. (*) I use the term "legacy code" in the Michael Feathers "Working Effectively with Legacy Code" sense: legacy code is any code that isn't covered by unit tests. Another technique I've used is to move the code to be tested to another class with public members and have the original code delegate to the new class to get things done. This is a variant of the "Replace Method with Method Object" refactoring, or sometimes it might be a full blown "Extract Class". Generally speaking, the smaller something is, the easier it is to test. While this may seem like a tautological statement, it's best to keep this in mind when dealing with a large tangled legacy mess. It's often easiest to backfill tests onto such code by extracting SUT into something smaller that's easier to test. -- "The Direct3D Graphics Pipeline" -- DirectX 9 version available for download http://legalizeadulthood.wordpress.com/the-direct3d-graphics-pipeline/ Legalize Adulthood! http://legalizeadulthood.wordpress.com

On 12/22/2011 12:25 AM, Chris Cleeland wrote:
I'm using the boost unit test framework to implement unit tests for code. I'm wondering if there is a best-practice for doing white-box testing/internal state inspection of types under test. My hack right now is to strip out "protected" and "private" from the header, but I hold my nose as I do that.
Thoughts/suggestions?
There are tricks to access private/protected members of a class without being a friend or anything like that. http://bloglitb.blogspot.com/2010/07/access-to-private-members-thats-easy.ht... Maybe that would be useful to you (haven't found a use for that yet)

On Thu, Dec 22, 2011 at 3:05 PM, Mathias Gaunard
On 12/22/2011 12:25 AM, Chris Cleeland wrote:
I'm using the boost unit test framework to implement unit tests for code. I'm wondering if there is a best-practice for doing white-box testing/internal state inspection of types under test. My hack right now is to strip out "protected" and "private" from the header, but I hold my nose as I do that.
Thoughts/suggestions?
There are tricks to access private/protected members of a class without being a friend or anything like that.
http://bloglitb.blogspot.com/2010/07/access-to-private-members-thats-easy.ht...
Maybe that would be useful to you (haven't found a use for that yet)
That's a nifty, though convoluted, trick! So far nothing I've tried has been as straightforward as #define private public I do like the idea proposed in one of the comments on that blog where somebody suggested having a public_cast<> operator to cast away protections. -- Chris Cleeland

[Please do not mail me a copy of your followup]
boost-users@lists.boost.org spake the secret code
So far nothing I've tried has been as straightforward as #define private public
....which won't work on all compilers. -- "The Direct3D Graphics Pipeline" -- DirectX 9 version available for download http://legalizeadulthood.wordpress.com/the-direct3d-graphics-pipeline/ Legalize Adulthood! http://legalizeadulthood.wordpress.com

On Thu, Dec 22, 2011 at 5:28 PM, Richard
[Please do not mail me a copy of your followup]
boost-users@lists.boost.org spake the secret code
thusly: So far nothing I've tried has been as straightforward as #define private public
....which won't work on all compilers.
You are correct, but it works on the limited set of compilers targeted by this project. I am fortunate that I do not need to do the general case. -- Chris Cleeland
participants (7)
-
Ahmed Badran
-
Chris Cleeland
-
Dave Abrahams
-
Gennadiy Rozental
-
legalize+jeeves@mail.xmission.com
-
Mathias Gaunard
-
Olaf Meeuwissen