testing command-line interface using Boost.Test
Dear all, I've been using Boost.Test with Test First and TTD approaches to library development and am one happy hacker. All tests are integrated with our continuous integration setup so we get test reports with every build. Now I'm starting on a few command-line utilities and would like to use Boost.Test to do the same. I have something simple going now with a test fixture to help with redirecting standard input, output and error and running commands in a way that the test runner does not hang when a command-line invocation returns a failure status. The gist of my tests would be as follows when testing the GNU coreutils `true` and `false` utilities BOOST_AUTO_TEST_CASE (test_false) { BOOST_CHECK_EQUAL (EXIT_FAILURE, my_system ("false")); } BOOST_AUTO_TEST_CASE (test_true) { BOOST_CHECK_EQUAL (EXIT_SUCCESS, my_system ("true")); } where my_system() basically does explicitly what system() would do for you. See [1] for an implementation of my_system(). When using system() instead of my_system(), the test runner fails the first test incorrectly(!) and hangs. It starts but never finishes the second test. Using my_system() the first tests still incorrectly fails but now at least the second tests completes and succeeds as expected. The first test produces unknown location(0): fatal error in "test_false": child has exited; pid: 19788; uid: 1000; exit value: 1 and I assume it never gets to the comparison that BOOST_CHECK_EQUAL() is supposed to do. Any suggestions on how to get this to work? Maybe even alternative approaches to testing command-line interfaces? [1] http://www.gnu.org/software/libc/manual/html_node/Process-Creation-Example.h... Thanks in advance, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962
Gennadiy Rozental
Olaf Meeuwissen
writes: unknown location(0): fatal error in "test_false": child has exited; pid: 19788; uid: 1000; exit value: 1
I believe this was an issue with older versions of Boost.Test. Newer releases should not try to catch SIGCHLD.
Thanks for the info. I should have mentioned that I'm using Debian's testing, so my Boost is at 1.46.1. The only newer Boost version that's listed on the website is 1.47.0. The documented changes do not list anything on Boost.Test. I'll have a look at the diff between these later but in the mean time, if you have any recollection of where I should be looking, that'd be helpful. Thanks in advance, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962
Olaf Meeuwissen
Gennadiy Rozental
writes: Olaf Meeuwissen
writes: unknown location(0): fatal error in "test_false": child has exited; pid: 19788; uid: 1000; exit value: 1
I believe this was an issue with older versions of Boost.Test. Newer releases should not try to catch SIGCHLD.
Thanks for the info.
I should have mentioned that I'm using Debian's testing, so my Boost is at 1.46.1. The only newer Boost version that's listed on the website is 1.47.0. The documented changes do not list anything on Boost.Test.
Hmm. It might not have been moved into release branch. Ever since we moved to new release approach my changes started to remain unreleased. My schedule usually does not coincide with boost releases. I am working on a BIG one though. I'll be sure to release all the changes along with it. Meanwhile feel free to build trunk version of boost.test. Gennadiy
Gennadiy Rozental
Olaf Meeuwissen
writes: Gennadiy Rozental
writes: Olaf Meeuwissen
writes: unknown location(0): fatal error in "test_false": child has exited; pid: 19788; uid: 1000; exit value: 1
I believe this was an issue with older versions of Boost.Test. Newer releases should not try to catch SIGCHLD.
Thanks for the info.
I should have mentioned that I'm using Debian's testing, so my Boost is at 1.46.1. The only newer Boost version that's listed on the website is 1.47.0. The documented changes do not list anything on Boost.Test.
No changes in boost/test/ and libs/test between the above two versions.
Hmm. It might not have been moved into release branch. Ever since we moved to new release approach my changes started to remain unreleased. My schedule usually does not coincide with boost releases. I am working on a BIG one though. I'll be sure to release all the changes along with it. Meanwhile feel free to build trunk version of boost.test.
I'm afraid I don't have the time to do so and keep up with the churn. Besides, I can't really release with a test suite that depends on a development snapshot of Boost.Test. I have taken a look at the differences with 1.47.0 and that made me go back to the documentation. I've found that I can make tests for failing command-line invocation pass when I run the tests with BOOST_TEST_CATCH_SYSTEM_ERRORS=no and I should be able to do so in code with execution_monitor::p_catch_system_errors = false; My only problem now is that I have no clue how to get at or where to look for the execution_monitor instance that I would need to change. Any suggestion? My test runners all have a fairly simple main that basically just return ::boost::test_unit::unit_test_main (init_test_runner, argc, argv); Thanks in advance, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962
Olaf Meeuwissen
My only problem now is that I have no clue how to get at or where to look for the execution_monitor instance that I would need to change.
You do not need execution_monitor instance. Yo ucan spacify runtime parameter using either environment variable or CLA --catch_system_error=no Gennadiy
Gennadiy Rozental
Olaf Meeuwissen
writes: My only problem now is that I have no clue how to get at or where to look for the execution_monitor instance that I would need to change.
You do not need execution_monitor instance. Yo ucan spacify runtime parameter using either environment variable or CLA --catch_system_error=no
True, but I would like to do this only for those tests that need it. That is, only for those tests that use the test fixture that provides the my_system() functionality. So if I have struct system_fixture { int run (const char *cmd) { return my_system (cmd); } }; BOOST_FIXTURE_TEST_CASE (test_false, system_fixture) { BOOST_CHECK_EQUAL (EXIT_FAILURE, run ("false")); } BOOST_AUTO_TEST_CASE (test_true) { BOOST_CHECK_EQUAL (EXIT_SUCCESS, my_system ("true")); } I don't want system error caught for test_false but I do want them still caught for test_true as that is not supposed to cause any. The example is a bit contrived and overly simplistic but I hope you see what I'm thinking of. Thanks in advance, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962
Olaf Meeuwissen
Gennadiy Rozental
writes: Olaf Meeuwissen
writes: My only problem now is that I have no clue how to get at or where to look for the execution_monitor instance that I would need to change.
You do not need execution_monitor instance. Yo ucan spacify runtime parameter using either environment variable or CLA --catch_system_error=no
True, but I would like to do this only for those tests that need it. That is, only for those tests that use the test fixture that provides the my_system() functionality. So if I have
1) I'd personally split this module in two and run one with and another without catch_system_errors 2) You can use boost::unit_test::unit_test_monitor singleton to manipulate these options at runtime. You'll have set and revert these values in fixture Gennadiy
Gennadiy Rozental
Olaf Meeuwissen
writes: Gennadiy Rozental
writes: Olaf Meeuwissen
writes: My only problem now is that I have no clue how to get at or where to look for the execution_monitor instance that I would need to change.
You do not need execution_monitor instance. Yo ucan spacify runtime parameter using either environment variable or CLA --catch_system_error=no
True, but I would like to do this only for those tests that need it. That is, only for those tests that use the test fixture that provides the my_system() functionality. So if I have
1) I'd personally split this module in two and run one with and another without catch_system_errors
I have considered that as well, but it seems a bit awkward to split tests into two modules just because some of them for success and some of them for failure. That said, even if I were to split them into two modules, I'd still want a programmatic interface that allows me to set this in the C++ sources. I'm using automake's `check` infra-structure and specifying command-line arguments or environment variables on a per module basis is a pain.
2) You can use boost::unit_test::unit_test_monitor singleton to manipulate these options at runtime. You'll have set and revert these values in fixture
I tried this but without any effect. Either alternative below in the fixture's constructor yields the same result. boost::unit_test::unit_test_monitor.p_catch_system_errors.set (true); boost::unit_test::unit_test_monitor.p_catch_system_errors.set (false); The results I'm getting for tests for the coreutils `false` and `true` utilities look like: Running 2 test cases... Entering test suite "src::core" Entering test case "test_false" Running: false unknown location(0): fatal error in "test_false": child has exited; pid: 12822; uid: 1000; exit value: 1 process.hh(117): last checkpoint: Running: false Leaving test case "test_false" Entering test case "test_true" Running: true core.cc(40): info: check run ("true", 0) passed Leaving test case "test_true" Leaving test suite "src::core" Looks like something is catching and acting upon system errors before unit_test_monitor gets a chance to do so. I think this happens in execution_monitor::catch_signals() but am not sure. Any hints would be appreciated. FTR, still using the Debian package for 1.46.1. Thanks in advance, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962
Olaf Meeuwissen
2) You can use boost::unit_test::unit_test_monitor singleton to manipulate these options at runtime. You'll have set and revert these values in fixture
I tried this but without any effect. Either alternative below in the fixture's constructor yields the same result.
Yes. Fixture is part of test case and it's too late. You can probable switch it in "fake" test case placed before those which require catch_system_error=false Gennadiy
Gennadiy Rozental
Olaf Meeuwissen
writes: 2) You can use boost::unit_test::unit_test_monitor singleton to manipulate these options at runtime. You'll have set and revert these values in fixture
I tried this but without any effect. Either alternative below in the fixture's constructor yields the same result.
Yes. Fixture is part of test case and it's too late. You can probable switch it in "fake" test case placed before those which require catch_system_error=false
Tried that but that doesn't do the trick either. Of course, even if it worked, it would completely screw unit test statistics and it's so ugly that I be ashamed of publishing the code ;-) Also tried with a test suite shared fixture but no such luck that way either. Besides test suite shared fixtures are supplanted by the fixture of any BOOST_FIXTURE_TEST_CASEs in that suite. # I tried with regular auto-test cases too and made sure the suite's # fixture took effect. My guess is that the execution_monitor gets a go at the system error before the unit_test_monitor gets a chance and I would really have to modify the former's p_catch_system_error attribute to make it work. Unfortunately, execution_monitor is not a singleton and there does not seem to be a way to access it from something that is (like framework, for example). Thanks for the help so far. I'm still open to suggestions but it looks like I'll be going with two test runners, each with their own main so that can set BOOST_TEST_CATCH_SYSTEM_ERRORS before unit_test_main(). Sincerely, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962
Olaf Meeuwissen
My guess is that the execution_monitor gets a go at the system error before the unit_test_monitor gets a chance and I would really have to
unit_test_monitor IS execution_monitor. No the issue here is that it picks up the value every time it start new test case from the value of runtime parameter, so you really can't have it differently in a different test cases.
Thanks for the help so far. I'm still open to suggestions but it looks like I'll be going with two test runners, each with their own main so that can set BOOST_TEST_CATCH_SYSTEM_ERRORS before unit_test_main().
You might as well use CLA. It's still going to have one global value for all test cases. You can also rebuild your boost version and comment out SIGCHLD handling. Gennadiy
Gennadiy Rozental
Olaf Meeuwissen
writes: My guess is that the execution_monitor gets a go at the system error before the unit_test_monitor gets a chance and I would really have to
unit_test_monitor IS execution_monitor. No the issue here is that it picks up the value every time it start new test case from the value of runtime parameter, so you really can't have it differently in a different test cases.
I was referring to the execution_monitor instance that is used in framework::run() which appears to be another instance than the execution_monitor singleton but I readily admit that I don't really understand what's going on in there.
Thanks for the help so far. I'm still open to suggestions but it looks like I'll be going with two test runners, each with their own main so that can set BOOST_TEST_CATCH_SYSTEM_ERRORS before unit_test_main().
You might as well use CLA. It's still going to have one global value for all test cases.
I know, but it's just that the automake provided test infrastructure makes it cumbersome to differentiate the CLAs for individual runners. It is easier to set the value from within the runner then.
You can also rebuild your boost version and comment out SIGCHLD handling.
Thanks, but no thanks. Mucking around in Boost code and rebuilding is not my idea of fun. I really just want to use it as is to help get my job done. Thanks, -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962
[Please do not mail me a copy of your followup] boost-users@lists.boost.org spake the secret code <87obydywlr.fsf@avasys.jp> thusly:
Now I'm starting on a few command-line utilities and would like to use Boost.Test [...]
The gist of my tests would be as follows when testing the GNU coreutils `true` and `false` utilities
BOOST_AUTO_TEST_CASE (test_false) { BOOST_CHECK_EQUAL (EXIT_FAILURE, my_system ("false")); }
BOOST_AUTO_TEST_CASE (test_true) { BOOST_CHECK_EQUAL (EXIT_SUCCESS, my_system ("true")); }
where my_system() basically does explicitly what system() would do for you.
IMO, you are approaching this all wrong. How are your unit tests going to run fast (in <100ms) if they are doing a fork/exec/system/whatever? Plus you're introducing lots of environmental dependencies on PATH, etc., that can make your tests fragile and give false failures. What I do instead is move all the entire implementation of the utility to a static library and the main() for the utility executable is just a simple delegator that calls the function in the library that does all the work. Then you can write a test executable against the static library and test everything in your utility. Instead of writing assertions against the exepcted cout/cerr of a child process, just use a stringstream that you pass into the utility library while the main() delegator passes in cout and cerr. This approach gives fast execution of unit tests, lets me easily write assertions against expected output and error messages and avoids the fragility of depending on the external execution environment. -- "The Direct3D Graphics Pipeline" -- DirectX 9 draft available for download http://legalizeadulthood.wordpress.com/the-direct3d-graphics-pipeline/ Legalize Adulthood! http://legalizeadulthood.wordpress.com
legalize+jeeves@mail.xmission.com (Richard) writes:
boost-users@lists.boost.org spake the secret code <87obydywlr.fsf@avasys.jp> thusly:
[testing of command-line utilities]
IMO, you are approaching this all wrong. How are your unit tests going to run fast (in <100ms) if they are doing a fork/exec/system/whatever? Plus you're introducing lots of environmental dependencies on PATH, etc., that can make your tests fragile and give false failures.
What I do instead is move all the entire implementation of the utility to a static library and the main() for the utility executable is just a simple delegator that calls the function in the library that does all the work.
Then you can write a test executable against the static library and test everything in your utility. Instead of writing assertions against the exepcted cout/cerr of a child process, just use a stringstream that you pass into the utility library while the main() delegator passes in cout and cerr.
This approach gives fast execution of unit tests, lets me easily write assertions against expected output and error messages and avoids the fragility of depending on the external execution environment.
Hadn't thought of that approach. Thanks for the suggestion. -- Olaf Meeuwissen, LPIC-2 FLOSS Engineer -- AVASYS CORPORATION FSF Associate Member #1962 Help support software freedom http://www.fsf.org/jf?referrer=1962
participants (3)
-
Gennadiy Rozental -
legalize+jeeves＠mail.xmission.com -
Olaf Meeuwissen