Re: [boost] [testing] Refactoring serialization tests, and possibly others.

12 Oct 2005

      On Tue, Sep 20, 2005 at 05:35:57PM +0200, troy d. straszheim wrote:
...
It would be interesting to see exactly where all the time is going, I 
have the feeling it is mostly in the build, and that if one simply 
clumped many of these tests together, you could eliminate a lot of 
duplicated effort (template instantiations and linking).  This is 
another motivation for switching to autoregistering tests:
I tried a simple hack and got a 4x increase in speed.  Having
converted all the serialization tests to autoregistering unit tests, I
took 9 typical test modules and put them together simply by creating a
file test_many.cpp that #includes

test_array.cpp
test_binary.cpp
test_contained_class.cpp
test_deque.cpp
test_map.cpp
test_derived.cpp
test_exported.cpp
test_derived_class.cpp
test_list.cpp

and compiling that, in the thinking that most of the time spent is
taken up by compiling, linking, and calculating dependencies of the
same code.  The "before" picture:

326.17s user 153.82s system 92% cpu 8:40.03 total

the first ~30 seconds of which is bjam calculating the dependencies
for each test module.

The "after" test_many.cpp hack starts compilation after ~4 seconds of
bjam checking dependencies, with a total time of

99.52s user 41.00s system 93% cpu 2:30.51 total

This is with the portability testing stuff that started this thread in
there, but not turned on.  Memory usage on the test_many
all-in-one-hack-module isn't noticeably higher, as the test modules
typically differ by a couple hundred lines.  The time/memory is all in
the header files.

There is still a compile-link-execute cycle for each archive type plus
it's dll version, so it would seem that there's still lots of
duplicated work.  There's plenty more gains to be had.

I need a few tips so I can tinker further, if you think that's a good
idea.  I think in terms of "make", I believe this is a problem.  Lemme
know if we should move this to the boostjam list or if I'm just too
far afield.

1)  The test_many.cpp above is a hack.  It would seem cleaner to have
    the test modules listed in the Jamfile and have bjam invoke the
    compiler as

    cc test1.cpp test2.cpp ... testN.cpp

    is there a clean way to do this?

2) If I can get 1), above, this one is moot.  I want to factor out
    some convenience functions into a file test_tools.cpp and have
    this compiled only once, and have test_tools.o added to each test
    executable's link.  I've tried adding test_tools.cpp after
    $(sources) in rule run-template in lib/serialization/test/Jamfile
    like this:

  rule run-template ( test-name : sources * : requirements * ) {                     
    return [                                                                         
    run                                                                              
    <lib>../../test/build/boost_unit_test_framework                                  
    <lib>../../filesystem/build/boost_filesystem                                     
    $(sources) test_tools.cpp                                                            
    : # command                                                                      
    : #                                                                              
    : # requirements                                                                 
    std::locale-support                                                              
    toolset::require-boost-spirit-support                                            
    <borland*><*><cxxflags>"-w-8080 -w-8071 -w-8057 -w-8062 -w-8008 -w-0018 -w-8066" 
    $(requirements)                                                                  
    : # test name                                                                    
    $(test-name)                                                                     
    : # default-build                                                                
    debug                                                                            
    ] ;                                                                              
  }                                                                                  

    but this of course just recompiles test_tools.cpp for each module.
    Dunno.

3) There are "demo" tests in the Jamfile, which are pulled in and
   compiled as tests by #defining main to test_main and #including the
   .cpp from the examples directory.  This means these demos need

    <lib>../../test/build/boost_prog_exec_monitor

    not boost_unit_test_framework.  I'm stumped on this one.  How to
    split up the tests into two groups, each of which takes different
    libs?  Should one toss these from the standard testing run
    completely and just put a separate Jamfile over in examples/ ?
    You could get the same code into both the tests and the examples
    by taking the contents of main() out into some other #inlcuded
    file, but then the examples would get obscured.  Dunno.

Thanks,

-t