On Wed, Oct 12, 2005 at 10:14:34AM -0500, Rene Rivera wrote:
troy d. straszheim wrote:
same code. The "before" picture:
326.17s user 153.82s system 92% cpu 8:40.03 total
the first ~30 seconds of which is bjam calculating the dependencies for each test module.
To add to what Dave said... We've been personally bitten by assuming that bjam is slow in certain places. Only to humiliate ourselves after looking at the profile data :-)
Sorry, I had no idea the phrase "bjam calculating dependencies" was a touchy subject. Please replace with "build system doing whatever it does before the first compilation starts, as timed by looking at my watch". I only understand the before/after picture to mean that bjam, the compiler and linker are all being told to repeat work unnecessarily; I see low-hanging fruit, and I want to pick and eat it. :) I'm *not* taking (and won't take) veiled cheap shots. I've pasted the full profiling info at the bottom of this mail. Be interested to hear what y'all make of it.
I'm assuming you deleted the results from the previous run, right?
Heh, yeah. :) I watched the compilations go by, and I double checked that ccache was turned off.
Yea... You just put all the sources into the bjam target. For example:
run test1.cpp test2.cpp test3.cpp : : : std::locale-support : my_test : debug ;
I just tried this, and test1, test2, test3 each get compiled separately. Here's the thing: the serialization unit tests each precompile to roughly 50k lines, of which at most 200 actually differ from test-to-test. When you then consider how much duplicated work there is for the compiler and linker, after all the templates in those 49,800 lines are instanatiated... there's your speedup. But one can just use the test_many.cpp which #includes all the other .cpps. Whatever you guys are comfortable with.
3) There are "demo" tests in the Jamfile, which are pulled in and compiled as tests by #defining main to test_main and #including the .cpp from the examples directory. This means these demos need
<lib>../../test/build/boost_prog_exec_monitor
not boost_unit_test_framework. I'm stumped on this one. How to split up the tests into two groups, each of which takes different libs? Should one toss these from the standard testing run completely and just put a separate Jamfile over in examples/ ? You could get the same code into both the tests and the examples by taking the contents of main() out into some other #inlcuded file, but then the examples would get obscured. Dunno.
I don't understand your confusion. You just do exactly what you said you want to do and refer to <lib>../../test/build/boost_prog_exec_monitor in the sources for those example/tests.
<lib>../../test/build/boost_prog_exec_monitor is fed directly to rule "run" inside rule run-template, which is invoked from run-(w)invoke, which is called sometimes from test-bsl-run_files, sometimes from test-bsl-run. But after this change, some files currently fed to test-bsl-run (namely those containing the string "demo") will need the exec_monitor, and some (all the others) will need the unit_test_framework. I could copy/past the whole group of rules and make some of them run-template-exec-monitor and some run-template-unit-test-framework, but that seemed ugly. Hopefully I've been clear, I have a lot of balls in the air. Thanks for your help, -t Here's that profiling info: ****** before ****** gross net # entries name 41 41 27247 MATCH 453 453 1329 PWD 0 0 1 find-to-root 6 6 96 GLOB 3454 2 1 boost-build 0 0 1 _poke 0 0 98 set-as-singleton 2 2 3113 DEPENDS 0 0 292 NOTFILE 0 0 91 ALWAYS 3 3 429 feature 3 0 312 free-feature 0 0 4 path-feature 0 0 1 dependency-feature 1 0 5 variant 980 398 5277 select-properties 30 29 7913 relevant-features 72 19 97 include-tools 47 27 5141 flags 104 92 36819 split-path 12 10 908 split 452 186 131436 get-values 382 382 152093 get-properties 8 1 1940 is-subset 93 93 39106 intersection 102 102 52329 unique 45 45 5277 normalize-properties 151 39 6055 sort 112 112 30312 bubble 112 112 28217 difference 0 0 1 declare-target-type 3451 6 615 load-jamfiles 671 3 1233 subproject 665 29 1234 SubDir 10 10 2463 FSubDir 262 44 5855 root-paths 283 20 9615 tokens-to-simple-path 309 169 15183 simplify-path-tokens 51 51 15183 reverse 89 89 15183 strip-initial 67 20 12173 FDirName 71 71 17277 join-path 0 0 1 project-root 3 3 1858 FGrist 16 6 1234 adjust-path-globals 0 0 1 project 0 0 1 path-global 2 0 5 import 0 0 3 declare-build-fail-test 0 0 3 declare-build-succeed-test 3448 0 18 test-bsl-run_files 3448 0 90 test-bsl-run_archive 1851 1 54 run-invoke 0 0 276 in-invocation-subdir 1 1 360 ECHO 3446 0 180 run-template 3446 2 180 run 3444 5 180 boost-test 1597 0 36 run-winvoke 0 0 2 test-suite 1 1 652 type-DEPENDS 0 0 90 get-library-name 3403 4 104 declare-local-target 0 0 104 expand-target-names 1 1 922 FGristFiles 13 3 194 declare-basic-target 6 3 284 expand-source-names 10 9 2784 ungrist 25 25 5990 select-gristed 1 1 198 declare-fake-targets 3387 1 90 main-target 2078 43 1318 expand-target-subvariants 6 6 1318 get-BUILD 18 18 7204 select-ungristed 1667 78 2636 expand-build-request 229 94 7640 segregate-free-properties 20 14 2636 report-free-property-conflicts 35 10 2636 remove-incompatible-builds 48 21 2636 segregate-overrides 29 29 5272 feature-default 6 6 5272 replace-properties 84 20 3864 multiply-property-sets 33 16 8416 distribute-feature 168 44 2646 fixup-path-properties 39 22 2636 make-path-property-sets 11 6 2636 remove-default-properties 2 2 2636 ungrist-properties 42 29 2636 split-path-at-grist 276 13 2186 toolset::requirements 263 21 2186 impose-requirements 1 1 598 std::locale-support 0 0 598 force-NT-static-link 0 0 598 toolset::require-boost-spirit-support 480 4 182 dependent-include 63 11 1228 target-path-of 4 4 2476 directory-of 24 3 1238 top-relative-tokens 4 4 1516 join 1 0 1228 protect-subproject 1 1 1228 protect-subdir 751 7 1228 enter-subproject 62 7 2546 relative-path 0 0 2 template 1 0 6 lib 0 0 10 template-modifier 26 12 624 target-id-of 0 0 6 dll 0 0 2 unless 0 0 3 install 0 0 3 stage 30 7 1318 split-target-subvariant 3025 29 704 subvariant-target 2 2 704 get-tag-features 52 26 1408 rename-target 2 2 704 FAppendSuffix 20 19 463 set-target-variables 2664 3 182 generate-dependencies 2638 8 364 link-libraries 2084 32 614 find-compatible-subvariant 3 2 738 is-link-compatible 6 0 4 library-file 22 1 96 Objects 0 0 187 object-name 21 2 187 Object 7 0 553 MakeLocate 6 6 553 MkDir 0 0 304 NOUPDATE 2 0 187 C++ 2 2 187 Cc-platform-specifics 0 0 187 C++-action 0 0 4 LibraryFromObjects 0 0 4 Archive 0 0 4 Archive-action 0 0 4 Ranlib 0 0 4 Ranlib-action 37 14 868 common-variant-tag 12 0 181 depend-on-static 13 10 364 depend-on-libs 3 0 183 depend-on-shared 1361 0 90 build-test 0 0 180 RMOLD 1354 5 90 run-test 20 1 90 executable-file 6 1 92 main-from-objects 3 2 90 Link-EXE 1 0 92 .do-link 1 0 92 Link-action 0 0 92 Chmod 0 0 90 test-executable(EXE) 1 1 90 capture-run-output 4 4 1822 INCLUDES 0 0 90 succeeded-test-file 36 4 90 dump-test 1 1 270 get-var-value 0 0 299 toolset::require-shared-libraries-support 4 0 2 dll-files 0 0 2 Link-DLL 0 0 180 toolset::require-wide-char-io-support 51 40 1732 HdrRule 3 3 1732 NOCARE 4 4 2029 remember-binding ****** after ****** gross net # entries name 15 15 7289 MATCH 54 54 161 PWD 0 0 1 find-to-root 0 0 16 GLOB 369 2 1 boost-build 0 0 1 _poke 0 0 18 set-as-singleton 0 0 793 DEPENDS 0 0 52 NOTFILE 0 0 11 ALWAYS 1 1 109 feature 1 0 72 free-feature 0 0 4 path-feature 0 0 1 dependency-feature 1 0 5 variant 95 36 605 select-properties 8 7 905 relevant-features 11 3 17 include-tools 7 3 901 flags 20 14 4451 split-path 5 2 196 split 42 20 15140 get-values 30 30 17589 get-properties 2 1 340 is-subset 10 10 4586 intersection 14 14 6081 unique 3 3 605 normalize-properties 18 4 695 sort 14 14 3496 bubble 10 10 3241 difference 0 0 1 declare-target-type 366 0 71 load-jamfiles 71 1 145 subproject 70 1 146 SubDir 1 1 287 FSubDir 25 1 703 root-paths 34 1 1119 tokens-to-simple-path 37 25 1791 simplify-path-tokens 3 3 1791 reverse 9 9 1791 strip-initial 6 1 1421 FDirName 7 7 2013 join-path 1 0 1 project-root 0 0 226 FGrist 1 0 146 adjust-path-globals 0 0 1 project 0 0 1 path-global 2 0 5 import 0 0 3 declare-build-fail-test 0 0 3 declare-build-succeed-test 363 0 2 test-bsl-run_files 363 0 10 test-bsl-run_archive 190 0 6 run-invoke 0 0 36 in-invocation-subdir 0 0 40 ECHO 363 0 20 run-template 363 0 20 run 363 0 20 boost-test 173 0 4 run-winvoke 0 0 2 test-suite 0 0 252 type-DEPENDS 0 0 10 get-library-name 360 1 24 declare-local-target 0 0 24 expand-target-names 0 0 138 FGristFiles 4 1 34 declare-basic-target 2 1 44 expand-source-names 2 2 480 ungrist 1 1 694 select-gristed 0 0 38 declare-fake-targets 356 0 10 main-target 208 2 150 expand-target-subvariants 0 0 150 get-BUILD 0 0 820 select-ungristed 167 10 300 expand-build-request 25 10 872 segregate-free-properties 3 0 300 report-free-property-conflicts 4 1 300 remove-incompatible-builds 2 1 300 segregate-overrides 4 4 600 feature-default 1 1 600 replace-properties 9 2 440 multiply-property-sets 4 0 960 distribute-feature 17 6 310 fixup-path-properties 6 2 300 make-path-property-sets 3 2 300 remove-default-properties 0 0 300 ungrist-properties 9 6 300 split-path-at-grist 27 0 250 toolset::requirements 27 3 250 impose-requirements 0 0 70 std::locale-support 0 0 70 force-NT-static-link 0 0 70 toolset::require-boost-spirit-support 52 0 22 dependent-include 3 0 140 target-path-of 0 0 300 directory-of 1 0 150 top-relative-tokens 2 2 188 join 0 0 140 protect-subproject 0 0 140 protect-subdir 83 2 140 enter-subproject 11 0 290 relative-path 0 0 2 template 0 0 6 lib 0 0 10 template-modifier 2 0 80 target-id-of 1 0 6 dll 0 0 2 unless 0 0 3 install 0 0 3 stage 4 0 150 split-target-subvariant 315 1 80 subvariant-target 1 0 80 get-tag-features 6 2 160 rename-target 0 0 80 FAppendSuffix 7 7 143 set-target-variables 280 0 22 generate-dependencies 277 1 44 link-libraries 206 1 70 find-compatible-subvariant 1 0 82 is-link-compatible 4 0 4 library-file 8 1 16 Objects 0 0 107 object-name 7 2 107 Object 1 0 153 MakeLocate 1 1 153 MkDir 0 0 64 NOUPDATE 0 0 107 C++ 0 0 107 Cc-platform-specifics 0 0 107 C++-action 0 0 4 LibraryFromObjects 0 0 4 Archive 0 0 4 Archive-action 0 0 4 Ranlib 0 0 4 Ranlib-action 6 2 100 common-variant-tag 1 0 21 depend-on-static 2 1 44 depend-on-libs 2 0 23 depend-on-shared 133 0 10 build-test 0 0 20 RMOLD 132 0 10 run-test 2 0 10 executable-file 2 0 12 main-from-objects 0 0 10 Link-EXE 1 1 12 .do-link 0 0 12 Link-action 0 0 12 Chmod 0 0 10 test-executable(EXE) 0 0 10 capture-run-output 1 1 1743 INCLUDES 0 0 10 succeeded-test-file 3 1 10 dump-test 0 0 30 get-var-value 0 0 35 toolset::require-shared-libraries-support 4 0 2 dll-files 1 0 2 Link-DLL 1 1 20 toolset::require-wide-char-io-support 47 40 1733 HdrRule 0 0 1733 NOCARE 1 1 2038 remember-binding