Profiling template instantiations
AMDG Over the last few days, I've been working on a tool that counts the number of class template instantiations in a translation unit. To use it from a jamfile import profile_templates : : directory of the tool ; template-profile result : some_file.cpp : <include>some_directories ; If everything works, this will create a file with the extension .tp which will contain the number of times each class template was instantiated, with the most common templates at the top. I've put a zip of a rough version in the vault. http://www.boost-consulting.com/vault/index.php?action=downloadfile&filename=profile_templates.zip&directory=Tools& Caveats: I've only tested with Visual C++ 9.0 and gcc 3.4.4 on cygwin so far. Other compilers won't work, yet. Other versions of these compilers may or may not work. It may be necessary to add link=static to the bjam command line. In Christ, Steven Watanabe
Steven wrote:
Over the last few days, I've been working on a tool that counts the number of class template instantiations in a translation unit.
Ingenious, you are automatically inserting a compile time error into every basic block of the template code to be profiled, compiling the translation unit and counting how many error are generated for each template to get a count of how many times the compiler tries (and fails) to instantiate the template. There may be a more direct way to extract this information with VTune related profiling features provided in icc. I'll follow up and let you know. Luke
AMDG Simonson, Lucanus J wrote:
Ingenious, you are automatically inserting a compile time error into every basic block of the template code to be profiled, compiling the translation unit and counting how many error are generated for each template to get a count of how many times the compiler tries (and fails) to instantiate the template.
Actually it only generates warnings. Otherwise, the compiler is liable to stop before it finishes compiling everything.
There may be a more direct way to extract this information with VTune related profiling features provided in icc. I'll follow up and let you know.
Ok. It would be great if compilers supported this directly. In Christ, Steven Watanabe
Steven Watanabe wrote:
AMDG
Over the last few days, I've been working on a tool that counts the number of class template instantiations in a translation unit.
To use it from a jamfile
import profile_templates : : directory of the tool ; template-profile result : some_file.cpp : <include>some_directories ;
If everything works, this will create a file with the extension .tp which will contain the number of times each class template was instantiated, with the most common templates at the top.
I've put a zip of a rough version in the vault.
Steven, this sounds really interesting! I tried to test it out on your test program but get a lot of BBv2 errors, first off I had to change: import profile_templates ; to import ../profile_templates ; in the test/Jamfile.v2 otherwise I get an error from BB that it can't find "profile_templates". But then I see: c:/data/boost/boost/trunk/libs/profile_templates/test\../profile_templates.jam:32: in load *** argument error * rule path.make ( native ) * called with: ( ) * missing argument native c:/data/boost/boost/trunk/tools/build/v2/util\path.jam:44:see definition of rule 'make' being called c:/data/boost/boost/trunk/tools/build/v2/kernel\modules.jam:267: in modules.import c:/data/boost/boost/trunk/tools/build/v2/build\project.jam:847: in import Jamfile.v2:10: in modules.load c:/data/boost/boost/trunk/tools/build/v2/build\project.jam:307: in load-jamfile c:/data/boost/boost/trunk/tools/build/v2/build\project.jam:63: in load c:/data/boost/boost/trunk/tools/build/v2/build\project.jam:167: in project.find c:/data/boost/boost/trunk/tools/build/v2\build-system.jam:521: in load c:\data\boost\boost\trunk\tools\build\v2/kernel\modules.jam:267: in import c:\data\boost\boost\trunk\tools\build\v2/kernel/bootstrap.jam:128: in boost-build c:\data\boost\boost\trunk\boost-build.jam:11: in module scope This kind of error usually happens when some variable is not defined, but I can't spot the problem off hand, any ideas? Anything in user-config.jam that I should have set up? Cheers, John.
AMDG John Maddock wrote:
Steven, this sounds really interesting!
I tried to test it out on your test program but get a lot of BBv2 errors, first off I had to change:
import profile_templates ;
to
import ../profile_templates ;
in the test/Jamfile.v2 otherwise I get an error from BB that it can't find "profile_templates".
Try import profile_templates : : .. ; I think that it fails because I removed the Jamfile from the top level directory which contains profile_templates.jam In Christ, Steven Watanabe
Steven Watanabe wrote:
Try
import profile_templates : : .. ;
I think that it fails because I removed the Jamfile from the top level directory which contains profile_templates.jam
That results in the even more cryptic: c:/data/boost/boost/trunk/tools/build/v2/kernel\modules.jam:231: in modules.import rule errors.error unknown in module modules. c:/data/boost/boost/trunk/tools/build/v2/build\project.jam:847: in import Jamfile.v2:10: in modules.load c:/data/boost/boost/trunk/tools/build/v2/build\project.jam:307: in load-jamfile c:/data/boost/boost/trunk/tools/build/v2/build\project.jam:63: in load c:/data/boost/boost/trunk/tools/build/v2/build\project.jam:167: in project.find c:/data/boost/boost/trunk/tools/build/v2\build-system.jam:521: in load c:\data\boost\boost\trunk\tools\build\v2/kernel\modules.jam:267: in import c:\data\boost\boost\trunk\tools\build\v2/kernel/bootstrap.jam:128: in boost-build c:\data\boost\boost\trunk\boost-build.jam:11: in module scope John.
AMDG John Maddock wrote:
Steven Watanabe wrote:
Try
import profile_templates : : .. ;
I think that it fails because I removed the Jamfile from the top level directory which contains profile_templates.jam
That results in the even more cryptic:
Oops. I was looking at modules.load instead of import. This seems to work: import modules ; modules.load profile_templates : ../profile_templates.jam ; In Christ, Steven Watanabe
Steven Watanabe wrote:
This seems to work:
import modules ; modules.load profile_templates : ../profile_templates.jam ;
OK I got the Jamfile building OK, but now I see an error in the postprocess step: profile_templates.process_raw_profile ..\..\..\bin.v2\libs\profile_templates\test\msvc-8.0\debug\threading-multi\test.tp "..\..\..\bin.v2\libs\profile_templates\src\msvc-8.0\debug\threading-multi\postprocess.exe" <"..\..\..\bin.v2\libs\profile_templates\test\msvc-8.0\debug\threading-multi\test.rtp"
"..\..\..\bin.v2\libs\profile_templates\test\msvc-8.0\debug\threading-multi\test.tp"
...failed profile_templates.process_raw_profile ..\..\..\bin.v2\libs\profile_templates\test\msvc-8.0\debug\threading-multi\test.tp... ...removing ..\..\..\bin.v2\libs\profile_templates\test\msvc-8.0\debug\threading-multi\test.tp A couple of other observations: * This takes an absolute age to run... isn't there a better way than processing all the Boost headers separately? Maybe pre-process the source, then insert the special markup and compile? * If a failure occurs and I re-run the Jamfile, then it just goes off processing all the Boost headers all over again, so there's some dependency analysis missing somewhere :-( Cheers, John.
AMDG John Maddock wrote:
Steven Watanabe wrote:
This seems to work:
import modules ; modules.load profile_templates : ../profile_templates.jam ;
OK I got the Jamfile building OK, but now I see an error in the postprocess step:
profile_templates.process_raw_profile ..\..\..\bin.v2\libs\profile_templates\test\msvc-8.0\debug\threading-multi\test.tp
"..\..\..\bin.v2\libs\profile_templates\src\msvc-8.0\debug\threading-multi\postprocess.exe" <"..\..\..\bin.v2\libs\profile_templates\test\msvc-8.0\debug\threading-multi\test.rtp"
"..\..\..\bin.v2\libs\profile_templates\test\msvc-8.0\debug\threading-multi\test.tp"
...failed profile_templates.process_raw_profile ..\..\..\bin.v2\libs\profile_templates\test\msvc-8.0\debug\threading-multi\test.tp... ...removing ..\..\..\bin.v2\libs\profile_templates\test\msvc-8.0\debug\threading-multi\test.tp
Add link=static. (It isn't finding the regex dll).
A couple of other observations:
* This takes an absolute age to run... isn't there a better way than processing all the Boost headers separately? Maybe pre-process the source, then insert the special markup and compile?
I was planning to see if I can hijack the scanner to avoid copying headers that aren't used.
* If a failure occurs and I re-run the Jamfile, then it just goes off processing all the Boost headers all over again, so there's some dependency analysis missing somewhere :-(
I've noticed that, too. I don't understand it yet. In Christ, Steven Watanabe
AMDG John Maddock wrote:
* If a failure occurs and I re-run the Jamfile, then it just goes off processing all the Boost headers all over again, so there's some dependency analysis missing somewhere :-(
Ok. I've figured this one out. The headers are being copied into a subdirectory of one of the <include> directories. When I do the glob-tree it finds the new headers as well. I'll put a new version with these fixes in the sandbox. In Christ, Steven Watanabe
AMDG John Maddock wrote:
* This takes an absolute age to run... isn't there a better way than processing all the Boost headers separately?
I think this is fixed now. Only the headers that are #included will be processed.
* If a failure occurs and I re-run the Jamfile, then it just goes off processing all the Boost headers all over again, so there's some dependency analysis missing somewhere :-(
This should be fixed. The code is in the sandbox http://svn.boost.org/svn/boost/sandbox/tools/profile_templates In Christ, Steven Watanabe
Steven strikes again! This is awesome work. I did have a problem with the post-processor with GCC 4.2.1. I had to use this regex string to get everything working (patch attached): boost::regex warning_message("(.*):\\d*: warning: division by zero in 'template_profiler_value / 0'"); Note the different quotes and the bit added for he line number. After that, it worked fine. Zach
AMDG Zach Laine wrote:
Steven strikes again! This is awesome work.
I did have a problem with the post-processor with GCC 4.2.1. I had to use this regex string to get everything working (patch attached):
boost::regex warning_message("(.*):\\d*: warning: division by zero in 'template_profiler_value / 0'");
Note the different quotes and the bit added for he line number. After that, it worked fine.
The extra stuff for the line number should be unnecessary because it will be matched by the (.*) I wanted to capture the line number in addition to the file name. I accidentally committed the fixes I've been making to my local repository. They should be in the sandbox now. I made the matching fuzzier by using "." Can you check that it actually works. In Christ, Steven Watanabe
On 05/10/08 19:06, Steven Watanabe wrote: [snip]
The code is in the sandbox http://svn.boost.org/svn/boost/sandbox/tools/profile_templates
Hi Steven, Thanks for this tool; however... I wanted to use another test driver instead of test.cpp. I thought just adding another target to the Jamfile.v2 would work: -{--Jamfile.v2-- template-profile test : test.cpp : <include>test_dir ; template-profile and_seq_find_if : and_seq_find_if.cpp : <include> test_dir ; -}--Jamfile.v2-- However, I get: ={--bjam output-- bjam link=static test /home/evansl/prog_dev/boost-svn/ro/boost-trunk/tools/build/v2/util/path.jam:528: in make-UNIX from module path error: Empty path passed to 'make-UNIX' /home/evansl/prog_dev/boost-svn/ro/boost-trunk/tools/build/v2/util/path.jam:44: in path.make from module path /home/evansl/prog_dev/boost-svn/ro/boost-trunk/tools/build/v2/build/property.jam:455: in property.translate-paths ... -}--bjam output-- and_seq_find_if.cpp is attached. How can one profile something besides test.cpp? -regards, Larrry
On 02/12/09 05:55, Larry Evans wrote:
On 05/10/08 19:06, Steven Watanabe wrote: [snip]
The code is in the sandbox http://svn.boost.org/svn/boost/sandbox/tools/profile_templates
Hi Steven,
Thanks for this tool; however...
I wanted to use another test driver instead of test.cpp. I thought just adding another target to the Jamfile.v2 would work:
-{--Jamfile.v2--
template-profile test : test.cpp : <include>test_dir ;
template-profile and_seq_find_if : and_seq_find_if.cpp : <include> test_dir ;
-}--Jamfile.v2--
Should have been <include>test_dir instead of having space there. Sorry for noise.
On 02/12/09 06:07, Larry Evans wrote:
On 02/12/09 05:55, Larry Evans wrote:
On 05/10/08 19:06, Steven Watanabe wrote: [snip]
The code is in the sandbox http://svn.boost.org/svn/boost/sandbox/tools/profile_templates
Hi Steven,
Thanks for this tool; however...
I wanted to use another test driver instead of test.cpp. I thought just adding another target to the Jamfile.v2 would work:
-{--Jamfile.v2--
template-profile test : test.cpp : <include>test_dir ;
template-profile and_seq_find_if : and_seq_find_if.cpp : <include> test_dir ;
-}--Jamfile.v2--
Should have been <include>test_dir instead of having space there.
Sorry for noise.
However, now I'm getting an error in the rtp file: from and_seq_find_if.cpp:2: /usr/include/c++/4.2/bits/stl_relops.h:74: error: redefinition of 'struct std::template_profiler_test' /usr/include/c++/4.2/x86_64-linux-gnu/bits/c++config.h:149: error: previous definition of 'struct std::template_profiler_test' Should I be concerned? stl_relops.h:74 is just: _GLIBCXX_BEGIN_NAMESPACE(std) -regards, Larry
On 02/12/09 07:50, Larry Evans wrote:
On 02/12/09 06:07, Larry Evans wrote:
On 02/12/09 05:55, Larry Evans wrote: [snip]
-{--Jamfile.v2--
template-profile test : test.cpp : <include>test_dir ;
template-profile and_seq_find_if : and_seq_find_if.cpp : <include> test_dir ;
-}--Jamfile.v2--
Should have been <include>test_dir instead of having space there.
Sorry for noise.
However, now I'm getting an error in the rtp file:
from and_seq_find_if.cpp:2: /usr/include/c++/4.2/bits/stl_relops.h:74: error: redefinition of 'struct std::template_profiler_test' /usr/include/c++/4.2/x86_64-linux-gnu/bits/c++config.h:149: error: previous definition of 'struct std::template_profiler_test'
Should I be concerned? stl_relops.h:74 is just:
_GLIBCXX_BEGIN_NAMESPACE(std)
The .cpp was modified by uncommenting the #include <and_seq_instances.hpp> and that file is attached.
AMDG Larry Evans wrote:
However, now I'm getting an error in the rtp file:
from and_seq_find_if.cpp:2: /usr/include/c++/4.2/bits/stl_relops.h:74: error: redefinition of 'struct std::template_profiler_test' /usr/include/c++/4.2/x86_64-linux-gnu/bits/c++config.h:149: error: previous definition of 'struct std::template_profiler_test'
Should I be concerned? stl_relops.h:74 is just:
_GLIBCXX_BEGIN_NAMESPACE(std)
The .cpp was modified by uncommenting the #include <and_seq_instances.hpp> and that file is attached.
It sounds like the perl script is mishandling namespaces. Can you post that line from the preprocessed source? (Should be and_seq_find_if.cpp in the build directory) In Christ, Steven Watanabe
On 02/12/09 11:01, Steven Watanabe wrote:
AMDG
Larry Evans wrote:
However, now I'm getting an error in the rtp file:
from and_seq_find_if.cpp:2: /usr/include/c++/4.2/bits/stl_relops.h:74: error: redefinition of 'struct std::template_profiler_test' /usr/include/c++/4.2/x86_64-linux-gnu/bits/c++config.h:149: error: previous definition of 'struct std::template_profiler_test'
Should I be concerned? stl_relops.h:74 is just:
_GLIBCXX_BEGIN_NAMESPACE(std)
The .cpp was modified by uncommenting the #include <and_seq_instances.hpp> and that file is attached.
It sounds like the perl script is mishandling namespaces. Can you post that line from the preprocessed source? (Should be and_seq_find_if.cpp in the build directory) [snip] Sure, here's that line and some context:
namespace std __attribute__ ((__visibility__ ("default"))) { PROFILE_TRACER(); # 161 "/usr/include/c++/4.2/x86_64-linux-gnu/bits/c++config.h" 3 } # 66 "/usr/include/c++/4.2/utility" 2 3 # 1 "/usr/include/c++/4.2/bits/stl_relops.h" 1 3 # 74 "/usr/include/c++/4.2/bits/stl_relops.h" 3 namespace std __attribute__ ((__visibility__ ("default"))) { PROFILE_TRACER(); namespace rel_ops { # 90 "/usr/include/c++/4.2/bits/stl_relops.h" 3 template <class _Tp> inline bool operator!=(const _Tp& __x, const _Tp& __y) { PROFILE_TRACER(); return !(__x == __y); } #
AMDG Larry Evans wrote:
However, now I'm getting an error in the rtp file:
from and_seq_find_if.cpp:2: /usr/include/c++/4.2/bits/stl_relops.h:74: error: redefinition of 'struct std::template_profiler_test' /usr/include/c++/4.2/x86_64-linux-gnu/bits/c++config.h:149: error: previous definition of 'struct std::template_profiler_test'
Should I be concerned? stl_relops.h:74 is just:
_GLIBCXX_BEGIN_NAMESPACE(std)
Sure, here's that line and some context:
namespace std __attribute__ ((__visibility__ ("default"))) { PROFILE_TRACER();
Thanks. Fixed in [51224] In Christ, Steven Watanabe
participants (6)
-
John Maddock -
Larry Evans -
Matt Calabrese -
Simonson, Lucanus J -
Steven Watanabe -
Zach Laine