Building boost faster: /MP
Hello, I'm sure most of you are aware of the /MP flag introduced in MSVC9. I know this flag works fine when set through Visual Studio 9, however, I'm not sure if it works when calling cl.exe directly. Since it's just a compiler flag I imagine it does. It would be great to see support for this flag in CMake and/or bjam to help build boost faster when using this particular compiler. Any thoughts on this? Sincerely, Robert Dailey
Forgot to mention one thing. I have some CMake code that will add support
for this. It's a simple feature that adds a cache variable only if you are
using generator that utilizes MSVC9. This cache variable is an integral
value and starts off with a default of 1, which translates to /MP1. If you
have an 8 core machine, you can set it to 8 and it will use /MP8. I'd be
willing to provide this implementation to the boost community if you guys
would like to have it.
Sincerely,
Robert Dailey
On Wed, Jul 15, 2009 at 1:17 PM, Robert Dailey
Hello, I'm sure most of you are aware of the /MP flag introduced in MSVC9. I know this flag works fine when set through Visual Studio 9, however, I'm not sure if it works when calling cl.exe directly. Since it's just a compiler flag I imagine it does. It would be great to see support for this flag in CMake and/or bjam to help build boost faster when using this particular compiler. Any thoughts on this?
Sincerely, Robert Dailey
Robert Dailey wrote:
Hello, I'm sure most of you are aware of the /MP flag introduced in MSVC9. I know this flag works fine when set through Visual Studio 9, however, I'm not sure if it works when calling cl.exe directly. Since it's just a compiler flag I imagine it does. It would be great to see support for this flag in CMake and/or bjam to help build boost faster when using this particular compiler. Any thoughts on this?
If this is a compiler flag, then can you try full build with cxxflags=-MP and then without and report the timings? If this is a magic way to make builds faster, it can be enabled by default. - Volodya
If this is a compiler flag, then can you try full build with
cxxflags=-MP
and then without and report the timings? If this is a magic way to make builds faster, it can be enabled by default.
It is a compiler flag, and it really improves build times in milticore environment, but it's somewhat limited: http://msdn.microsoft.com/en-us/library/bb385193.aspx
Igor R wrote:
If this is a compiler flag, then can you try full build with
cxxflags=-MP
and then without and report the timings? If this is a magic way to make builds faster, it can be enabled by default.
It is a compiler flag, and it really improves build times in milticore environment, but it's somewhat limited: http://msdn.microsoft.com/en-us/library/bb385193.aspx
But /MP only affects cl.exe when you give it more than one source file, and AFAIK bjam still spawns one cl.exe per source file. (Which is a pity. The way cl.exe works, I believe it actually improves compile time to give it multiple files to work on even without /MP.) Try the -j option to bjam instead, to make it spawn multiple cl.exe processes. Sebastian
On Wed, Jul 15, 2009 at 2:26 PM, Igor R
If this is a compiler flag, then can you try full build with
cxxflags=-MP
and then without and report the timings? If this is a magic way to make builds faster, it can be enabled by default.
It is a compiler flag, and it really improves build times in milticore environment, but it's somewhat limited: http://msdn.microsoft.com/en-us/library/bb385193.aspx
Yes, it is limited in that it does not support a couple of other flags simultaneously. However, if you think about it, /MP is a replacement of /Gm (Incremental Build) and /Yc (Precompiled Headers). When you can build 4 or 8 source files simultaneously, that's a lot better than these two features, which do not speed up the build quite as much as this feature does. Additionally, this feature provides an increase in build speed without the disadvantages of using PCH and other alternatives. I think that /MP should be made a default, but give the user the ability to disable it JUST in case they need to use any of the documented incompatible language features. That's just my two cents.
Yes, it is limited in that it does not support a couple of other flags simultaneously. However, if you think about it, /MP is a replacement of /Gm (Incremental Build) and /Yc (Precompiled Headers).
IMHO, it's much more disappointing that /MP has almost no effect with /LTCG (and /GL). With these switches no much work is done in "compile" time - actual compilation occurs in the link-time, in one single process.
On Wed, Jul 15, 2009 at 3:37 PM, Igor R
Yes, it is limited in that it does not support a couple of other flags simultaneously. However, if you think about it, /MP is a replacement of /Gm (Incremental Build) and /Yc (Precompiled Headers).
IMHO, it's much more disappointing that /MP has almost no effect with /LTCG (and /GL). With these switches no much work is done in "compile" time - actual compilation occurs in the link-time, in one single process.
/MP is most valuable to the preprocessor, especially when using Boost in your project. When you use boost in almost every single file, on a very powerful machine you could still end up waiting 1-3 seconds for each file to compile. This is, of course, subject to what part(s) of boost you're using. In short, I think /MP introduces extreme performance improvements at most, and no benefit at all at least. It might not be perfect, but it's a very good first step for Microsoft.
/MP is most valuable to the preprocessor, especially when using Boost in your project. When you use boost in >almost every single file, on a very
powerful machine you could still end up waiting 1-3 seconds for each file
to compile. This is, of course, subject to what part(s) of boost you're using. In short, I think /MP >introduces extreme performance improvements at most, and no benefit at all at least. It might not be perfect, >but it's a very good first step for Microsoft.
My main build machine is high-spec. I routinely use bjam -j N, but find that in the code that uses Boost most intensively, CPU is no longer the main determinant of compile time. Some source files take ~1GB of memory to compile - so trying to do several such files in parallel causes the meager 3GB or so of memory to be exhausted with all the problems that that entails. My /next/ build machine is not going to be constrained by the (Windows?) 32bit 4GB memory barrier!
On Wed, Jul 15, 2009 at 4:16 PM, Pete Bartlett
My main build machine is high-spec. I routinely use bjam -j N, but find that in the code that uses Boost most intensively, CPU is no longer the main determinant of compile time. Some source files take ~1GB of memory to compile - so trying to do several such files in parallel causes the meager 3GB or so of memory to be exhausted with all the problems that that entails. My /next/ build machine is not going to be constrained by the (Windows?) 32bit 4GB memory barrier!
At the risk of going off-topic, you can still use >4GB on a 32-bit Windows machine. The only catch is a single process can't use more than 2GB (or 3GB with some special work). -- Cory Nelson http://int64.org
At the risk of going off-topic, you can still use >4GB on a 32-bit Windows machine. The only catch is a single process can't use more than 2GB (or 3GB with some special work).
http://msdn.microsoft.com/en-us/library/aa366778(VS.85).aspx
participants (6)
-
Cory Nelson -
Igor R -
Pete Bartlett -
Robert Dailey -
Sebastian Redl -
Vladimir Prus