[build] Headers rule does both too much and too little
Folks I don't think the headers rule is working very well at present - it seems to both do too much and too little depending on the context: Too much: If I build the type_traits tests so everything is up to date, and then touch say type_with_alignment.hpp, then in libs/type_traits/test do: bjam add_const_test And for some strange reason bjam thinks there's a dependency on type_with_alignment.hpp from add_const_test, and there really isn't. In fact *every* test seems to be out of date now. Too little: The headers under boost/config/ are included via macros so aren't placed in the dependency graph. I can fix this easily enough in the Boost.Config tests, but ideally this should be fixed at the top level by making everything under boost/config/ a dependency of boost/config.hpp. This is currently causing some Boost.Config tests to fail in the online matrix, I bet there are others as well. Anyone any ideas? Thanks, John.
John Maddock wrote:
If I build the type_traits tests so everything is up to date, and then touch say type_with_alignment.hpp, then in libs/type_traits/test do:
bjam add_const_test
And for some strange reason bjam thinks there's a dependency on type_with_alignment.hpp from add_const_test, and there really isn't.
You have #include "check_type.hpp" in add_const_test.cpp, and in
check_type.hpp you have #include
John Maddock wrote:
You have #include "check_type.hpp" in add_const_test.cpp, and in check_type.hpp you have #include
, and from then on, you can imagine what happens. Oh, that would be that then :(
I don't know how smart b2's scanner is; my simplistic one would be fooled by
/**/#include
AMDG On 02/18/2014 09:39 AM, Peter Dimov wrote:
I don't know how smart b2's scanner is; my simplistic one would be fooled by
/**/#include
which is, I believe, a valid pp directive, although I haven't, of course, checked. :-)
The pattern is "#[ \t]*include[ ]*(<(.*)>|\"(.*)\")" (no anchors), so this will match. In Christ, Steven Watanabe
Steven Watanabe wrote:
AMDG
On 02/18/2014 09:39 AM, Peter Dimov wrote:
I don't know how smart b2's scanner is; my simplistic one would be fooled by
/**/#include
...
The pattern is "#[ \t]*include[ ]*(<(.*)>|\"(.*)\")" (no anchors), so this will match.
This will match
// #include "something.hpp"
then?
OK, I change my suggestion to
#/**/include
On 18/02/2014 19:03, Peter Dimov wrote:
Steven Watanabe wrote:
AMDG
On 02/18/2014 09:39 AM, Peter Dimov wrote:
I don't know how smart b2's scanner is; my simplistic one would be fooled by
/**/#include
...
The pattern is "#[ \t]*include[ ]*(<(.*)>|\"(.*)\")" (no anchors), so this will match.
This will match
// #include "something.hpp"
then?
OK, I change my suggestion to
#/**/include
Probably over-complicated: the inclusion of Boost.Test is I believe a relic from the time when all the type-traits tests could be built as one big test program. I think they've gone beyond that now, so some cleanup is probably in order. Thanks, John.
----------------------------------------
Date: Tue, 18 Feb 2014 19:10:30 +0000 From: boost.regex@virgin.net To: boost@lists.boost.org Subject: Re: [boost] [build] Headers rule does both too much and too little
On 18/02/2014 19:03, Peter Dimov wrote:
Steven Watanabe wrote:
AMDG
On 02/18/2014 09:39 AM, Peter Dimov wrote:
I don't know how smart b2's scanner is; my simplistic one would be fooled by
/**/#include
...
The pattern is "#[ \t]*include[ ]*(<(.*)>|\"(.*)\")" (no anchors), so this will match.
This will match
// #include "something.hpp"
then?
OK, I change my suggestion to
#/**/include
Probably over-complicated: the inclusion of Boost.Test is I believe a relic from the time when all the type-traits tests could be built as one big test program. I think they've gone beyond that now, so some cleanup is probably in order.
I'd prefer to not try to trick the dependency scanner into thinking there isn't a dependency, when in fact, there is one. If this include shouldn't be there, then it should be removed, but as long as it's there, is should be tracked as a dependency and if it changes, the test should be rebuilt.
John Maddock wrote:
Too little:
The headers under boost/config/ are included via macros so aren't placed in the dependency graph. I can fix this easily enough in the Boost.Config tests, but ideally this should be fixed at the top level by making everything under boost/config/ a dependency of boost/config.hpp.
You can fix that from your side - if you're so inclined - by including the omitted headers in an #if 0 block to enable the scanner to see them. Currently a number of tests seems to be failing due to a missing config/user.hpp, so it's a real problem. But there's also something else that is wrong with the headers, as other tests seem to be using older header versions, and that doesn't seem to be caused by a missed dependency.
On 18/02/2014 18:57, Peter Dimov wrote:
John Maddock wrote:
Too little:
The headers under boost/config/ are included via macros so aren't placed in the dependency graph. I can fix this easily enough in the Boost.Config tests, but ideally this should be fixed at the top level by making everything under boost/config/ a dependency of boost/config.hpp.
You can fix that from your side - if you're so inclined - by including the omitted headers in an #if 0 block to enable the scanner to see them.
Currently a number of tests seems to be failing due to a missing config/user.hpp, so it's a real problem. But there's also something else that is wrong with the headers, as other tests seem to be using older header versions, and that doesn't seem to be caused by a missed dependency.
I see that as well - there are quite a number of failures in the Math lib tests which I simply do not see locally (and a few of us have tried to reproduce). There are frankly too many to try and track down one at a time by pestering the test-runners: who probably have better things to do anyway. Frankly it's all rather discouraging :-( John.
On Feb 18, 2014, at 12:13 PM, John Maddock wrote:
On 18/02/2014 18:57, Peter Dimov wrote:
John Maddock wrote:
Too little:
The headers under boost/config/ are included via macros so aren't placed in the dependency graph. I can fix this easily enough in the Boost.Config tests, but ideally this should be fixed at the top level by making everything under boost/config/ a dependency of boost/config.hpp.
You can fix that from your side - if you're so inclined - by including the omitted headers in an #if 0 block to enable the scanner to see them.
Currently a number of tests seems to be failing due to a missing config/user.hpp, so it's a real problem. But there's also something else that is wrong with the headers, as other tests seem to be using older header versions, and that doesn't seem to be caused by a missed dependency.
I see that as well - there are quite a number of failures in the Math lib tests which I simply do not see locally (and a few of us have tried to reproduce). There are frankly too many to try and track down one at a time by pestering the test-runners: who probably have better things to do anyway. Frankly it's all rather discouraging :-(
I can try to help if you can point me to any unexpected (non-reproducible) problem with either Sandia tester (Mac or Linux)?
AMDG On 02/18/2014 10:57 AM, Peter Dimov wrote:
John Maddock wrote:
Too little:
The headers under boost/config/ are included via macros so aren't placed in the dependency graph. I can fix this easily enough in the Boost.Config tests, but ideally this should be fixed at the top level by making everything under boost/config/ a dependency of boost/config.hpp.
You can fix that from your side - if you're so inclined - by including the omitted headers in an #if 0 block to enable the scanner to see them.
Currently a number of tests seems to be failing due to a missing config/user.hpp, so it's a real problem. But there's also something else that is wrong with the headers, as other tests seem to be using older header versions, and that doesn't seem to be caused by a missed dependency.
Are the testers using incremental builds? This shouldn't be happening in a full test run, because the regression scripts delete the header links and then re-create them. In Christ, Steven Watanabe
Steven Watanabe wrote:
Are the testers using incremental builds?
I don't think so. This run, for example: http://www.boost.org/development/tests/develop/developer/output/teeks99-02e-... is apparently using an old version of shared_count.hpp, and there's no yellow i. This one: http://www.boost.org/development/tests/develop/developer/output/VC8%20jc-bel... fails due to a missing config/user.hpp. Doesn't appear incremental as well. This one: http://www.boost.org/development/tests/develop/developer/output/DebSidC++-bo... appears to be using an old version of assert.hpp.
AMDG On 02/18/2014 02:07 PM, Peter Dimov wrote:
Steven Watanabe wrote:
Are the testers using incremental builds?
I don't think so. This run, for example:
http://www.boost.org/development/tests/develop/developer/output/teeks99-02e-...
is apparently using an old version of shared_count.hpp, and there's no yellow i.
This tester is using --force-update, which could also be a problem.
This one:
http://www.boost.org/development/tests/develop/developer/output/VC8%20jc-bel...
fails due to a missing config/user.hpp. Doesn't appear incremental as well.
This one we've discussed on the testing list. Summary: The regression test system's cleanup command is deleting a large number of headers, because git doesn't know about junctions (git clean treats a junction as an ordinary directory and deletes the contents).
This one:
http://www.boost.org/development/tests/develop/developer/output/DebSidC++-bo...
appears to be using an old version of assert.hpp.
--force-update here too. In Christ, Steven Watanabe
Steven Watanabe wrote:
This tester is using --force-update, which could also be a problem.
I'm not sure what --force-update is supposed to do when using git. I looked at regression.py, but this wasn't much help. :-) The simplest fix is then to either tell testers to not use it, or ignore it when not using SVN?
On Tue, Feb 18, 2014 at 5:09 PM, Peter Dimov
Steven Watanabe wrote:
This tester is using --force-update, which could also be a problem.
I'm not sure what --force-update is supposed to do when using git. I looked at regression.py, but this wasn't much help. :-) The simplest fix is then to either tell testers to not use it, or ignore it when not using SVN?
I'll ignore the option in the new version of the scripts.. And I'm slowly removing the SVN options since the git part of testing is mostly working now. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
On Tue, Feb 18, 2014 at 4:07 PM, Peter Dimov
Steven Watanabe wrote:
Are the testers using incremental builds?
I don't think so. This run, for example:
http://www.boost.org/development/tests/develop/developer/output/teeks99-02e-...
is apparently using an old version of shared_count.hpp, and there's no yellow i.
Sorry I haven't responded to this sooner...I should probably have thrown my $0.02 since I seem to be the regression runner people are seeing this on. There are a couple things at work here. Linux runners: All the teeks99 runners with the same number are actually on the same (virtual) machine, they get run sequentially (each with a different letter after the number). To save disk space, I have one copy of the git repo that I copy into the directory (you can't do a local git clone with submodules!) for that runner just before kicking off the run.py script. As of last week, I had accidentally left a copy of the headers inside that copy of the repo, so it got copied with it. I would have expected it to be brought up to date each time, but that wasn't happening. Late last week, I deleted the copy of the headers from the repo I was copying (on all three machines teeks99-03 -04 -05) everything looks like it is working fine (it forced the headers to be generated each time). tldr; We need to investigate why the headers aren't being correctly re-generated. Windows runners: Here each runner has its own repo, so nothing was being copied in. It seemed that there was the same problem, headers that should have been re-generated weren't being re-generated. To try to force the re-generation, I added a line to my script to delete the headers before starting each run. This caused immediate failures everywhere (on teeks99-01...which is where I tried this first). Apparently, since I first generated them the code had changed over to using junctions, which weren't working for some reason with the test code. As of yesterday, things were looking better, (someone fixed the junction problem?) so I pushed this fix out to my 2nd windows machine (teeks99-02). However, this morning, I see that -01 is failing again with the same problem, so I expect to see that spread to -02 soon. This appears to be related to the "Tests failing, missing user.hpp" thread [http://lists.boost.org/boost-testing/2014/02/7561.php]. Tom
Windows runners: Here each runner has its own repo, so nothing was being copied in. It seemed that there was the same problem, headers that should have been re-generated weren't being re-generated. To try to force the re-generation, I added a line to my script to delete the headers before starting each run. This caused immediate failures everywhere (on teeks99-01...which is where I tried this first). Apparently, since I first generated them the code had changed over to using junctions, which weren't working for some reason with the test code. As of yesterday, things were looking better, (someone fixed the junction problem?) so I pushed this fix out to my 2nd windows machine (teeks99-02). However, this morning, I see that -01 is failing again with the same problem, so I expect to see that spread to -02 soon.
Tom, I wonder if I can get you to investigate: http://www.boost.org/development/tests/develop/developer/output/teeks99-02b-... It's one of a large number of Boost.Math tests that's failing on the teeks machines, but not the others, nor can any of us reproduce locally. Many thanks, John.
participants (7)
-
Ahmed Charles -
Belcourt, Kenneth -
John Maddock -
Peter Dimov -
Rene Rivera -
Steven Watanabe -
Tom Kent