Windows Regression Runners - Failing on Develop
Starting sometime between 3am and 5pm on September 11, 2023, all the windows regression runners started failing on the develop branch. Master branch runners on the same VM are still doing fine ( https://www.boost.org/development/tests/master/developer/summary.html). The output doesn't have anything jump out at me. Nor were there any commits to the develop branch in that window that seem suspicious to me. The regression run output ends with: ...found 1 target... # Searching for "process_jam_log.exe" in "C:\teeks99-09\run\boost_regression\stage\bin"... Traceback (most recent call last): File "run.py", line 71, in <module> runner(root) File "C:\teeks99-09\run\boost_regression_src\regression.py", line 286, in __init__ self.main() File "C:\teeks99-09\run\boost_regression_src\regression.py", line 623, in main getattr(self,action_m)() File "C:\teeks99-09\run\boost_regression_src\regression.py", line 580, in command_regression self.command_setup() File "C:\teeks99-09\run\boost_regression_src\regression.py", line 351, in command_setup self.build_if_needed(self.process_jam_log,self.pjl_toolset) File "C:\teeks99-09\run\boost_regression_src\regression.py", line 714, in build_if_needed tool[ 'build_path' ] = self.tool_path( tool ) File "C:\teeks99-09\run\boost_regression_src\regression.py", line 739, in tool_path , '\n'.join( [ name_or_spec[ 'path' ], build_dir ] ) Exception: Cannot find "process_jam_log.exe" in any of the following locations: C:\teeks99-09\run\boost_regression\stage\bin\process_jam_log.exe C:\teeks99-09\run\boost_regression\stage\bin With the full output available here: https://gist.github.com/teeks99/bb4230f6a1efdd7af11bd93341c950bf Any ideas on this would be much appreciated.
Tom Kent wrote:
Starting sometime between 3am and 5pm on September 11, 2023, all the windows regression runners started failing on the develop branch.
Master branch runners on the same VM are still doing fine ( https://www.boost.org/development/tests/master/developer/summary.html) .
The output doesn't have anything jump out at me. Nor were there any commits to the develop branch in that window that seem suspicious to me.
The regression run output ends with: ...found 1 target...
[...]
With the full output available here: https://gist.github.com/teeks99/bb4230f6a1efdd7af11bd93341c950bf
Any ideas on this would be much appreciated.
According to the log, the script tries to build process_jam_log.exe using msvc-10.0. Since process_jam_log depends on Boost.Filesystem, which now requires C++11, the build silently fails.
On 9/28/23 02:06, Peter Dimov via Boost wrote:
Tom Kent wrote:
Starting sometime between 3am and 5pm on September 11, 2023, all the windows regression runners started failing on the develop branch.
Master branch runners on the same VM are still doing fine ( https://www.boost.org/development/tests/master/developer/summary.html) .
The output doesn't have anything jump out at me. Nor were there any commits to the develop branch in that window that seem suspicious to me.
The regression run output ends with: ...found 1 target...
[...]
With the full output available here: https://gist.github.com/teeks99/bb4230f6a1efdd7af11bd93341c950bf
Any ideas on this would be much appreciated.
According to the log, the script tries to build process_jam_log.exe using msvc-10.0.
Since process_jam_log depends on Boost.Filesystem, which now requires C++11, the build silently fails.
Which brings me back to the idea[1] that Boost.Build should probably show a warning when an explicitly requested target is not built because of unsatisfied requirements. Is this something we can (or should) do on Boost level or should we request a Boost.Build enhancement? [1]: https://github.com/boostorg/auto_index/issues/4#issuecomment-1718949433
Andrey Semashev wrote:
According to the log, the script tries to build process_jam_log.exe using msvc-10.0.
Since process_jam_log depends on Boost.Filesystem, which now requires C++11, the build silently fails.
Which brings me back to the idea[1] that Boost.Build should probably show a warning when an explicitly requested target is not built because of unsatisfied requirements.
I grow increasingly convinced that the way these requirements work (add <build>no if the requirement isn't met) is no longer what we need. It's much preferable to have the compile error instead. In other words, the libraries adding requirements of this kind is in my opinion not a feature. It creates more problems than it solves - if it even solves any.
On Thu, Sep 28, 2023 at 11:40 AM Peter Dimov via Boost <boost@lists.boost.org> wrote:
Andrey Semashev wrote:
According to the log, the script tries to build process_jam_log.exe using msvc-10.0.
Since process_jam_log depends on Boost.Filesystem, which now requires C++11, the build silently fails.
Which brings me back to the idea[1] that Boost.Build should probably show a warning when an explicitly requested target is not built because of unsatisfied requirements.
I grow increasingly convinced that the way these requirements work (add <build>no if the requirement isn't met) is no longer what we need.
It's much preferable to have the compile error instead.
In other words, the libraries adding requirements of this kind is in my opinion not a feature. It creates more problems than it solves - if it even solves any.
+1 And I remember making that point multiple times in the past. -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supone Nada -- Robot Dreams - http://robot-dreams.net
On Wed, Sep 27, 2023 at 6:06 PM Peter Dimov via Boost <boost@lists.boost.org> wrote:
Tom Kent wrote:
Starting sometime between 3am and 5pm on September 11, 2023, all the windows regression runners started failing on the develop branch.
Master branch runners on the same VM are still doing fine ( https://www.boost.org/development/tests/master/developer/summary.html) .
The output doesn't have anything jump out at me. Nor were there any commits to the develop branch in that window that seem suspicious to me.
The regression run output ends with: ...found 1 target...
[...]
With the full output available here: https://gist.github.com/teeks99/bb4230f6a1efdd7af11bd93341c950bf
Any ideas on this would be much appreciated.
According to the log, the script tries to build process_jam_log.exe using msvc-10.0.
Since process_jam_log depends on Boost.Filesystem, which now requires C++11, the build silently fails.
Any ideas on how I can convince that build to use msvc-14.3 while keeping msvc-10.0 in my user-config.jam? I'm guessing there is some auto-picking logic at play that I might be able to control? I'd like to keep still testing with that version of visual studio a bit longer, thus I don't want to completely get rid of it. Thanks, Tom
Tom Kent wrote:
Any ideas on how I can convince that build to use msvc-14.3 while keeping msvc-10.0 in my user-config.jam? I'm guessing there is some auto-picking logic at play that I might be able to control? I'd like to keep still testing with that version of visual studio a bit longer, thus I don't want to completely get rid of it.
Easiest is probably just to put 14.3 first (in user-config).
On Wed, Sep 27, 2023 at 7:12 PM Peter Dimov <pdimov@gmail.com> wrote:
Tom Kent wrote:
Any ideas on how I can convince that build to use msvc-14.3 while keeping msvc-10.0 in my user-config.jam? I'm guessing there is some auto-picking logic at play that I might be able to control? I'd like to keep still testing with that version of visual studio a bit longer, thus I don't want to completely get rid of it.
Easiest is probably just to put 14.3 first (in user-config).
That did make msvc-14.3 the build tool (b2 documentation on user-config.jam could be improved to say order matters), however it didn't solve the problem. New output here: https://gist.github.com/teeks99/4b0bf5fea31c972be8f3d79fd71bf774 The end: [3] msvc-14.3/release/build-no/link-static/python-2.7/threadapi-win32/threading-multi/visibility-hidden ...found 1 target... # Searching for "process_jam_log.exe" in "C:\teeks99-09\run\boost_regression\stage\bin"... Traceback (most recent call last): File "run.py", line 71, in <module> runner(root) File "C:\teeks99-09\run\boost_regression_src\regression.py", line 286, in __init__ self.main() File "C:\teeks99-09\run\boost_regression_src\regression.py", line 623, in main getattr(self,action_m)() File "C:\teeks99-09\run\boost_regression_src\regression.py", line 580, in command_regression self.command_setup() File "C:\teeks99-09\run\boost_regression_src\regression.py", line 351, in command_setup self.build_if_needed(self.process_jam_log,self.pjl_toolset) File "C:\teeks99-09\run\boost_regression_src\regression.py", line 714, in build_if_needed tool[ 'build_path' ] = self.tool_path( tool ) File "C:\teeks99-09\run\boost_regression_src\regression.py", line 739, in tool_path , '\n'.join( [ name_or_spec[ 'path' ], build_dir ] ) Exception: Cannot find "process_jam_log.exe" in any of the following locations: C:\teeks99-09\run\boost_regression\stage\bin\process_jam_log.exe C:\teeks99-09\run\boost_regression\stage\bin Tom
Tom Kent wrote:
On Wed, Sep 27, 2023 at 7:12 PM Peter Dimov <pdimov@gmail.com <mailto:pdimov@gmail.com> > wrote:
Any ideas on how I can convince that build to use msvc-14.3 while keeping msvc-10.0 in my user-config.jam? I'm guessing there is some auto-
Tom Kent wrote: picking
logic at play that I might be able to control? I'd like to keep still testing with that version of visual studio a bit longer, thus I don't want to completely get rid of it.
Easiest is probably just to put 14.3 first (in user-config).
That did make msvc-14.3 the build tool (b2 documentation on user-config.jam could be improved to say order matters), however it didn't solve the problem.
New output here: https://gist.github.com/teeks99/4b0bf5fea31c972be8f3d79fd71bf774
The end:
[3] msvc-14.3/release/build-no/link-static/python-2.7/threadapi- win32/threading-multi/visibility-hidden ...found 1 target...
That's odd. Even the "has BCrypt API" check fails, and it should succeed. What does config.log in the binary directory contain?
On Thu, Sep 28, 2023 at 8:03 AM Peter Dimov <pdimov@gmail.com> wrote:
On Wed, Sep 27, 2023 at 7:12 PM Peter Dimov <pdimov@gmail.com <mailto:pdimov@gmail.com> > wrote:
Tom Kent wrote: > Any ideas on how I can convince that build to use msvc-14.3 while keeping > msvc-10.0 in my user-config.jam? I'm guessing there is some auto- picking > logic at play that I might be able to control? I'd like to keep still testing with > that version of visual studio a bit longer, thus I don't want to completely get rid > of it.
Easiest is probably just to put 14.3 first (in user-config).
That did make msvc-14.3 the build tool (b2 documentation on user-config.jam could be improved to say order matters), however it didn't solve the
Tom Kent wrote: problem.
New output here: https://gist.github.com/teeks99/4b0bf5fea31c972be8f3d79fd71bf774
The end:
[3] msvc-14.3/release/build-no/link-static/python-2.7/threadapi- win32/threading-multi/visibility-hidden ...found 1 target...
That's odd. Even the "has BCrypt API" check fails, and it should succeed.
What does config.log in the binary directory contain?
I'm not familiar enough with that to make much sense out of it, but it is here: https://gist.github.com/teeks99/a59fde8c9fbb9208b3a5dac24a97d4ef Tom
Tom Kent wrote:
New output here:
https://gist.github.com/teeks99/4b0bf5fea31c972be8f3d79fd71bf774
The end:
[3] msvc-14.3/release/build-no/link-static/python-2.7/threadapi- win32/threading-multi/visibility-hidden ...found 1 target...
That's odd. Even the "has BCrypt API" check fails, and it should succeed.
What does config.log in the binary directory contain?
I'm not familiar enough with that to make much sense out of it, but it is here: https://gist.github.com/teeks99/a59fde8c9fbb9208b3a5dac24a97d4ef
Well the reason all of Filesystem's configure checks fail, even those that ought to succeed, is C:\teeks99-09\run\boost_root\libs\filesystem\src\platform_config.hpp(74): fatal error C1083: Cannot open include file: 'boost/winapi/config.hpp': No such file or directory The reason 'boost/winapi/config.hpp' isn't found is because the corresponding header symlink in $BOOST_ROOT/boost isn't created, for some reason. I can reproduce the failure when I delete my boost/ subdirectory, and when I recreate it with `b2 headers`, I can un-reproduce it. b2 is supposed to create these header symlinks as it goes, though; the explicit `b2 headers` is generally unnecessary. Something must have changed to throw it off, I'm not sure what.
AMDG On 9/28/23 19:59, Peter Dimov via Boost wrote:
C:\teeks99-09\run\boost_root\libs\filesystem\src\platform_config.hpp(74): fatal error C1083: Cannot open include file: 'boost/winapi/config.hpp': No such file or directory
The reason 'boost/winapi/config.hpp' isn't found is because the corresponding header symlink in $BOOST_ROOT/boost isn't created, for some reason.
I can reproduce the failure when I delete my boost/ subdirectory, and when I recreate it with `b2 headers`, I can un-reproduce it.
b2 is supposed to create these header symlinks as it goes, though; the explicit `b2 headers` is generally unnecessary. Something must have changed to throw it off, I'm not sure what.
When b2 fails to create a symlink it's usually because the header is only included through a macro, and the #include scanner misses it. The fact that entire directories are symlinked also masks most issues. If the directory is getting split because multiple libraries contain headers in the same directory, then that can cause new issues to appear. In Christ, Steven Watanabe
Steven Watanabe wrote:
When b2 fails to create a symlink it's usually because the header is only included through a macro, and the #include scanner misses it.
That was my first thought, but it's a normal include: https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a8... It might have to do something with the fact that platform_config.hpp is in src/; not sure how this gets into the include path. https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a8... But boost/winapi/config.hpp is also included directly from the config check source: https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a8... so the include scanner must be able to see it anyway. boost/winapi is a junction/symlink for me, so it's not the split either.
Steven Watanabe wrote:
When b2 fails to create a symlink it's usually because the header is only included through a macro, and the #include scanner misses it.
That was my first thought, but it's a normal include:
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193be d7280a894a06726/src/platform_config.hpp#L74
It might have to do something with the fact that platform_config.hpp is in src/; not sure how this gets into the include path.
Here: https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a8...
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193be d7280a894a06726/config/has_bcrypt.cpp#L9
But boost/winapi/config.hpp is also included directly from the config check source:
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193be d7280a894a06726/config/has_bcrypt.cpp#L17C11-L17C34
so the include scanner must be able to see it anyway.
boost/winapi is a junction/symlink for me, so it's not the split either.
Further experimentation reveals that the header links are correctly created when I try to build the has_bcrypt exe directly: C:\boost-git\develop>rd/s/q boost C:\boost-git\develop>b2 libs/filesystem/config//has_bcrypt Performing configuration checks - default address-model : 64-bit (cached) [1] - default architecture : x86 (cached) [1] - symlinks supported : yes (cached) [1] msvc-14.3 ...found 40 targets... ...updating 7 targets... mklink-or-dir boost mklink-or-dir boost\predef symbolic link created for boost\predef <<===>> ..\libs\predef\include\boost\predef mklink-or-dir boost\winapi symbolic link created for boost\winapi <<===>> ..\libs\winapi\include\boost\winapi mklink-or-dir boost\filesystem symbolic link created for boost\filesystem <<===>> ..\libs\filesystem\include\boost\filesystem link.mklink boost\config.hpp symbolic link created for boost\config.hpp <<===>> ..\libs\config\include\boost\config.hpp mklink-or-dir boost\system symbolic link created for boost\system <<===>> ..\libs\system\include\boost\system mklink-or-dir boost\detail mklink-or-dir boost\config symbolic link created for boost\config <<===>> ..\libs\config\include\boost\config link.mklink boost\detail\workaround.hpp symbolic link created for boost\detail\workaround.hpp <<===>> ..\..\libs\config\include\boost\detail\workaround.hpp ...patience... link.mklink boost\version.hpp symbolic link created for boost\version.hpp <<===>> ..\libs\config\include\boost\version.hpp compile-c-c++ bin.v2\libs\filesystem\config\msvc-14.3\debug\threading-multi\has_bcrypt.obj has_bcrypt.cpp msvc.link bin.v2\libs\filesystem\config\msvc-14.3\debug\threading-multi\has_bcrypt.exe ...updated 13 targets... So it's only when it's being built by configure.builds: https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a8... are the headers not being linked.
On Fri, Sep 29, 2023 at 11:24 AM Peter Dimov via Boost < boost@lists.boost.org> wrote:
Steven Watanabe wrote:
When b2 fails to create a symlink it's usually because the header is only included through a macro, and the #include scanner misses it.
That was my first thought, but it's a normal include:
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193be d7280a894a06726/src/platform_config.hpp#L74
It might have to do something with the fact that platform_config.hpp is in src/; not sure how this gets into the include path.
Here:
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a8...
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193be d7280a894a06726/config/has_bcrypt.cpp#L9
But boost/winapi/config.hpp is also included directly from the config check source:
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193be d7280a894a06726/config/has_bcrypt.cpp#L17C11-L17C34
so the include scanner must be able to see it anyway.
boost/winapi is a junction/symlink for me, so it's not the split either.
Further experimentation reveals that the header links are correctly created when I try to build the has_bcrypt exe directly:
C:\boost-git\develop>rd/s/q boost
C:\boost-git\develop>b2 libs/filesystem/config//has_bcrypt Performing configuration checks
- default address-model : 64-bit (cached) [1] - default architecture : x86 (cached) [1] - symlinks supported : yes (cached)
[1] msvc-14.3 ...found 40 targets... ...updating 7 targets... mklink-or-dir boost mklink-or-dir boost\predef symbolic link created for boost\predef <<===>> ..\libs\predef\include\boost\predef mklink-or-dir boost\winapi symbolic link created for boost\winapi <<===>> ..\libs\winapi\include\boost\winapi mklink-or-dir boost\filesystem symbolic link created for boost\filesystem <<===>> ..\libs\filesystem\include\boost\filesystem link.mklink boost\config.hpp symbolic link created for boost\config.hpp <<===>> ..\libs\config\include\boost\config.hpp mklink-or-dir boost\system symbolic link created for boost\system <<===>> ..\libs\system\include\boost\system mklink-or-dir boost\detail mklink-or-dir boost\config symbolic link created for boost\config <<===>> ..\libs\config\include\boost\config link.mklink boost\detail\workaround.hpp symbolic link created for boost\detail\workaround.hpp <<===>> ..\..\libs\config\include\boost\detail\workaround.hpp ...patience... link.mklink boost\version.hpp symbolic link created for boost\version.hpp <<===>> ..\libs\config\include\boost\version.hpp compile-c-c++ bin.v2\libs\filesystem\config\msvc-14.3\debug\threading-multi\has_bcrypt.obj has_bcrypt.cpp msvc.link bin.v2\libs\filesystem\config\msvc-14.3\debug\threading-multi\has_bcrypt.exe ...updated 13 targets...
So it's only when it's being built by configure.builds:
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a8...
are the headers not being linked.
Sometime this morning, this seems to have migrated to master as well as develop. There was a bit of conversation about symlinks, but I'm not sure that those are ever being made on this machine, as windows has some weird rules about what users can make them. Failing that, does it make copies? If anyone needs access to this environment (i.e. have ideas but not a windows environment), I can give remote desktop credentials for a test machine. Tom
On 10/2/23 03:03, Tom Kent via Boost wrote:
On Fri, Sep 29, 2023 at 11:24 AM Peter Dimov via Boost < boost@lists.boost.org> wrote:
Steven Watanabe wrote:
When b2 fails to create a symlink it's usually because the header is only included through a macro, and the #include scanner misses it.
That was my first thought, but it's a normal include:
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193be d7280a894a06726/src/platform_config.hpp#L74
It might have to do something with the fact that platform_config.hpp is in src/; not sure how this gets into the include path.
Here:
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a8...
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193be d7280a894a06726/config/has_bcrypt.cpp#L9
But boost/winapi/config.hpp is also included directly from the config check source:
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193be d7280a894a06726/config/has_bcrypt.cpp#L17C11-L17C34
so the include scanner must be able to see it anyway.
boost/winapi is a junction/symlink for me, so it's not the split either.
Further experimentation reveals that the header links are correctly created when I try to build the has_bcrypt exe directly:
C:\boost-git\develop>rd/s/q boost
C:\boost-git\develop>b2 libs/filesystem/config//has_bcrypt Performing configuration checks
- default address-model : 64-bit (cached) [1] - default architecture : x86 (cached) [1] - symlinks supported : yes (cached)
[1] msvc-14.3 ...found 40 targets... ...updating 7 targets... mklink-or-dir boost mklink-or-dir boost\predef symbolic link created for boost\predef <<===>> ..\libs\predef\include\boost\predef mklink-or-dir boost\winapi symbolic link created for boost\winapi <<===>> ..\libs\winapi\include\boost\winapi mklink-or-dir boost\filesystem symbolic link created for boost\filesystem <<===>> ..\libs\filesystem\include\boost\filesystem link.mklink boost\config.hpp symbolic link created for boost\config.hpp <<===>> ..\libs\config\include\boost\config.hpp mklink-or-dir boost\system symbolic link created for boost\system <<===>> ..\libs\system\include\boost\system mklink-or-dir boost\detail mklink-or-dir boost\config symbolic link created for boost\config <<===>> ..\libs\config\include\boost\config link.mklink boost\detail\workaround.hpp symbolic link created for boost\detail\workaround.hpp <<===>> ..\..\libs\config\include\boost\detail\workaround.hpp ...patience... link.mklink boost\version.hpp symbolic link created for boost\version.hpp <<===>> ..\libs\config\include\boost\version.hpp compile-c-c++ bin.v2\libs\filesystem\config\msvc-14.3\debug\threading-multi\has_bcrypt.obj has_bcrypt.cpp msvc.link bin.v2\libs\filesystem\config\msvc-14.3\debug\threading-multi\has_bcrypt.exe ...updated 13 targets...
So it's only when it's being built by configure.builds:
https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a8...
are the headers not being linked.
Sometime this morning, this seems to have migrated to master as well as develop.
There was a bit of conversation about symlinks, but I'm not sure that those are ever being made on this machine, as windows has some weird rules about what users can make them. Failing that, does it make copies?
Yes, it should copy the headers if symlinks cannot be created. But, if Peter is correct, the problem is not that it can't create the symlinks. The problem is that Boost.Build doesn't track headers required by configure checks. I think the regression testing scripts should be running `b2 headers` before building anything. Automatic creation of the unified include tree has never worked reliably, so `b2 headers` is a mandatory step after git checkout. IMHO, the automatic creation of symlinks should be removed, as it's not useful and just wastes build time.
participants (5)
-
Andrey Semashev -
Peter Dimov -
René Ferdinand Rivera Morell -
Steven Watanabe -
Tom Kent