Boost Test Matrix - no clang release builds?
I was putting together some additional testing for a library and my CI jobs were failing in release mode. I was able to reproduce the issue locally. On examination of the boost test matrix however I was unable to see the same failures. When I dug into it a little bit more, it looks like the clang linux build jobs only build the default variant (debug), and therefore will not catch release-mode optimization issues when running the unit tests. Here's one of the job details: https://www.boost.org/development/tests/develop/teeks99-02-dc6-17-Docker-64o... Given this matrix is the last line of defense before code is merged from develop to master, I'd suggest that the build jobs should all be targeting variant=release instead of variant=debug. The former directly impacts a release, and the latter does not. - Jim
On 11/11/2018 5:26 PM, James E. King III via Boost wrote:
I was putting together some additional testing for a library and my CI jobs were failing in release mode. I was able to reproduce the issue locally. On examination of the boost test matrix however I was unable to see the same failures. When I dug into it a little bit more, it looks like the clang linux build jobs only build the default variant (debug), and therefore will not catch release-mode optimization issues when running the unit tests.
Here's one of the job details: https://www.boost.org/development/tests/develop/teeks99-02-dc6-17-Docker-64o...
Given this matrix is the last line of defense before code is merged from develop to master, I'd suggest that the build jobs should all be targeting variant=release instead of variant=debug. The former directly impacts a release, and the latter does not.
There are regression tests for both 'develop' and 'master' branches.
- Jim
On Sun, Nov 11, 2018 at 4:26 PM James E. King III via Boost < boost@lists.boost.org> wrote:
I was putting together some additional testing for a library and my CI jobs were failing in release mode. I was able to reproduce the issue locally. On examination of the boost test matrix however I was unable to see the same failures. When I dug into it a little bit more, it looks like the clang linux build jobs only build the default variant (debug), and therefore will not catch release-mode optimization issues when running the unit tests.
Here's one of the job details:
https://www.boost.org/development/tests/develop/teeks99-02-dc6-17-Docker-64o...
Given this matrix is the last line of defense before code is merged from develop to master, I'd suggest that the build jobs should all be targeting variant=release instead of variant=debug. The former directly impacts a release, and the latter does not.
Interesting catch. There is the 'teeks99-02-mc7-opt-Docker-64on64' [1] configuration that manually specifies the -O2 flag (and a similar one for gcc [2]). Does this find the issue that you are looking for? What does variant=release do on posix platforms? For visual studio builds there is a specific set of things that are defaulted for "release" builds vs. "debug" builds, but I don't think that translates exactly to GCC/Clang. (Please correct me if I'm wrong). Tom [1] https://www.boost.org/development/tests/master/teeks99-02-mc7-opt-Docker-64o... [2] https://www.boost.org/development/tests/master/teeks99-02-mg8-opt-Docker-64o...
On 12/11/2018 11:26, James E. King III wrote:
I was putting together some additional testing for a library and my CI jobs were failing in release mode. I was able to reproduce the issue locally. On examination of the boost test matrix however I was unable to see the same failures. When I dug into it a little bit more, it looks like the clang linux build jobs only build the default variant (debug), and therefore will not catch release-mode optimization issues when running the unit tests.
Excluding compiler bugs and intentional differences (asserts, iterator debugging, and the like), differences in behaviour between debug and release builds usually indicates that the code relies on undefined behaviour somewhere. Consider running UBsan or a similar tester over the affected library.
participants (4)
-
Edward Diener -
Gavin Lambert -
James E. King III -
Tom Kent