Do we still want develop branch?
Hi, The recent case of a commit in develop that was left unmerged to master made me wondering if people still find the develop branch useful. Historically (since the SVN days), we used a separate branch for active development, which we then merged to the stable branch that was used for creating Boost releases. This allowed to test libraries together before changes propagate to the stable branch. This was useful back when the official test matrix was the primary gauge of the Boost "health", so when something broke, it was rather visible, even if the breakage manifested in downstream libraries. With the adoption of git and CI, the relevance of the test matrix has diminished, and, presumably, people have shifted towards monitoring their libraries' CI. Which means, people are less likely to notice that something broke downstream, unless the downstream developer notices and reports the problem. Although I have been that downstream developer myself quite a few times, I have to admit that this role of the develop branch of "an integration testing field" is not well filled in the current workflow. Noticing the breakage is mostly luck (i.e. a matter of me making a commit to my library and noticing the CI failure caused by an upstream dependency) rather than a rule. Additionally, I have been asked on a few occasions to avoid development directly in the develop branch and use separate feature branches instead. That is, do your commits in a feature branch, let the CI pass, then merge to develop, let the CI pass again, then merge to master. Apparently, some people are using this workflow by default to avoid breakage in develop, which means develop is no longer the branch where the actual development happens. (This workflow actually makes develop meaningless because your CI over the feature branch already tests your changes against develop of the rest of Boost. Which means after merging to develop you're running the same CI again, and might as well just merge to master straight away.) Then there's the recurring problem that led to this post - people forget to merge to master from time to time. Or simply don't have time to merge. Or the library is no longer actively maintained. Or the commit is left out of master for any other reason, which means users don't receive their bug fixes in a release. Personally, I do try to remember the PRs I make during the release cycle and look through them just before the release to make sure they are merged to master. Some of you probably received gentle merge reminders from me. I'm not sure if anyone else is doing this, but I can say this is an active mental effort on my part, which I'd rather not have to make. And even then I can't be sure I remember all PRs or have the time to cycle though them before the release. Which brings me to my question. Do people still find the develop branch useful? Maybe we'd be better off just dropping it and performing all development in feature branches before merging straight to master? Also, if you do find develop useful, what do you think about creating a GitHub Action to automatically merge develop to master once the CI on develop succeeds? Maybe there are other ideas how to avoid commits unmerged to master? Thanks.
Andrey Semashev wrote:
(This workflow actually makes develop meaningless because your CI over the feature branch already tests your changes against develop of the rest of Boost. Which means after merging to develop you're running the same CI again, and might as well just merge to master straight away.)
You are, but others aren't. They aren't testing your feature branch, but they are testing your develop branch each time they merge to develop.
On 11/6/23 02:02, Peter Dimov via Boost wrote:
Andrey Semashev wrote:
(This workflow actually makes develop meaningless because your CI over the feature branch already tests your changes against develop of the rest of Boost. Which means after merging to develop you're running the same CI again, and might as well just merge to master straight away.)
You are, but others aren't. They aren't testing your feature branch, but they are testing your develop branch each time they merge to develop.
Sure, but again, does that integration testing actually work? For how long do you hold the changes in develop before merging to master? What about libraries that don't run their CI in this time frame? The more you linger, the more likely you'll forget to merge.
Andrey Semashev wrote:
On 11/6/23 02:02, Peter Dimov via Boost wrote:
Andrey Semashev wrote:
(This workflow actually makes develop meaningless because your CI over the feature branch already tests your changes against develop of the rest of Boost. Which means after merging to develop you're running the same CI again, and might as well just merge to master straight away.)
You are, but others aren't. They aren't testing your feature branch, but they are testing your develop branch each time they merge to develop.
Sure, but again, does that integration testing actually work?
It does, at least some of the time.
For how long do you hold the changes in develop before merging to master?
A week or two, usually.
The more you linger, the more likely you'll forget to merge.
I don't forget to merge because I have mergecheck.bat which does @for %%i in (libs/accumulators libs/array libs/assert libs/bimap libs/bind libs/chrono libs/compat libs/container_hash libs/core libs/crc libs/describe libs/endian libs/function libs/headers libs/iostreams libs/lambda libs/lambda2 libs/mp11 libs/msm libs/numeric/conversion libs/ptr_container libs/ratio libs/smart_ptr libs/system libs/thread libs/throw_exception libs/timer libs/tuple libs/type_erasure libs/typeof libs/uuid libs/variant2 tools/boostdep tools/boost_install tools/check_build tools/cmake tools/quickbook) do @echo --- %%i --- & git -C %%i diff --name-status origin/master..origin/develop I used to have the equivalent on Travis, scheduled weekly: https://github.com/pdimov/merge-reminder/blob/master/.travis.yml but Travis unfortunately died, so now I run the script manually. :-)
On 11/6/23 02:55, Peter Dimov via Boost wrote:
Andrey Semashev wrote:
On 11/6/23 02:02, Peter Dimov via Boost wrote:
Andrey Semashev wrote:
(This workflow actually makes develop meaningless because your CI over the feature branch already tests your changes against develop of the rest of Boost. Which means after merging to develop you're running the same CI again, and might as well just merge to master straight away.)
You are, but others aren't. They aren't testing your feature branch, but they are testing your develop branch each time they merge to develop.
Sure, but again, does that integration testing actually work?
It does, at least some of the time.
For how long do you hold the changes in develop before merging to master?
A week or two, usually.
The more you linger, the more likely you'll forget to merge.
I don't forget to merge because I have mergecheck.bat which does
@for %%i in (libs/accumulators libs/array libs/assert libs/bimap libs/bind libs/chrono libs/compat libs/container_hash libs/core libs/crc libs/describe libs/endian libs/function libs/headers libs/iostreams libs/lambda libs/lambda2 libs/mp11 libs/msm libs/numeric/conversion libs/ptr_container libs/ratio libs/smart_ptr libs/system libs/thread libs/throw_exception libs/timer libs/tuple libs/type_erasure libs/typeof libs/uuid libs/variant2 tools/boostdep tools/boost_install tools/check_build tools/cmake tools/quickbook) do @echo --- %%i --- & git -C %%i diff --name-status origin/master..origin/develop
I used to have the equivalent on Travis, scheduled weekly:
https://github.com/pdimov/merge-reminder/blob/master/.travis.yml
but Travis unfortunately died, so now I run the script manually. :-)
FWIW, there seems to be a way to trigger GHA workflows on schedule: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workf... Anyway, I don't have such a script, and I suspect I'm not alone. I know I would just as easily forget to run it in a week or two as I would forget to merge, so it wouldn't really help me much. Which is why I typically merge to master as soon as the CI on develop passes (i.e. when I receive an email notification). It just keeps my mental backlog shorter.
Andrey Semashev wrote:
I used to have the equivalent on Travis, scheduled weekly:
https://github.com/pdimov/merge-reminder/blob/master/.travis.yml
but Travis unfortunately died, so now I run the script manually. :-)
FWIW, there seems to be a way to trigger GHA workflows on schedule:
https://docs.github.com/en/actions/using-workflows/events-that-trigger-workf...
Thanks. I modified my merge-reminder to use GHA: https://github.com/pdimov/merge-reminder/blob/master/.github/workflows/ci.ym... Everyone should feel free to steal this and make his own merge reminder.
On 11/5/23 3:02 PM, Peter Dimov via Boost wrote:
Andrey Semashev wrote:
(This workflow actually makes develop meaningless because your CI over the feature branch already tests your changes against develop of the rest of Boost. Which means after merging to develop you're running the same CI again, and might as well just merge to master straight away.)
You are, but others aren't. They aren't testing your feature branch, but they are testing your develop branch each time they merge to develop.
Right. But I never signed up to spend time testing/debugging other peoples libraries! I really resent doing this. I'm fully occupied just testing/debugging my own errors (and of those that submit PRS) Robert Ramey
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey wrote:
On 11/5/23 3:02 PM, Peter Dimov via Boost wrote:
Andrey Semashev wrote:
(This workflow actually makes develop meaningless because your CI over the feature branch already tests your changes against develop of the rest of Boost. Which means after merging to develop you're running the same CI again, and might as well just merge to master straight away.)
You are, but others aren't. They aren't testing your feature branch, but they are testing your develop branch each time they merge to develop.
Right. But I never signed up to spend time testing/debugging other peoples libraries! I really resent doing this. I'm fully occupied just testing/debugging my own errors (and of those that submit PRS)
If we at Boost don't test our own libraries, who is supposed to?
On 11/5/23 4:39 PM, Peter Dimov via Boost wrote:
If we at Boost don't test our own libraries, who is supposed to?
Why do I have to test and chase down bugs in libraries I don't even use? That you have to ask the question illustrates the problem. Testing as it's currently constituted isn't working. It needs to be rethought. Maybe we should have another thread for this. FWIW - I did propose and implement and alternative for the Boost Library Incubator - but I failed to convince enough people to consider the issue. Perhaps the time is ripe for serious discussion. BTW - this is only one of several issues based in common C++ software practices and misconceptions that are hindering progress in Boost. Again - another thread. And of course they apply to C++ and the standards process in general. But they're more obvious in Boost because it's much higher visibility than other software. Maybe we agree for now that all software released by Boost should be tested? And that this is not being done now? Robert Ramey
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey wrote:
On 11/5/23 4:39 PM, Peter Dimov via Boost wrote:
If we at Boost don't test our own libraries, who is supposed to?
Why do I have to test and chase down bugs in libraries I don't even use?
That's what we do. We write code for other people to use, and we don't necessarily use theirs in return. The C++ code in Boost is a random sample of the C++ code outside of Boost, and if we have a problem, (a) ignoring it will not make it disappear and (b) that problem will also exist outside of Boost. So if you do nothing, many others will then encounter the same issue, and complain. Someone will need to take care of it. If not you, then who?
On 11/6/23 3:27 AM, Peter Dimov via Boost wrote:
Robert Ramey wrote:
On 11/5/23 4:39 PM, Peter Dimov via Boost wrote:
If we at Boost don't test our own libraries, who is supposed to?
Why do I have to test and chase down bugs in libraries I don't even use?
That's what we do. We write code for other people to use, and we don't necessarily use theirs in return.
The C++ code in Boost is a random sample of the C++ code outside of Boost, and if we have a problem, (a) ignoring it will not make it disappear and We're ignoring it now - that's the problem! (b) that problem will also exist outside of Boost. So if you do nothing, many others will then encounter the same issue, and complain.
Right. If we were properly testing Boost, these things would be caught before release. Apparently, no one is responsable for maintaining the test script. Maybe can fix that.
Someone will need to take care of it. If not you, then who?
The person who is responsable for maintenance of the library. I really don't have a lot of extra time to deal with other peoples responsibilities.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey wrote:
Someone will need to take care of it. If not you, then who?
The person who is responsable for maintenance of the library. I really don't have a lot of extra time to deal with other peoples responsibilities.
And how will that person know if you deliberately set things up so that you aren't seeing the problem and therefore don't inform him?
On 11/6/23 8:20 AM, Peter Dimov via Boost wrote:
Robert Ramey wrote:
Someone will need to take care of it. If not you, then who?
The person who is responsable for maintenance of the library. I really don't have a lot of extra time to deal with other peoples responsibilities.
And how will that person know if you deliberately set things up so that you aren't seeing the problem and therefore don't inform him?
He needs to write more and better tests. If he's using the boost serialization library to test his software, He's got a problem. I'm almost 76 year old. Is it really a good idea to depend on me for the future integrity of the Boost release? Robert Ramey
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey wrote:
On 11/6/23 8:20 AM, Peter Dimov via Boost wrote:
Robert Ramey wrote:
Someone will need to take care of it. If not you, then who?
The person who is responsable for maintenance of the library. I really don't have a lot of extra time to deal with other peoples responsibilities.
And how will that person know if you deliberately set things up so that you aren't seeing the problem and therefore don't inform him?
He needs to write more and better tests. If he's using the boost serialization library to test his software, He's got a problem.
Test coverage is important, but tests written beforehand can never reasonably cover every usage scenario on every (as of yet unknown) platform. That's why the correct methodology is, when a downstream use is broken by a change, to add tests that make sure breakage doesn't occur again.
I'm almost 76 year old. Is it really a good idea to depend on me for the future integrity of the Boost release?
Certainly not. But that's not the question. The question is which approach produces higher quality releases, and so far each time you've said "I think that doing X will improve the quality" you've been, in my opinion, wrong.
On 11/6/23 9:50 AM, Peter Dimov via Boost wrote:
Robert Ramey wrote:
Test coverage is important, but tests written beforehand can never reasonably cover every usage scenario on every (as of yet unknown) platform.
That's why the correct methodology is, when a downstream use is broken by a change, to add tests that make sure breakage doesn't occur again.
Agreed. Over 24 years that means one has a lot of tests. This is the only way to make sure one isn't just cycling the same changes in and out. That is, it's the only way to guarantee that library quality is constantly improving: Kaisan
I'm almost 76 year old. Is it really a good idea to depend on me for the future integrity of the Boost release?
Certainly not. But that's not the question. The question is which approach produces higher quality releases, and so far each time you've said "I think that doing X will improve the quality" you've been, in my opinion, wrong.
LOL - Hows the current approach working for us? Robert Ramey
On 11/6/23 11:40 AM, Peter Dimov via Boost wrote:
Robert Ramey wrote:
LOL - Hows the current approach working for us?
Surprisingly well, all things considered But again, just because you can point to things that are wrong with the current approach does not at all mean that some other approach will be better.
I'm not sure that's fair. Aside from pointing out the problems, I've also tried to suggest alternatives.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Mon, Nov 6, 2023 at 10:17 AM Robert Ramey via Boost <boost@lists.boost.org> wrote:
Apparently, no one is responsable for maintaining the test script. Maybe can fix that.
Indeed...https://lists.boost.org/Archives/boost//2017/07/237250.php -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supone Nada -- Robot Dreams - http://robot-dreams.net
On 11/6/23 10:15 AM, René Ferdinand Rivera Morell via Boost wrote:
On Mon, Nov 6, 2023 at 10:17 AM Robert Ramey via Boost <boost@lists.boost.org> wrote:
Apparently, no one is responsable for maintaining the test script. Maybe can fix that.
Indeed...https://lists.boost.org/Archives/boost//2017/07/237250.php
I don't remember seeing this, but I guess it addresses my original question. I did try to promote the idea that boost tools be subjected to the same development requirements as boost libraries regarding: reviews, tests, documentation, etc.... But the idea didn't catch on. I also tried to organize a effort to get take a more formal approach to implementation of CMake support. That idea also fizzled. Peter stepped up and unveiled solution as a fait compli implement CMakeLists.txt in all the boost libraries. So job was done. Or was it. I already had CMake build/test implemented in my libraries so I assumed that the official one was more or less similar. I didn't think about it much after that. Recently I came to understand that the current boost usage of CMake doesn't include the tests. Wow - what is the point of that? If we eliminated b2 and depended totally on CMake then ... we be running no tests and we'd have no disputes about testing/CI . The normal boost development procededure would have caught this requirement oversite at the beginning. Sooo... I renew my periodic and probably pointless exhortation to start subjecting our tools to the same standard that we demand of of our libraries and implement procedures to guarantee that. Robert Ramey
On 11/6/23 06:36, Robert Ramey via Boost wrote:
On 11/5/23 4:39 PM, Peter Dimov via Boost wrote:
If we at Boost don't test our own libraries, who is supposed to?
Why do I have to test and chase down bugs in libraries I don't even use?
If you depend on a library that causes a CI failure, you *are* using it and are affected by it. You can stop using it and let others deal with the problem, or you can be a helpful fellow maintainer and report the problem upstream. Bonus points if you debug and propose a solution, e.g. in the form of a PR. Of course, you don't have to, but if you're willing to spend your free time on Boost anyway, this is a good way to do this, as it benefits everyone, including you and your users.
Hmmm - a rich and timely topic. The original author of the model that use in boost wrote an update to the description of his/our model here: https://nvie.com/posts/a-successful-git-branching-model/ We have serious problems with testing. I'll include my comments below. On 11/5/23 1:10 PM, Andrey Semashev via Boost wrote:
Hi,
The recent case of a commit in develop that was left unmerged to master made me wondering if people still find the develop branch useful.
Historically (since the SVN days), we used a separate branch for active development, which we then merged to the stable branch that was used for creating Boost releases. This allowed to test libraries together before changes propagate to the stable branch. This was useful back when the official test matrix was the primary gauge of the Boost "health", so when something broke, it was rather visible, even if the breakage manifested in downstream libraries.
With the adoption of git and CI, the relevance of the test matrix has diminished,
Personally, I would dispute that. I think that it's the only part of our testing infrastructure that still works. and, presumably, people have shifted towards monitoring
their libraries' CI.
The CI for the boost serialization does not and never has worked. This is easy to veryify just by looking at any of the CI output for this library. And this demonstrable fact has been pointed out numerous times. This situation is never addressed. I doubt that the serialization library is the only one with this issue. The sadist part of all this is that even if it did "work" it would still be useless. It's not uncommon for a large and/or complex library to fail one or more tests on one or more compilers due to issues in the compiler itself. These can't be "fixed" in the library. The test matrix shows all the tests x all the environments. One can easily see if any failure is general or isolated to a particular environment. The current CI just registers pass/fail for the whole library and all the environments. Some times someone will suggest skipping a particular test for a particular library so the CI comes of clean. This is basically hiding the error. Users considering using a library in their own enviroment are basically mislead that their the library works everywhere - which is demonstrably untrue. It's a fraud. The output of the CI is very user unfriendly. The current CI is very slow and consumes a ridiculous amount of resources. Which means, people are less likely to notice that
something broke downstream, unless the downstream developer notices and reports the problem. Although I have been that downstream developer myself quite a few times, I have to admit that this role of the develop branch of "an integration testing field" is not well filled in the current workflow. Noticing the breakage is mostly luck (i.e. a matter of me making a commit to my library and noticing the CI failure caused by an upstream dependency) rather than a rule.
A useful observation. The problem is that the current CI tests the development branch of one's library agains the develop branch of all the other libraries. So now my library looks like it "works" but when (and only when) it could fail when run against all master branches. So the testing shows pass when it actually fails when shipped with the master. Note that as we speak the test matrix for the master branch isn't working so it seems we never test the software which is actually being shipped.
Additionally, I have been asked on a few occasions to avoid development directly in the develop branch and use separate feature branches instead. That is, do your commits in a feature branch, let the CI pass, then merge to develop, let the CI pass again, then merge to master. Apparently, some people are using this workflow by default to avoid breakage in develop, which means develop is no longer the branch where the actual development happens.
true and very useful. See the web link above
(This workflow actually makes develop meaningless because your CI over the feature branch already tests your changes against develop of the rest of Boost. Which means after merging to develop you're running the same CI again, and might as well just merge to master straight away.)
I see the merit in this observation. Personally, on my own machine, I test my development or feature branch agains the master branch of all the other libraries. It's the only way to know that when the changes are merged into the master the software will still work.
Then there's the recurring problem that led to this post - people forget to merge to master from time to time. Or simply don't have time to merge. Or the library is no longer actively maintained. Or the commit is left out of master for any other reason, which means users don't receive their bug fixes in a release.
Right
Personally, I do try to remember the PRs I make during the release cycle and look through them just before the release to make sure they are merged to master. Some of you probably received gentle merge reminders from me. I'm not sure if anyone else is doing this, but I can say this is an active mental effort on my part, which I'd rather not have to make. And even then I can't be sure I remember all PRs or have the time to cycle though them before the release.
Which brings me to my question. Do people still find the develop branch useful? Maybe we'd be better off just dropping it and performing all development in feature branches before merging straight to master?
I think this a good idea. Among other things, it would effectively mean that everyone would be using my method of testing the "next release version" rather than the current develop version.
Also, if you do find develop useful, what do you think about creating a GitHub Action to automatically merge develop to master once the CI on develop succeeds? Maybe there are other ideas how to avoid commits unmerged to master?
Yipes - more surprising behavior. Flog your idea above instead. When I started writing this response, I had a negative predisposition. But as I started to articulate my reaction, I've come around to your point of view. A modest proposal ================= Immediately a) start using your suggested workflow - update documents, procedures etc. b) drop the whole boost CI - it doesn't work, consumes a ridiculous amount of resource, and would still be useless if did. c) continue to run the test matrix as is - but of course the develop branch would not be necessary. At our leasure, we could redo the CI to be useful and efficient. Robert Ramey
Thanks.
Your welcome.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
The CI for the boost serialization does not and never has worked. This is easy to veryify just by looking at any of the CI output for this library. And this demonstrable fact has been pointed out numerous times. This situation is never addressed.
I doubt that the serialization library is the only one with this issue.
The sadist part of all this is that even if it did "work" it would still be useless. It's not uncommon for a large and/or complex library to fail one or more tests on one or more compilers due to issues in the compiler itself. These can't be "fixed" in the library. The test matrix shows all the tests x all the environments. One can easily see if any failure is general or isolated to a particular environment. The current CI just registers pass/fail for the whole library and all the environments. Some times someone will suggest skipping a particular test for a particular library so the CI comes of clean. This is basically hiding the error. Users considering using a library in their own enviroment are basically mislead that their the library works everywhere - which is demonstrably untrue. It's a fraud.
Personally I find that CI works just fine, with one caveat: you do need to update the scripts on a semi-regular basis otherwise you will find lots of useless failures often caused by changes to the host environment outside our control. This of course reflects real world usage. A typical example - clang on ubuntu jammy just stopped working - the cause was an update of the system default compiler from gcc-12 to gcc-13 which rendered the clang version we were testing non-functional. The solution is to do what you would tell a user to do - update the clang version to one that can handle gcc-13's std lib!
The output of the CI is very user unfriendly.
It's not awful, it's just a dump of the build output - just like you get if you run tests locally. It could of course be better. I just do what I would do for console output and search for "..failed" to find the failures. Oh and you can download the logs and process them yourself if you want to.
The current CI is very slow and consumes a ridiculous amount of resources.
Actually I'm astonished how fast it runs, if you haven't updated your scripts in a while though, it's likely you're using github images which have been deprecated, and will basically never run now (or take a very long time to do so). Resources, I would agree on. Best, John.
John Maddock wrote:
A typical example - clang on ubuntu jammy just stopped working - the cause was an update of the system default compiler from gcc-12 to gcc-13 which rendered the clang version we were testing non-functional. The solution is to do what you would tell a user to do - update the clang version to one that can handle gcc-13's std lib!
Or just use the ubuntu:22.04 container instead of GHA's image. Given that the GHA images break something every so often, I'm starting to think that we should just use containers for everything and isolate ourselves from the changes in the base images.
On 11/6/23 14:30, Peter Dimov via Boost wrote:
John Maddock wrote:
A typical example - clang on ubuntu jammy just stopped working - the cause was an update of the system default compiler from gcc-12 to gcc-13 which rendered the clang version we were testing non-functional. The solution is to do what you would tell a user to do - update the clang version to one that can handle gcc-13's std lib!
Or just use the ubuntu:22.04 container instead of GHA's image.
Given that the GHA images break something every so often, I'm starting to think that we should just use containers for everything and isolate ourselves from the changes in the base images.
I suspect, Docker has its overhead. Apart from downloading the Docker image (is it cached? who knows), setting it up for testing usually requires installation of more packages. Furthermore, I'm not sure if that package downloading is backed by GHA cache (probably not), which means you're pulling packaged from Ubuntu repos instead of the GHA local cache, which is configured in the GHA images. I don't need to remind you that network issues often cause spurious CI failures, which means the less network activity the better.
Andrey Semashev wrote:
On 11/6/23 14:30, Peter Dimov via Boost wrote:
John Maddock wrote:
A typical example - clang on ubuntu jammy just stopped working - the cause was an update of the system default compiler from gcc-12 to gcc-13 which rendered the clang version we were testing non-functional. The solution is to do what you would tell a user to do - update the clang version to one that can handle gcc-13's std lib!
Or just use the ubuntu:22.04 container instead of GHA's image.
Given that the GHA images break something every so often, I'm starting to think that we should just use containers for everything and isolate ourselves from the changes in the base images.
I suspect, Docker has its overhead. Apart from downloading the Docker image (is it cached? who knows), setting it up for testing usually requires installation of more packages. Furthermore, I'm not sure if that package downloading is backed by GHA cache (probably not), which means you're pulling packaged from Ubuntu repos instead of the GHA local cache, which is configured in the GHA images. I don't need to remind you that network issues often cause spurious CI failures, which means the less network activity the better.
That's why I've been (irrationally) avoiding containers as much as possible, but I suspect that none of the above is true in practice.
On Mon, 6 Nov 2023 at 12:31, Peter Dimov via Boost <boost@lists.boost.org> wrote:
Given that the GHA images break something every so often, I'm starting to think that we should just use containers for everything and isolate ourselves from the changes in the base images.
I've been doing this in MySQL for some time and it works well.
On 11/6/23 1:24 AM, John Maddock via Boost wrote:
The CI for the boost serialization does not and never has worked. This is easy to veryify just by looking at any of the CI output for this library. And this demonstrable fact has been pointed out numerous times. This situation is never addressed.
I doubt that the serialization library is the only one with this issue.
The sadist part of all this is that even if it did "work" it would still be useless. It's not uncommon for a large and/or complex library to fail one or more tests on one or more compilers due to issues in the compiler itself. These can't be "fixed" in the library. The test matrix shows all the tests x all the environments. One can easily see if any failure is general or isolated to a particular environment. The current CI just registers pass/fail for the whole library and all the environments. Some times someone will suggest skipping a particular test for a particular library so the CI comes of clean. This is basically hiding the error. Users considering using a library in their own enviroment are basically mislead that their the library works everywhere - which is demonstrably untrue. It's a fraud.
Personally I find that CI works just fine, with one caveat: you do need to update the scripts on a semi-regular basis otherwise you will find lots of useless failures often caused by changes to the host environment outside our control. This of course reflects real world usage.
A typical example - clang on ubuntu jammy just stopped working - the cause was an update of the system default compiler from gcc-12 to gcc-13 which rendered the clang version we were testing non-functional. The solution is to do what you would tell a user to do - update the clang version to one that can handle gcc-13's std lib!
The output of the CI is very user unfriendly.
It's not awful, it's just a dump of the build output
Right - that's awful - just like you get
if you run tests locally. It could of course be better.
I had to write my own library_status program to make sense of it. I've displayed the results on several ocasions to a universally unimpressed audience. Oh well. Of course this doesn't help when displaying CI output.
I just do what I would do for console output and search for "..failed" to find the failures.
I sometimes do that. It's complicated by the inclusion of tests which are meant to fail compile and/or link. And a lot of noise. Its not particularly useful when there is a failure in one environment and not in others - e.g. static vs linked libraries, etc. The test matrix is much, much better for this. Maybe the CI output could be funneled to a program which produces text matrix like output. (oh wait, I'm already doing this!).
Oh and you can download the logs and process them yourself if you want to.
Not all the information required is in the logs. Some of it is in the process jam log program (or ?) Now I don't remember.
The current CI is very slow and consumes a ridiculous amount of resources.
Actually I'm astonished how fast it runs,
actually I only concluded that from the fact that I often receive the CI output hours? / days? after a PR has been submitted. It also runs the whole CI on the most trivial of changes - like removing a space from a comment. Users often submit changes like this. if you haven't updated your
scripts in a while though, it's likely you're using github images which have been deprecated, and will basically never run now (or take a very long time to do so).
Resources, I would agree on.
Best, John.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Am 06.11.23 um 01:09 schrieb Robert Ramey via Boost:
I doubt that the serialization library is the only one with this issue. Most libraries have passing CI on multiple CI providers and we (Boost devs) work together helping each other in most cases. We have Boost.CI which has templates that are updated when common issues are found so other Boost devs can just C&P that update into their scripts. The sadist part of all this is that even if it did "work" it would still be useless. It's not uncommon for a large and/or complex library to fail one or more tests on one or more compilers due to issues in the compiler itself. These can't be "fixed" in the library. The test matrix shows all the tests x all the environments. One can easily see if any failure is general or isolated to a particular environment. That is true. But I'd consider this one big advantage of Boost that it has workarounds in place for many compiler deficits such that Boost works in more environments and even with compiler bugs than other projects. The output of the CI is very user unfriendly. How so? It is a nice list with a red or green mark next to a short description of the configuration (e.g. OS, compiler version, CXX standard) and has the full logs The test matrix on the other hand is missing easy access to logs and feedback on changes not yet merged to develop but merging without extensive tests risks breaking downstream libraries. A useful observation. The problem is that the current CI tests the development branch of one's library agains the develop branch of all the other libraries. So now my library looks like it "works" but when (and only when) it could fail when run against all master branches. So the testing shows pass when it actually fails when shipped with the master. Note that as we speak the test matrix for the master branch isn't working so it seems we never test the software which is actually being shipped. The same CI runs against master branch for pushes or merges of/to master I tend to actually open a a PR from develop to master to verify nothing will be broken on master after the merge. I see the merit in this observation. Personally, on my own machine, I test my development or feature branch agains the master branch of all the other libraries. It's the only way to know that when the changes are merged into the master the software will still work. See above: It is useful but not the only way and limitted in scope (not as many compilers/configurations tested as e.g. the Boost.CI template does)
Then there's the recurring problem that led to this post - people forget to merge to master from time to time. Or simply don't have time to merge. Or the library is no longer actively maintained. Or the commit is left out of master for any other reason, which means users don't receive their bug fixes in a release. Right
I agree here that this is a major issue and maybe the release managers can check (e.g. with Peters script) for that common issue and maybe even initiate merges. I do see a value in having a separate develop and master branch: During release cycles you can continue working on develop while master is frozen for the ongoing release. Besides that it is true that feature branches should be enough and develop could be removed. Alex
On 11/6/23 12:45, Alexander Grund via Boost wrote:
I do see a value in having a separate develop and master branch: During release cycles you can continue working on develop while master is frozen for the ongoing release. Besides that it is true that feature branches should be enough and develop could be removed.
To be clear, if Boost as a whole stops using develop, it doesn't prohibit you from having a branch like that; you can even name it develop, if you like. That is, if master is frozen, you are free to continue your work in other branches. Of course, you would be testing your changes against master branches of the rest of Boost. I suspect, it wouldn't make much of a difference in terms of the "stability" of the Boost release.
On 11/6/23 6:59 AM, Andrey Semashev via Boost wrote:
On 11/6/23 12:45, Alexander Grund via Boost wrote:
I do see a value in having a separate develop and master branch: During release cycles you can continue working on develop while master is frozen for the ongoing release. Besides that it is true that feature branches should be enough and develop could be removed.
To be clear, if Boost as a whole stops using develop, it doesn't prohibit you from having a branch like that; you can even name it develop, if you like. That is, if master is frozen, you are free to continue your work in other branches.
Of course, you would be testing your changes against master branches of the rest of Boost. I suspect, it wouldn't make much of a difference in terms of the "stability" of the Boost release.
That's what I do on my own machine. It's hugely effective in avoiding the time spent in tracking down failures which turn out to failures in other libraries. This would result in a big improvement in boost. Robert Ramey
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 11/6/23 6:59 AM, Andrey Semashev via Boost wrote:
On 11/6/23 12:45, Alexander Grund via Boost wrote:
To be clear, if Boost as a whole stops using develop, it doesn't prohibit you from having a branch like that; you can even name it develop, if you like. That is, if master is frozen, you are free to continue your work in other branches.
Thats what I would probably do.
Of course, you would be testing your changes against master branches of the rest of Boost. I suspect, it wouldn't make much of a difference in terms of the "stability" of the Boost release.
This is what I do on my local machine. I think it would make a significant contribution of the quality of the boost release.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 11/6/23 1:45 AM, Alexander Grund via Boost wrote:
Am 06.11.23 um 01:09 schrieb Robert Ramey via Boost:
How so? It is a nice list with a red or green mark next to a short description of the configuration (e.g. OS, compiler version, CXX standard)
Hmmm - I'm not seeing this. Maybe I'm not doing something right or haven't invested enough time in it. and has the full logs
The test matrix on the other hand is missing easy access to logs and feedback on changes not yet merged to develop but merging without extensive tests risks breaking downstream libraries.
Right the test matrix only includes tests merged into the development branch.
The same CI runs against master branch for pushes or merges of/to master I tend to actually open a a PR from develop to master to verify nothing will be broken on master after the merge.
I see the merit in this observation. Personally, on my own machine, I test my development or feature branch agains the master branch of all the other libraries. It's the only way to know that when the changes are merged into the master the software will still work. See above: It is useful but not the only way and limitted in scope (not as many compilers/configurations tested as e.g. the Boost.CI template does)
Right, buts its my only option given the circumstances.
I agree here that this is a major issue and maybe the release managers can check (e.g. with Peters script) for that common issue and maybe even initiate merges.
Seems to me it would be easy to make a script which the release manager would run before release which would flag all libraries which have unmerged changes on the development branch dated before the current date. This might resolve the whole issue.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 11/6/23 03:09, Robert Ramey via Boost wrote:
On 11/5/23 1:10 PM, Andrey Semashev via Boost wrote:
With the adoption of git and CI, the relevance of the test matrix has diminished,
Personally, I would dispute that. I think that it's the only part of our testing infrastructure that still works.
and, presumably, people have shifted towards monitoring
their libraries' CI.
The CI for the boost serialization does not and never has worked. This is easy to veryify just by looking at any of the CI output for this library. And this demonstrable fact has been pointed out numerous times. This situation is never addressed.
I doubt that the serialization library is the only one with this issue.
The sadist part of all this is that even if it did "work" it would still be useless. It's not uncommon for a large and/or complex library to fail one or more tests on one or more compilers due to issues in the compiler itself. These can't be "fixed" in the library.
While it is true that tests sometimes fail because of the environment issues, this problem is not specific to CI; it also happens with our volunteer testers. But, with CI, it may be easier to work around - for example, by using a standard Docker image or installing missing packages, or configuring the environment in the CI scripts. With the test matrix, your only option is to contact the tester and try and solve the problem with him. Which may not be easy to do, since not all testers share their contacts. I remember this being a major problem back when I was relying on the test matrix. Now, if you mean compiler bugs and such that affect your library, then the failure is legitimate, as it will likely affect your users that also use that compiler. The solution here would be to implement workarounds in the library or, if not feasible, declare that compiler unsupported (preferably, in the library docs) and remove it from testing. Note that this course of action would be the same regardless whether you're using CI or the test matrix. Of course, there may be middle cases, when the compiler bug affects only a minor feature of the library, but the point remains - you either work around it in the feature implementation, or declare that the feature is not supported on that compiler and remove it from testing.
The test matrix shows all the tests x all the environments. One can easily see if any failure is general or isolated to a particular environment. The current CI just registers pass/fail for the whole library and all the environments.
The only benefit that the test matrix provides is the breakdown by individual tests. With CI, you typically have to search though the logs to know which test has failed. You do get the breakdown by jobs (which are equivalent to environments). It may not be presented the same way as in the matrix, and you may prefer one or the other presentation, but the information is there.
Some times someone will suggest skipping a particular test for a particular library so the CI comes of clean. This is basically hiding the error. Users considering using a library in their own enviroment are basically mislead that their the library works everywhere - which is demonstrably untrue. It's a fraud.
As I said above, it's not about hiding a failure and trying to deceive the user. It's about supporting or not supporting a configuration. A CI is supposed to test the configurations that you intend to support and highlight problems with those configurations, should they appear. The configurations you don't support need not be present in the CI, as you don't care whether they are failing or not. Furthermore, as a user, CI (or the test matrix) is not going to be the first place where I look to see whether the library supports my configuration. My first choice would probably be the documentation, and if it lacks the information, I will probably try the library myself. The reason why CI or the test matrix is not useful for this is because, as a user, I have no idea what the test results show. If a test is failing, I have no idea what it is testing, and why it is failing. Is the core functionality broken? Is it some corner case that I will never hit? Is it some intermittent failure e.g. due to network issues? Even discovering the testing configuration (compiler version, installed libraries, build options, etc.) may not be a trivial task for an outside user - even less trivial if the test results are in a less commonly used CI service. Bottom line is, CI (or the test matrix) is a tool primarily for the library maintainer, who can interpret its results and act accordingly. For me, as a maintainer, CI does a better job than the test matrix, even despite the shortcomings it has. And I'm not denying that there are shortcomings. However, CI vs. test matrix is a bit off-topic in this discussion.
Which means, people are less likely to notice that something broke downstream, unless the downstream developer notices and reports the problem. Although I have been that downstream developer myself quite a few times, I have to admit that this role of the develop branch of "an integration testing field" is not well filled in the current workflow. Noticing the breakage is mostly luck (i.e. a matter of me making a commit to my library and noticing the CI failure caused by an upstream dependency) rather than a rule.
A useful observation. The problem is that the current CI tests the development branch of one's library agains the develop branch of all the other libraries. So now my library looks like it "works" but when (and only when) it could fail when run against all master branches. So the testing shows pass when it actually fails when shipped with the master. Note that as we speak the test matrix for the master branch isn't working so it seems we never test the software which is actually being shipped.
Technically, you can configure CI to test however you like, including testing your feature branches and PRs against master. You can even test both against develop and master, if you like, although it will double the testing time. You write the CI scripts, including checkout commands, so you decide. However, as long as our workflow includes the develop branch, it doesn't make sense to test against master, as you will be merging your changes into develop, not master. If you want to test whether your changes will not break master, you could create a PR merging develop to master and run the CI on that, but that is separate from feature branches and other PRs that target develop.
Personally, on my own machine, I test my development or feature branch agains the master branch of all the other libraries. It's the only way to know that when the changes are merged into the master the software will still work.
But then you merge the feature branch to develop, right? In that case, you could be breaking downstream libraries in develop, or your own library in develop because some dependency of yours has changed in develop.
Andrey Semashev wrote:
However, CI vs. test matrix is a bit off-topic in this discussion.
One incredibly important advantage of CI over the test matrix is that it runs on pull requests. A pull request passing CI is not a guarantee of anything, but it's still miles better than nothing. This is also why it's important for CI to pass on the base branch (before the PR) - so that failures can be attributed to the proposed change and not to other factors. Yes, this all requires constant maintenance, and forces one to pay attention to environments one doesn't normally use for local testing, and to changes to other libraries one doesn't "use". Such is life. This incidentally is one of the functions of the develop branch - sometimes several separate PRs pass in isolation, but fail when simultaneously applied.
On 11/6/23 5:13 AM, Andrey Semashev via Boost wrote:
Now, if you mean compiler bugs and such that affect your library, then the failure is legitimate, as it will likely affect your users that also use that compiler.
The solution here would be to implement workarounds in the library or,
Boost libraries are only required to support standard C++ at the time they are first released. That's all. For a large library, it's a huge amount of work to address all the problems in all the compiler implementations. if not feasible, declare that compiler unsupported
(preferably, in the library docs) and remove it from testing.
Or just leave the failue in the test matrix - it's effectively self documenting.
Note that this course of action would be the same regardless whether you're using CI or the test matrix. Of course, there may be middle cases, when the compiler bug affects only a minor feature of the library, but the point remains - you either work around it in the feature implementation, or declare that the feature is not supported on that compiler and remove it from testing.
If one has an exhaustive suite of feature focused tests, the test matrix indicates which features don't work. Sadly, most libraries don't have enough tests to do this. I believe the serialization library is exceptional in this regard. One suggestion I received to address the CI issues was to cut down the number of tests and/or skip failing tests! Talk about blaming the victim!
The test matrix shows all the tests x all the environments. One can easily see if any failure is general or isolated to a particular environment. The current CI just registers pass/fail for the whole library and all the environments.
The only benefit that the test matrix provides is the breakdown by individual tests.
That is an indispensible benefit.
With CI, you typically have to search though the logs to know which test has failed. You do get the breakdown by jobs (which are equivalent to environments). It may not be presented the same way as in the matrix, and you may prefer one or the other presentation, but the information is there.
This is a hugely time consuming exercise! Actually, the proper response here is for the CI to provide test matrix like output.
Some times someone will suggest skipping a particular test for a particular library so the CI comes of clean. This is basically hiding the error. Users considering using a library in their own enviroment are basically mislead that their the library works everywhere - which is demonstrably untrue. It's a fraud.
As I said above, it's not about hiding a failure and trying to deceive the user.
LOL - it's hiding the failure to so the developer can convince himself that there are not issues.
It's about supporting or not supporting a configuration. A CI is supposed to test the configurations that you intend to support and highlight problems with those configurations, should they appear. The configurations you don't support need not be present in the CI, as you don't care whether they are failing or not.
I don't see it that way at all. I see it as display of the current status of the library in light of the environment we're stuck in. It's about providing visibility as to the current state of the library. We write to the standard - not to individual compiler quirks.
Furthermore, as a user, CI (or the test matrix) is not going to be the first place where I look to see whether the library supports my configuration.
why not? My first choice would probably be the documentation, Which is not generally maintained / updated due to test output. Especially when the test script has been updated to hide the failures. and
if it lacks the information, I will probably try the library myself.
Glad you have the time for that. The
reason why CI or the test matrix is not useful for this is because, as a user, I have no idea what the test results show. If a test is failing, I have no idea what it is testing, and why it is failing. Is the core functionality broken? Is it some corner case that I will never hit? Is it some intermittent failure e.g. due to network issues? Even discovering the testing configuration (compiler version, installed libraries, build options, etc.) may not be a trivial task for an outside user - even less trivial if the test results are in a less commonly used CI service.
Actually, the current test matrix has the facility whereby one can click on failing cell an it opens a new tab with the output from that particular test. Very quick and useful.
Bottom line is, CI (or the test matrix) is a tool primarily for the library maintainer, who can interpret its results and act accordingly.
which is not me.
For me, as a maintainer, CI does a better job than the test matrix, even despite the shortcomings it has. And I'm not denying that there are shortcomings. Good for you.
However, CI vs. test matrix is a bit off-topic in this discussion.
Hmmm - it is the discussion. My original point is that CI is useless for improving library quality. The test matrix is better but the scripts are not being maintained. As I've suggest elsewhere, maybe the solution is to a) make a script to be run before release which verifies the master and develop branches are in sync. b) OR maybe eliminate the develop branch entirely c) add test matrix like output to the CI
Technically, you can configure CI to test however you like, including testing your feature branches and PRs against master. You can even test both against develop and master, if you like, although it will double the testing time. You write the CI scripts, including checkout commands, so you decide.
The development over head now includes CI scripting. Add that to Git, github, b2 and CMake, documentation tools and assorted other tools one needs to keep up with it's a real burden on developers. No wonder we have difficulties attracting more boost developers!
However, as long as our workflow includes the develop branch, it doesn't make sense to test against master, as you will be merging your changes into develop, not master. If you want to test whether your changes will not break master, you could create a PR merging develop to master and run the CI on that, but that is separate from feature branches and other PRs that target develop.
But then you merge the feature branch to develop, right?
No. a) I test the feature branch on my own machine - no CI. Review test matrix like output. For boost dependencies I test against master. b) Then I merge feature brand onto my local develop branch - again using the master branch for other boost dependencies. c) Then I push the changes on my local copy of the develop branch to the git hub version. d) Watch the test matrix. Unfortunately, this tests agains the develop branch of other boost libraries. So sometimes this reveals a bug / dependency issue in another boost library. Then I have to harass the current maintainer - if there is one - or post a message here. In that case,
you could be breaking downstream libraries in develop, or your own library in develop because some dependency of yours has changed in develop.
I don't think I've ever had that happen - but maybe it has. I generally program to the published API so hopefully I'm not effected by changes in other libraries. However, sometimes the author/maintainer changes some some aspect of the API and catches me with my pants down. Usually, a gentle reminder address the problem. The two libraries I maintain have extensive test suites. Only very occasionaly have I had unintended API breakage. This is over the time of 22 years. The safe numerics library still has some issues that I'm aware of, but as it not widely used, they don't often come up. I'm working trying to fix them. Robert Ramey
On 11/6/23 20:30, Robert Ramey via Boost wrote:
On 11/6/23 5:13 AM, Andrey Semashev via Boost wrote:
Now, if you mean compiler bugs and such that affect your library, then the failure is legitimate, as it will likely affect your users that also use that compiler.
The solution here would be to implement workarounds in the library or,
Boost libraries are only required to support standard C++ at the time they are first released. That's all. For a large library, it's a huge amount of work to address all the problems in all the compiler implementations.
Supporting only strictly conforming compilers isn't practical, because there's no such compilers in the real world. If you want your library to be useful other than an academic exercise, you will have to face real users running real compilers, with all their bugs and missing C++ features.
if not feasible, declare that compiler unsupported
(preferably, in the library docs) and remove it from testing.
Or just leave the failue in the test matrix - it's effectively self documenting.
No, it's not. Not for an outside viewer.
The test matrix shows all the tests x all the environments. One can easily see if any failure is general or isolated to a particular environment. The current CI just registers pass/fail for the whole library and all the environments.
The only benefit that the test matrix provides is the breakdown by individual tests.
That is an indispensible benefit.
Evidently, I can manage without. In return I get much more valuable benefits, like faster turnaround, testing PRs and feature branches, notifications, and most importantly, being able to see the build/test logs (which were often missing last time I used the test matrix).
With CI, you typically have to search though the logs to know which test has failed. You do get the breakdown by jobs (which are equivalent to environments). It may not be presented the same way as in the matrix, and you may prefer one or the other presentation, but the information is there.
This is a hugely time consuming exercise!
Not really. Ctrl+F, type "...fail", it takes about 5 seconds for me.
Some times someone will suggest skipping a particular test for a particular library so the CI comes of clean. This is basically hiding the error. Users considering using a library in their own enviroment are basically mislead that their the library works everywhere - which is demonstrably untrue. It's a fraud.
As I said above, it's not about hiding a failure and trying to deceive the user.
LOL - it's hiding the failure to so the developer can convince himself that there are not issues.
That would be a pointless exercise, as he would discover many new things about himself and his library from his users rather soon. Or not, which is worse, because that would mean he has no users left.
It's about supporting or not supporting a configuration. A CI is supposed to test the configurations that you intend to support and highlight problems with those configurations, should they appear. The configurations you don't support need not be present in the CI, as you don't care whether they are failing or not.
I don't see it that way at all. I see it as display of the current status of the library in light of the environment we're stuck in. It's about providing visibility as to the current state of the library. We write to the standard - not to individual compiler quirks.
Again, if you want your libraries to be practically useful, you do need to take compiler quirks into account, however irritating is may be.
Furthermore, as a user, CI (or the test matrix) is not going to be the first place where I look to see whether the library supports my configuration.
why not?
My first choice would probably be the documentation,
Which is not generally maintained / updated due to test output. Especially when the test script has been updated to hide the failures.
and
if it lacks the information, I will probably try the library myself.
Glad you have the time for that.
The reason why CI or the test matrix is not useful for this is because, as a user, I have no idea what the test results show. If a test is failing, I have no idea what it is testing, and why it is failing. Is the core functionality broken? Is it some corner case that I will never hit? Is it some intermittent failure e.g. due to network issues? Even discovering the testing configuration (compiler version, installed libraries, build options, etc.) may not be a trivial task for an outside user - even less trivial if the test results are in a less commonly used CI service.
Actually, the current test matrix has the facility whereby one can click on failing cell an it opens a new tab with the output from that particular test. Very quick and useful.
Useful to whom? Certainly not to an outside user, who has no idea of your tests and what the failure signifies. At most, he gathers that your library doesn't work on that compiler, which may or may not be true.
Bottom line is, CI (or the test matrix) is a tool primarily for the library maintainer, who can interpret its results and act accordingly.
which is not me.
For me, as a maintainer, CI does a better job than the test matrix, even despite the shortcomings it has. And I'm not denying that there are shortcomings. Good for you.
However, CI vs. test matrix is a bit off-topic in this discussion.
Hmmm - it is the discussion. My original point is that CI is useless for improving library quality. The test matrix is better but the scripts are not being maintained.
As I've suggest elsewhere, maybe the solution is to a) make a script to be run before release which verifies the master and develop branches are in sync. b) OR maybe eliminate the develop branch entirely c) add test matrix like output to the CI
I would be ok with a), but this is one more burden on the release managers, who probably already have their plate full. I think, it would be better to have a more distributed solution that everyone would be able to use on their own, or some sort of automation. For b), it looks like people still find develop useful, so removing it probably won't be acceptable. As for c), the CI is an external service not under our control. I'm not sure how one would transform its UI to something like the test matrix. Perhaps, though its public API? In any case, it doesn't look trivial, and given that everyone (except you) seem content with the current UI, it is unlikely that this will be implemented any time soon.
Technically, you can configure CI to test however you like, including testing your feature branches and PRs against master. You can even test both against develop and master, if you like, although it will double the testing time. You write the CI scripts, including checkout commands, so you decide.
The development over head now includes CI scripting. Add that to Git, github, b2 and CMake, documentation tools and assorted other tools one needs to keep up with it's a real burden on developers. No wonder we have difficulties attracting more boost developers!
Well, you can't have a cake and eat it too. Nothing is free. Testing flexibility has its price too. Also, I do not think the requirement of knowledge of how to set up GitHub Actions CI is a significant factor for scaring off new developers; it's not a factor at all. People are much more likely to be familiar with GHA than with our test matrix, Boost.Build and review process.
However, as long as our workflow includes the develop branch, it doesn't make sense to test against master, as you will be merging your changes into develop, not master. If you want to test whether your changes will not break master, you could create a PR merging develop to master and run the CI on that, but that is separate from feature branches and other PRs that target develop.
But then you merge the feature branch to develop, right?
No.
Yet what you're saying below sounds like "yes".
a) I test the feature branch on my own machine - no CI. Review test matrix like output. For boost dependencies I test against master. b) Then I merge feature brand onto my local develop branch - again using the master branch for other boost dependencies. c) Then I push the changes on my local copy of the develop branch to the git hub version. d) Watch the test matrix. Unfortunately, this tests agains the develop branch of other boost libraries. So sometimes this reveals a bug / dependency issue in another boost library. Then I have to harass the current maintainer - if there is one - or post a message here.
On 11/6/23 12:53 PM, Andrey Semashev via Boost wrote:
But then you merge the feature branch to develop, right?
No.
Yet what you're saying below sounds like "yes".
LOL - of course you're right. I do merge the feature branch into develop. Soon before release, I merge develop branch into the release branch.
a) I test the feature branch on my own machine - no CI. Review test matrix like output. For boost dependencies I test against master. b) Then I merge feature brand onto my local develop branch - again using the master branch for other boost dependencies. c) Then I push the changes on my local copy of the develop branch to the git hub version. d) Watch the test matrix. Unfortunately, this tests agains the develop branch of other boost libraries. So sometimes this reveals a bug / dependency issue in another boost library. Then I have to harass the current maintainer - if there is one - or post a message here.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
The development over head now includes CI scripting. Add that to Git, github, b2 and CMake, documentation tools and assorted other tools one needs to keep up with it's a real burden on developers. No wonder we have difficulties attracting more boost developers!
Having CIs is an established practice, so not having them is very much likely gonna scare them off than having them. But yeah, you need to take care of CIs, and that takes time. IMHO that's part of what development implies currently.
On 11/6/23 2:04 PM, Ruben Perez via Boost wrote:
The development over head now includes CI scripting. Add that to Git, github, b2 and CMake, documentation tools and assorted other tools one needs to keep up with it's a real burden on developers. No wonder we have difficulties attracting more boost developers!
But yeah, you need to take care of CIs, and that takes time. IMHO that's part of what development implies currently.
Agreed. Regardless of whether it's established practice, I don't believe that it contributes much to boost quality. Actually, I've seen more than one shop where the build doesn't work, yet they run it every night regardless. Where the tests don't pass, and they ship anyway. So the fact that it's established practice by itself is not convincing to me. Robert Ramey
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Am 06.11.23 um 18:30 schrieb Robert Ramey via Boost:
Actually, the proper response here is for the CI to provide test matrix like output.
Given that we have a centralized repository with the CI scripts that most libraries use this is actually possible. I imagine it is easy enough to `tee` the b2-test output to a file, run some (Python?) script over it and append an overview of the status for each test to the end of the output. I guess having at least a list of failed tests in an easy-to-read text format improves the output. Unfortunately we can't (easily) generate HTML pages with tables etc. like the test matrix. But the above could at least improve the situation. Or maybe B2 could give this (already?). At least the CMake test output is easier to read having a list of tests, their status and on failure their output.
On Tue, Nov 7, 2023 at 2:03 AM Alexander Grund via Boost <boost@lists.boost.org> wrote:
Am 06.11.23 um 18:30 schrieb Robert Ramey via Boost:
Actually, the proper response here is for the CI to provide test matrix like output.
Given that we have a centralized repository with the CI scripts that most libraries use this is actually possible.
I imagine it is easy enough to `tee` the b2-test output to a file, run some (Python?) script over it and append an overview of the status for each test to the end of the output. I guess having at least a list of failed tests in an easy-to-read text format improves the output. Unfortunately we can't (easily) generate HTML pages with tables etc. like the test matrix. But the above could at least improve the situation.
Or maybe B2 could give this (already?). At least the CMake test output is easier to read having a list of tests, their status and on failure their output.
B2 can output an XML file of the tests it runs with a variety of information in it. Just run the usual test invocation and add an --out-xml=<file.xml>. There's a Python program <https://github.com/boostorg/regression/blob/develop/testing/src/process_jam_log.py> that can process that xml file and generate another xml file to be used by the current report generation (I wrote that maybe 2 decades ago?). But that program can trivially be changed to output HTML, or anything else like AsciiDoc for example. And, yes, it would be possible for B2 to inherently output a summary of tests at the bottom of a build. -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supone Nada -- Robot Dreams - http://robot-dreams.net
On 11/7/23 12:03 AM, Alexander Grund via Boost wrote:
Am 06.11.23 um 18:30 schrieb Robert Ramey via Boost:
Actually, the proper response here is for the CI to provide test matrix like output.
Given that we have a centralized repository with the CI scripts that most libraries use this is actually possible.
I imagine it is easy enough to `tee` the b2-test output to a file, run some (Python?) script over it and append an overview of the status for each test to the end of the output. I guess having at least a list of failed tests in an easy-to-read text format improves the output. Unfortunately we can't (easily) generate HTML pages with tables etc. like the test matrix. But the above could at least improve the situation.
Or maybe B2 could give this (already?). At least the CMake test output is easier to read having a list of tests, their status and on failure their output.
Or maybe just build and run this: https://github.com/robertramey/library_status To see what the output looks like: download the files library_status.html and links.html to some directory on your local machine and open library_status.html in with your web browser. (for some reason, the styling is coming up weird. Doesn't happen on my copy. I'll look into this) Robert Ramey
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey wrote:
The CI for the boost serialization does not and never has worked. This is easy to veryify just by looking at any of the CI output for this library. And this demonstrable fact has been pointed out numerous times. This situation is never addressed.
Looking at your last GHA build on develop https://github.com/boostorg/serialization/actions/runs/6845902041 which for some reason only includes Windows jobs, there's a failure on gcc, caused by the fact that the unqualified call to `visit` at https://github.com/boostorg/serialization/blob/a20c4d97c37e5f437c8ba78f29683... finds std::visit via ADL and tries to instantiate it, which fails. This looks like a bug in libstdc++ 8 to me, as std::visit is required to SFINAE, but it could just be that the standard at the time didn't require it. This is not in any way a failure of CI - test_variant doesn't compile under GCC 8, so it tells you so. What else do you expect to happen? And who and how is supposed to "address this situation"?
On 11/5/23 1:10 PM, Andrey Semashev via Boost wrote:
Then there's the recurring problem that led to this post - people forget to merge to master from time to time.
FWIW - I merge to develop when Marshall makes his announcement of the release date. So it's only once every 3 months. Not a huge burden. BUT it means that any breakage in other libraries due to changes in the serialization library aren't discover until it's late in process. Robert Ramey
On 11/6/23 06:40, Robert Ramey via Boost wrote:
On 11/5/23 1:10 PM, Andrey Semashev via Boost wrote:
Then there's the recurring problem that led to this post - people forget to merge to master from time to time.
FWIW - I merge to develop when Marshall makes his announcement of the release date. So it's only once every 3 months. Not a huge burden. BUT it means that any breakage in other libraries due to changes in the serialization library aren't discover until it's late in process.
I'm not sure I understand. Are you saying you merge new features to master rather than develop? If so, you may be causing problems to your downstream users, as they are doing the opposite.
On 11/6/23 6:51 AM, Andrey Semashev via Boost wrote:
On 11/6/23 06:40, Robert Ramey via Boost wrote:
On 11/5/23 1:10 PM, Andrey Semashev via Boost wrote:
Then there's the recurring problem that led to this post - people forget to merge to master from time to time.
FWIW - I merge to develop when Marshall makes his announcement of the release date. So it's only once every 3 months. Not a huge burden. BUT it means that any breakage in other libraries due to changes in the serialization library aren't discover until it's late in process.
I'm not sure I understand. Are you saying you merge new features to master rather than develop? If so, you may be causing problems to your downstream users, as they are doing the opposite.
I develop and merge into the develop branch. When marshal announces a release, I merge the develop branch into the master branch. What is confusing about this?
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 11/6/23 19:26, Robert Ramey via Boost wrote:
On 11/6/23 6:51 AM, Andrey Semashev via Boost wrote:
On 11/6/23 06:40, Robert Ramey via Boost wrote:
On 11/5/23 1:10 PM, Andrey Semashev via Boost wrote:
Then there's the recurring problem that led to this post - people forget to merge to master from time to time.
FWIW - I merge to develop when Marshall makes his announcement of the release date. So it's only once every 3 months. Not a huge burden. BUT it means that any breakage in other libraries due to changes in the serialization library aren't discover until it's late in process.
I'm not sure I understand. Are you saying you merge new features to master rather than develop? If so, you may be causing problems to your downstream users, as they are doing the opposite.
I develop and merge into the develop branch.
When marshal announces a release, I merge the develop branch into the master branch. What is confusing about this?
Above you wrote that you merge to *develop* on Marshall's announcement. If you meant *master* then then that's fine.
On 11/6/23 8:32 AM, Andrey Semashev via Boost wrote:
On 11/6/23 19:26, Robert Ramey via Boost wrote:
I develop and merge into the develop branch.
When marshal announces a release, I merge the develop branch into the master branch. What is confusing about this?
Above you wrote that you merge to *develop* on Marshall's announcement. If you meant *master* then then that's fine.
Right. My mistake. I merge into master from develop on Marshall's anouncement. This implies that: a) I'm only mucking with the master once in 3 mo or so. b) users/testers/etc. depending on the serialization library master branch have a stable platform during their own development. One of the thing's that drove me nuts when I developed using the develop branch for other libraries (boost.test, e.g.) was that the environment was always changing. I was constantly having failures due to other libraries in development. OK this is what the develop branch is for. But it makes it impossible to use in test. One can't run an experiment if he's changing all the variables simultaneously. Robert Ramey
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Sun, Nov 5, 2023 at 1:10 PM Andrey Semashev via Boost <boost@lists.boost.org> wrote:
Maybe there are other ideas how to avoid commits unmerged to master?
You avoid them by making sure to merge.
what do you think about creating a GitHub Action to automatically merge develop to master once the CI on develop succeeds
No I don't like that at all because it implies that a Boost-tier author or maintainer is not capable of the simple task of keeping master up to date with develop. If they are not capable of that, then I can't help but wonder what else they are incapable of. Will they forget to write unit tests, or add documentation for newly committed code? If you want automation, how about sending a reminder email "There are unmerged changes in develop which are at least a week old." Thanks
Vinnie Falco wrote:
what do you think about creating a GitHub Action to automatically merge develop to master once the CI on develop succeeds
No I don't like that at all because it implies that a Boost-tier author or maintainer is not capable of the simple task of keeping master up to date with develop. If they are not capable of that, then I can't help but wonder what else they are incapable of.
One empirical correlation is between maintainers who forget to merge to master and maintainers who merge broken PRs into develop. That's one of the current functions of the develop branch - to isolate the master branch (at least somewhat) from breakage. Yes, we do have a problem with fixes not getting to master, but what the people arguing against the current scheme don't realize is that if we drop it, we'll have different problems, and the end result will not necessarily be an improvement. (My prediction is "far from it.")
On 11/6/23 14:09, Vinnie Falco wrote:
On Sun, Nov 5, 2023 at 1:10 PM Andrey Semashev via Boost <boost@lists.boost.org> wrote:
Maybe there are other ideas how to avoid commits unmerged to master?
You avoid them by making sure to merge.
what do you think about creating a GitHub Action to automatically merge develop to master once the CI on develop succeeds
No I don't like that at all because it implies that a Boost-tier author or maintainer is not capable of the simple task of keeping master up to date with develop. If they are not capable of that, then I can't help but wonder what else they are incapable of. Will they forget to write unit tests, or add documentation for newly committed code?
I wouldn't go that far. We're all humans (as far as I'm aware), and most of us work on Boost on our spare time. Shit happens. I see nothing wrong about looking for a way to reduce the opportunities for shit to happen.
If you want automation, how about sending a reminder email "There are unmerged changes in develop which are at least a week old."
Yeah, that's one way to do this. But why not go one step further and just merge it?
On Sun, Nov 5, 2023 at 1:10 PM Andrey Semashev via Boost <boost@lists.boost.org> wrote:
The recent case of a commit in develop that was left unmerged to master made me wondering if people still find the develop branch useful.
I find it very useful for all my repositories. This is the standard procedure: * Work is submitted as a pull request * Once CI passes and reviewers approve it is merged to develop * At this point the work becomes available to third parties who perform testing on the develop branch * After a week or two has passed (varies depending on the changes) with no third-party complaints develop is merged to master The implicit contract with users is that the master branch will be safer for tracking work performed between releases. While the develop branch is suited for the collaboration of authors, maintainers, contributors, and code reviewers. Thanks
participants (8)
-
Alexander Grund -
Andrey Semashev -
John Maddock -
Peter Dimov -
René Ferdinand Rivera Morell -
Robert Ramey -
Ruben Perez -
Vinnie Falco