At the risk of sounding like a broken record (if nothing else, this could serve as a confirmation that these things are indeed possible): John Maddock via Boost <boost@lists.boost.org> writes:
* The library author does nothing more than list (maybe in json or something similarly popular) the systems they want to test on - and these would be abstract Boost-wide names like "core", "bleeding edge", "legacy-msvc" etc.
Yes, we call them "target configuration classes", such as: latest: https://ci.cppget.org/?build-configs=latest default: https://ci.cppget.org/?build-configs=default legacy: https://ci.cppget.org/?build-configs=legacy You can intersect them with other classes (platforms, compiler, debug/release etc) to end up with something like your "legacy-msvc": https://build2.org/stage/bpkg/doc/build2-package-manager-manual.xhtml#manife...
* Updates and changes to CI would be announced here first, if in doubt new compilers *might* go in "bleeding-edge" first and "core" a release later.
Right, our current policy, for example, is to retain at least two latest versions of each compiler on each platform with the latest version tested in the optimized, ndebug, and static variants. Andrey Semashev via Boost <boost@lists.boost.org> writes:
While I can see the appeal of treating the CI as a black box, I personally want the opposite. That is, I want to know *exactly* what I'm testing and how I'm testing. This goes beyond listing specific compilers or even their versions; it is not uncommon for me to configure environment, including installing additional packages or setting up environment variables and compiler switches as part of the CI run.
I don't think one necessarily precludes the other. In our case you get the vanilla build by default but you can do all of the above declaratively in the manifest (well, perhaps except the environment variables, but we could add that): https://build2.org/stage/bpkg/doc/build2-package-manager-manual.xhtml#manife...
2. Lack of notifications.
We have email notifications, which can be configured (per package) for all build results, warning or error only, or error only: https://build2.org/stage/bpkg/doc/build2-package-manager-manual.xhtml#manife...
3. Problematic debugging. It was not uncommon that a test run failed because of some misconfiguration on the runner's side. And it was also not uncommon that build logs were unavailable.
We have interactive CI where you can login into the machine and investigate the issue directly.