May 2023 - Boost - lists.preview.boost.org

Re: [boost] Full-Text Boost Search
by Vinnie Falco 10 May '23

10 May '23

On Wed, May 10, 2023 at 1:46 AM Antony Polukhin via Boost <boost(a)lists.boost.org> wrote: > Adding support of it to Doxygen should do the job of implementing both > searches and helping the C++ community in general. To add some context to what Mohammad is saying, I've been using Doxygen for years as part of my "docca" toolchain for building C++ library reference pages. For example: https://www.boost.org/doc/libs/1_82_0/libs/json/doc/html/json/ref/boost__js… I've reached the limit of what Doxygen is capable of and now it is blocking the improvement of documentation. Doxygen's XML output is always behind the HTML output, and there is no really good Asciidoc output. Furthermore Doxygen uses outdated technology for analyzing the C++ program. That is why I have created a new program called "Mr. Dox." This program uses clang's unstable libtooling API to generate an AST tree of a library's actual translation units (the .cpp files) and not just looking at headers using fancy regex or the simplified libclang API. Some discussion on that: https://www.foonathan.net/2019/11/standardese-post-mortem/ Because MrDox uses the exact same API that clang uses to actually compile your program, it can understand every C++ construct. If clang can compile it, then MrDox can see it. That means we can see deduction guides, alias templates, class template specializations, and much much more. The capability of extracting C++ source code features with perfect fidelity allows us to build features which enhance the generated reference in ways that Doxygen never can. For example we can recurse deeply into the AST and analyze the expression inside a conditional noexcept specification. We can understand abbreviated function template parameters. In fact, MrDox can eventually understand C++ features which are not yet part of the Standard but merged into the trunk of LLVM because they are supported by Technical Specifications ("TS") approved by the C++ Standardization Committee. This means MrDox can always stay current and keep pace with every innovation that comes to C++. The program is still in development but we are targeting the winter release date of Boost for having an packaged executable that is ready for production, with a good Asciidoc generator. Check it out here: https://github.com/cppalliance/mrdox Thanks! Vinnie

1 0

Re: [boost] Web documentation question
by Marshall Clow 10 May '23

10 May '23

> On May 10, 2023, at 2:43 AM, Alain O' Miniussi via Boost <boost(a)lists.boost.org> wrote: > > Hi, > > I sometime need to contribute to boost but it is clearly not a regular activity. > And each time it takes me an unreasonable amount of time just to find the relevant git clone command. > > I would like to know if something is inherently wrong with: > 1) go to www.boost.org > 2) unfold the "Development" menu on the right > 3) hope to find the information regarding the development cycle > > right now, these sections mostly contains pages with link the the wiki, which is expected to be at svn.boost.org which cannot be reached. > > If the information is actually at https://github.com/boostorg/wiki/wiki/Getting-Started%3A-Overview (thanks you stackoverflow) why isn't that page referenced from https://www.boost.org/development/index.html or https://github.com/boostorg/boost ? Thank you for bringing this to my (our?) attention. I am currently attending C++Now, and have no time to deal with this this week, but I have created https://github.com/boostorg/website/issues/754 <https://github.com/boostorg/website/issues/754> to track this so it will not be lost. — Marshall

1 0

Re: [boost] Full-Text Boost Search
by Antony Polukhin 10 May '23

10 May '23

On Tue, May 9, 2023, 17:16 Mohammad Nejati [Ashtum] via Boost < boost(a)lists.boost.org> wrote: <...> > Thank you for your time, and I look forward to your suggestions and > feedback. > There's a way to implement fulltext search for the whole C++ community and make the C++ docs much better worldwide. The Doxygen tool for building the docs from C++ sources has a long standing issue to implement fulltext client search https://github.com/doxygen/doxygen/issues/4111 Recently (at leats for me) a JS library appeard. With Lunar.js you could do server side searches with node.js and client side searches. Adding support of it to Doxygen should do the job of implementing both searches and helping the C++ community in general. >

2 1

Web documentation question
by Alain O' Miniussi 10 May '23

10 May '23

Hi, I sometime need to contribute to boost but it is clearly not a regular activity. And each time it takes me an unreasonable amount of time just to find the relevant git clone command. I would like to know if something is inherently wrong with: 1) go to www.boost.org 2) unfold the "Development" menu on the right 3) hope to find the information regarding the development cycle right now, these sections mostly contains pages with link the the wiki, which is expected to be at svn.boost.org which cannot be reached. If the information is actually at https://github.com/boostorg/wiki/wiki/Getting-Started%3A-Overview (thanks you stackoverflow) why isn't that page referenced from https://www.boost.org/development/index.html or https://github.com/boostorg/boost ? Thanks ---- Alain Miniussi DSI, Pôles Calcul et Genie Log. Observatoire de la Côte d'Azur Tél. : +33492003009 (Mont-Gros) +33483618544 (Sophia Antipolis) +33609650665

1 0

Re: [boost] The Future of Boost - CI
by René Ferdinand Rivera Morell 10 May '23

10 May '23

On Tue, May 9, 2023 at 4:21 PM Vinnie Falco via Boost <boost(a)lists.boost.org> wrote: > > On Tue, May 9, 2023 at 2:04 PM Tom Kent via Boost <boost(a)lists.boost.org> wrote: > > I'm happy to go with whatever the community is looking to do on this front. > > Some options as I see it: > > > > * Keep the current test matrix as a compliment to the various CI systems > > starting to roll out. > > * Merge my runner system in with the new C++ Alliance CI system > > * Shutdown the test matrix and move fully to C++ Alliance cloud CI > > We haven't put any thought or effort into how our CI might interface > with the test matrix, so I think for now the best strategy is no > strategy: Keep things as they are. > > But... since we are talking about it, my inner tech geek is tickled by > the idea of a decentralized network of runners using authors', > maintainers', and contributors' already existing commodity hardware to > run a specially prepared docker container that tests some > configurations of boost libraries and reports the results to a central > server. > > While the execution of the current test matrix is somewhat flawed, I > think the idea behind it is actually rather brilliant. In exchange for > much longer turnaround times we get a depth of coverage that cannot be > matched on cloud systems. And it is basically "free" since the > computing model is inverted: the expensive compilation and testing is > performed by numerous cheap developer machines with spare cycles, > while the cheap collation and presentation of the results is handled > by the cloud. > > If we are going to improve the test matrix I think the way to do it is > to treat it like any other software project. Assemble a team and start > writing maintainable code with documentation. We probably want to use > some off the shelf solution for deploying runners on people's PCs and > laptops, and then write our own set of scripts and dockerfiles (?) > which spread around the testing of all the different configurations of > the libraries. > > The most important part of this software project is to address the > headaches of the current system: when there is a failure, it is > difficult (and often impossible) to diagnose from the logs. There are > many false positives and noisy things like warnings which have no > possible resolution, and so on. In other words we will need to be able > to open issues on the test matrix software, triage them, get the fixes > for them in, have them tested, and then deploy a new version just like > every other software project. Once upon a time, while I was still the Testing Manager, I had the same idea and partially implemented it to run on Google cloud data services. But I just ran out of time and interest after that thing that happened. -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supone Nada -- Robot Dreams - http://robot-dreams.net

1 0

Re: [boost] The Future of Boost
by Vladimir Batov 10 May '23

10 May '23

Vinnie, thank you for putting it all together and sharing. Interesting, educational, entertaining and sobering read. Your enthusiasm and work are encouraging, commendable and promising. Very impressive and hugely appreciated. 30 years ago I'd rolled the sleeves up and jumped in. Now I wish you and your efforts all the success and the very best. Will keep fingers crossed. Thanks again. On 2023-05-09 01:46, Vinnie Falco via Boost wrote: > Greetings! > > I'm Vinnie Falco, Boost library author, C++ enthusiast, and the > founder of The C++ Alliance, a 501(c)(3) non-profit...

2 1

Re: [boost] The Future of Boost - CI
by Robert Ramey 09 May '23

09 May '23

On 5/9/23 2:03 PM, Tom Kent via Boost wrote: Let me say that you're efforts are very, very much appreciated. This is the only testing system which has worked (more or less) reliably over years. It is the only thing I feel I can actually trust. How long have you been doing this? Seems to me that it's been since I made the serialization library in 2003? Was it you that set this up and has kept it up for 20 years? It's tribute to your capability that it works so invisibility that we don't even know much about who you are. > On Mon, May 8, 2023 at 5:22 PM Andrey Semashev via Boost < > boost(a)lists.boost.org> wrote: > >> On 5/8/23 20:40, John Maddock via Boost wrote: >> >>> Machine time could well be donated by volunteers and perhaps replace the >>> current test/status matrix, which is fine, but requires you to go off >>> seeking for results, which may or may not have cycled yet. Plus that >>> matrix relies on a "build the whole of Boost" approach which >>> increasingly simply does not scale. >> >> I'm really grateful to the volunteers that run the tests and maintain >> the official test matrix, but honestly, I'm not paying attention to it >> anymore. I have three main issues with it: >> >> 1. Slow turnaround. From my memory, it could take weeks or more for the >> runners to run the tests over a commit I made. With this order of times, >> it is impossible to perform continued development while maintaining code >> in working state. >> >> 2. Lack of notifications. >> >> 3. Problematic debugging. It was not uncommon that a test run failed >> because of some misconfiguration on the runner's side. And it was also >> not uncommon that build logs were unavailable. >> >> So, while, again, I'm most grateful to the people that made public >> testing possible at all before we had the current CI services, today we >> do have CI services with the above problems fixed (more or less), and >> I'm spoiled by them. It is true that the public CI resources are limited >> and insufficient at times, so, IMHO, the way forward would be towards >> fixing this problem without losing the convenience of the CI services >> we've become used to. >> > > As the person responsible for most (currently all) of the test runners in > the official test matrix, I thought I'd throw a couple cents in here. > > I currently have three machines running these tests. These machines were > purchased by boost (then from SFC) in 2017, and are a bit old but still > running well. > > One of the machines is running windows server, it cycles running through > the last six versions of visual studio (plus the latest one with > `/std:c++latest`) for develop + master. > > The two other machines are linux runners. One of them just runs the latest > version of clang + gcc for develop + master, at about 2hr a pop there > should never be a commit that goes more than ~8hr without one of these > configurations running. > > The other machine runs a *huge* number (>150) of gcc/clang configs. This > takes approximately a week to get through all of them, but provides a > breadth of testing that isn't available anywhere else. See the table in > this readme [1] for the full list, as well as a bit more about the runners. > > To address a bit of #3 above, these are fully running on docker containers, > I'd be more than happy to add other users configs to this list. > > I've also got a couple raspberry pi machines running gcc + clang / > develop + master, but they are *very* slow (20hrs?). I've also got a > RISC-V SBC that I want to get this going on, but haven't found the time yet. > Architecture *shouldn't* matter for most of what we do, but there are a > couple edge cases where it can be useful. > > > > I don't spend a lot of time on caring and feeding of this, so would be > happy to keep it going in the future...but the down side of that is that > there have been (and I'm sure will continue to be) instances where things > break and go unnoticed for weeks. (I just noticed MSVC has been failing > all the develop runs for weeks, almost certainly a config issue on my end) > I wouldn't think this would be too difficult to address. Maybe we can get someone to write some sort of watchdog, ... if it doesn't exist already. > > All that said, I'm not sure where this fits into the picture going forward. > > Andrey's points above (esp. 2&3) are very valid. I wouldn't depend on a CI > system like this in my day-job. > > There are plenty more issues with the system as well...specifically: > * Tons of time wasted re-building things that haven't changed (sometimes > for years!) > * No history saved beyond the most recent build > * Processing of results a centralized point of failure > > > > I'm happy to go with whatever the community is looking to do on this front. > Some options as I see it: > > * Keep the current test matrix as a compliment to the various CI systems > starting to roll out. > * Merge my runner system in with the new C++ Alliance CI system > * Shutdown the test matrix and move fully to C++ Alliance cloud CI > Contrary to information reported on this list, the current CI system does not work. For typical example see: https://github.com/boostorg/serialization/pull/282 So your work is currently irreplaceable. If we have resources, I'd like to see some enhancements in the current test matrix.. a) I made my own version of the test matrix html generator which organizes the test results by compiler/static vs dynamic/debug vs release, etc. etc. https://github.com/robertramey/library_status#library_status It's derived from the current test matrix display code. I'd like to see it evaluated as a replacement. b) accumulate past test results for a time. Permit them to be queried by compiler, date range, static vs dynamic, etc... c) Permit test results run on user machines to be uploaded to the results database. Note that b) and c) are what Cmake test does and it's very useful. This permits test results on configurations actually being used to be included in the matrix. Of course I'm not expecting you to do this. I'm suggesting that "someone" do this this. Thanks again for all your work. Robert Ramey > Tom > > > [1]https://github.com/teeks99/boost-build/blob/master/Regression/README.md > > _______________________________________________ > Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

1 0

Re: [boost] The Future of Boost - CI
by Vinnie Falco 09 May '23

09 May '23

On Tue, May 9, 2023 at 2:04 PM Tom Kent via Boost <boost(a)lists.boost.org> wrote: > I'm happy to go with whatever the community is looking to do on this front. > Some options as I see it: > > * Keep the current test matrix as a compliment to the various CI systems > starting to roll out. > * Merge my runner system in with the new C++ Alliance CI system > * Shutdown the test matrix and move fully to C++ Alliance cloud CI We haven't put any thought or effort into how our CI might interface with the test matrix, so I think for now the best strategy is no strategy: Keep things as they are. But... since we are talking about it, my inner tech geek is tickled by the idea of a decentralized network of runners using authors', maintainers', and contributors' already existing commodity hardware to run a specially prepared docker container that tests some configurations of boost libraries and reports the results to a central server. While the execution of the current test matrix is somewhat flawed, I think the idea behind it is actually rather brilliant. In exchange for much longer turnaround times we get a depth of coverage that cannot be matched on cloud systems. And it is basically "free" since the computing model is inverted: the expensive compilation and testing is performed by numerous cheap developer machines with spare cycles, while the cheap collation and presentation of the results is handled by the cloud. If we are going to improve the test matrix I think the way to do it is to treat it like any other software project. Assemble a team and start writing maintainable code with documentation. We probably want to use some off the shelf solution for deploying runners on people's PCs and laptops, and then write our own set of scripts and dockerfiles (?) which spread around the testing of all the different configurations of the libraries. The most important part of this software project is to address the headaches of the current system: when there is a failure, it is difficult (and often impossible) to diagnose from the logs. There are many false positives and noisy things like warnings which have no possible resolution, and so on. In other words we will need to be able to open issues on the test matrix software, triage them, get the fixes for them in, have them tested, and then deploy a new version just like every other software project. Thanks

1 0

Re: [boost] The Future of Boost - CI
by Tom Kent 09 May '23

09 May '23

On Mon, May 8, 2023 at 5:22 PM Andrey Semashev via Boost < boost(a)lists.boost.org> wrote: > On 5/8/23 20:40, John Maddock via Boost wrote: > > > Machine time could well be donated by volunteers and perhaps replace the > > current test/status matrix, which is fine, but requires you to go off > > seeking for results, which may or may not have cycled yet. Plus that > > matrix relies on a "build the whole of Boost" approach which > > increasingly simply does not scale. > > I'm really grateful to the volunteers that run the tests and maintain > the official test matrix, but honestly, I'm not paying attention to it > anymore. I have three main issues with it: > > 1. Slow turnaround. From my memory, it could take weeks or more for the > runners to run the tests over a commit I made. With this order of times, > it is impossible to perform continued development while maintaining code > in working state. > > 2. Lack of notifications. > > 3. Problematic debugging. It was not uncommon that a test run failed > because of some misconfiguration on the runner's side. And it was also > not uncommon that build logs were unavailable. > > So, while, again, I'm most grateful to the people that made public > testing possible at all before we had the current CI services, today we > do have CI services with the above problems fixed (more or less), and > I'm spoiled by them. It is true that the public CI resources are limited > and insufficient at times, so, IMHO, the way forward would be towards > fixing this problem without losing the convenience of the CI services > we've become used to. > As the person responsible for most (currently all) of the test runners in the official test matrix, I thought I'd throw a couple cents in here. I currently have three machines running these tests. These machines were purchased by boost (then from SFC) in 2017, and are a bit old but still running well. One of the machines is running windows server, it cycles running through the last six versions of visual studio (plus the latest one with `/std:c++latest`) for develop + master. The two other machines are linux runners. One of them just runs the latest version of clang + gcc for develop + master, at about 2hr a pop there should never be a commit that goes more than ~8hr without one of these configurations running. The other machine runs a *huge* number (>150) of gcc/clang configs. This takes approximately a week to get through all of them, but provides a breadth of testing that isn't available anywhere else. See the table in this readme [1] for the full list, as well as a bit more about the runners. To address a bit of #3 above, these are fully running on docker containers, I'd be more than happy to add other users configs to this list. I've also got a couple raspberry pi machines running gcc + clang / develop + master, but they are *very* slow (20hrs?). I've also got a RISC-V SBC that I want to get this going on, but haven't found the time yet. Architecture *shouldn't* matter for most of what we do, but there are a couple edge cases where it can be useful. I don't spend a lot of time on caring and feeding of this, so would be happy to keep it going in the future...but the down side of that is that there have been (and I'm sure will continue to be) instances where things break and go unnoticed for weeks. (I just noticed MSVC has been failing all the develop runs for weeks, almost certainly a config issue on my end) All that said, I'm not sure where this fits into the picture going forward. Andrey's points above (esp. 2&3) are very valid. I wouldn't depend on a CI system like this in my day-job. There are plenty more issues with the system as well...specifically: * Tons of time wasted re-building things that haven't changed (sometimes for years!) * No history saved beyond the most recent build * Processing of results a centralized point of failure I'm happy to go with whatever the community is looking to do on this front. Some options as I see it: * Keep the current test matrix as a compliment to the various CI systems starting to roll out. * Merge my runner system in with the new C++ Alliance CI system * Shutdown the test matrix and move fully to C++ Alliance cloud CI Tom [1]https://github.com/teeks99/boost-build/blob/master/Regression/README.md

1 0

Re: [boost] The Future of Boost - CI
by Boris Kolpackov 09 May '23

09 May '23

At the risk of sounding like a broken record (if nothing else, this could serve as a confirmation that these things are indeed possible): John Maddock via Boost <boost(a)lists.boost.org> writes: > * The library author does nothing more than list (maybe in json or something > similarly popular) the systems they want to test on - and these would be > abstract Boost-wide names like "core", "bleeding edge", "legacy-msvc" etc. Yes, we call them "target configuration classes", such as: latest: https://ci.cppget.org/?build-configs=latest default: https://ci.cppget.org/?build-configs=default legacy: https://ci.cppget.org/?build-configs=legacy You can intersect them with other classes (platforms, compiler, debug/release etc) to end up with something like your "legacy-msvc": https://build2.org/stage/bpkg/doc/build2-package-manager-manual.xhtml#manif… > * Updates and changes to CI would be announced here first, if in doubt new > compilers *might* go in "bleeding-edge" first and "core" a release later. Right, our current policy, for example, is to retain at least two latest versions of each compiler on each platform with the latest version tested in the optimized, ndebug, and static variants. Andrey Semashev via Boost <boost(a)lists.boost.org> writes: > While I can see the appeal of treating the CI as a black box, I > personally want the opposite. That is, I want to know *exactly* what I'm > testing and how I'm testing. This goes beyond listing specific compilers > or even their versions; it is not uncommon for me to configure > environment, including installing additional packages or setting up > environment variables and compiler switches as part of the CI run. I don't think one necessarily precludes the other. In our case you get the vanilla build by default but you can do all of the above declaratively in the manifest (well, perhaps except the environment variables, but we could add that): https://build2.org/stage/bpkg/doc/build2-package-manager-manual.xhtml#manif… > 2. Lack of notifications. We have email notifications, which can be configured (per package) for all build results, warning or error only, or error only: https://build2.org/stage/bpkg/doc/build2-package-manager-manual.xhtml#manif… > 3. Problematic debugging. It was not uncommon that a test run failed > because of some misconfiguration on the runner's side. And it was also > not uncommon that build logs were unavailable. We have interactive CI where you can login into the machine and investigate the issue directly.

1 0