Interest for C++20 modules support of boost officially
Hi Boost Developers, I am Chuanqi Xu, the maintainer for modules & serialization in clang and also a fan of C++20 modules. The post itself was motivated by a discussion (https://cpplang.slack.com/archives/C27KZLB0X/p1734335425348809 <https://cpplang.slack.com/archives/C27KZLB0X/p1734335425348809 >). You don’t need to read it if you’re not interested. I’ll try to make the post itself self contained. The original story comes from a high quality experiment for C++20 modules in boost (https://anarthal.github.io/cppblog/modules2 <https://anarthal.github.io/cppblog/modules2 >). The blog reads pretty good to me. However, anarthal (the author) told me later that the process of supporting modules in boost got suspended since the authors (or the maintainers) don’t want it since no one is going to use them. And if I want it, I need to send a mail here. Then this is the beginning of the story. First of first, I want to emphasize that “no one is going to use them” is not true. We’ve already used C++20 modules at larger scale internally for roughly a year. For boost (and other third party libraries), we choose to wrap a module ourselves as a workaround (with some compiler extensions to ease the refacotoring, but we don’t like it. We still want standard behavior). We didn’t reach out since we feel boost will support C++20 modules later or sooner so we don’t need to do anything specially. (Maybe we were wrong. We didn’t understand the ecosystem enough). Until yesterday, I just knew that boost didn’t know such needs. Before I wrote this email, I wondered a lot of words to emphasize that our need are a good example and the users of closed source are important users too. But just now, I searched the use of boost in modules in GitHub. I just realized I can show it (the importance of C++20 modules support for boost) much easily with the open source world. I searched `boost path:*.cppm` in GitHub and got the following list: https://github.com/infiniflow/infinity/blob/fa922afd58c973a9a317d2c09ad8cea8... <https://github.com/infiniflow/infinity/blob/fa922afd58c973a9a317d2c09ad8cea8... > https://github.com/ossia/score/blob/0ce6f62a97651a1248a92b3e13ce6ede10341d1f... <https://github.com/ossia/score/blob/0ce6f62a97651a1248a92b3e13ce6ede10341d1f... > https://github.com/RichardLuo0/make-dot-cpp/blob/e2ef10fd43060a2b56fc8f87f14... <https://github.com/RichardLuo0/make-dot-cpp/blob/e2ef10fd43060a2b56fc8f87f14... > https://github.com/slaakko/cmajor-mod/blob/32e6eb8dc099b395fa751340de96f4781... <https://github.com/slaakko/cmajor-mod/blob/32e6eb8dc099b395fa751340de96f4781... > https://github.com/df-com/dragonfly/blob/9dfcbc1153f292eb8581f1df7459600cc33... <https://github.com/df-com/dragonfly/blob/9dfcbc1153f292eb8581f1df7459600cc33... > https://github.com/nofe1248/Helium/blob/92ec4452b8c05e6d86c972f843196a4392a7... <https://github.com/nofe1248/Helium/blob/92ec4452b8c05e6d86c972f843196a4392a7... > https://github.com/LagrangeDev/liblagrange/blob/10e3b48f8d47fd65de00b14cfed6... <https://github.com/LagrangeDev/liblagrange/blob/10e3b48f8d47fd65de00b14cfed6... > https://github.com/yudaichen/feature-db/blob/e94ade969af682ed5dd25d70bf282ba... <https://github.com/yudaichen/feature-db/blob/e94ade969af682ed5dd25d70bf282ba... > This is not a complete list even for my search only and I believe there are other patterns to use boost with modules now (e.g., at least `*.cppm` is not the defacto suffix). And I also believe, from my experience as a C++ user, if boost provide modules, I may try to use it. But if boost didn’t provide it, maybe I won’t ask for it if I don’t want modules super bad. Since modules doesn’t add any new functionality actually. So most users may not ask for it explicitly. But if boost provides it, people may give it a try. The above text are the body of the post. I hope it can show it is needed. — Following of are some technical discussion for the extra maintainance burden by introducing modules. The conclusion is, I think, the burden is pretty low. Since modules are not complex to users. Different from other big features in C++, modules are pretty easy to understand. The complexity of modules are majorly on tools and the ecosystem. For programmers who don’t get involved to much to the ecosystem (or infrastructure), modules should be pretty easy to understand. For authors of libraries, after boost introduce modules, you only need to do 2 extra things: - In headers, add a macro (assume its name as BOOST_MODULE_EXPORT, which will be ‘export’ conditionally) to the entities that need to be exported. - In headers, include standard headers and other dependent headers conditionally so that we don’t include anything if the headers are in a module unit. e.g., https://github.com/boostorg/pfr/blob/f09e6aeae9d050897fff72b93d5f5e866cc5e11... <https://github.com/boostorg/pfr/blob/f09e6aeae9d050897fff72b93d5f5e866cc5e11... > then that’s roughly all you need for maintaining it at source level. For building and distributing related things, anarthal told me that the boost will handle it globally so the individual arthors don’t need to care about it. — In slack, there are some other discussion about module names and linking problems (https://cpplang.slack.com/archives/C27KZLB0X/p1734345716728609 <https://cpplang.slack.com/archives/C27KZLB0X/p1734345716728609 >). But I feel they are implementation details. I may not talk too much for that as a user at least now. And also there are opinions for configuration flags. These are the jobs of build systems (cmake is working on it). I think boost don’t need to worry for that. Similarly, at least libc++ has similar problems but libc++ still released std module. — Boost has a special position in the C++’s ecosystem. The most simple and impressive answer I’ve heard for the question “when can we start to use modules” is, “when boost makes it”. Thanks, Chuanqi
On Tue, Dec 17, 2024 at 11:28 AM Chuanqi Xu via Boost <boost@lists.boost.org> wrote:
Boost has a special position in the C++’s ecosystem. The most simple and impressive answer I’ve heard for the question “when can we start to use modules” is, “when boost makes it”.
I just want to +1 this. Boost using modules would go a long way for increasing adoption. Boost is a core dependency in many projects and this would certainly unblock many initiatives.
Boost has a special position in the C++’s ecosystem. The most simple and impressive answer I’ve heard for the question “when can we start to use modules” is, “when boost makes it”.
This seems like a good candidate for a GSOC project. It would shift the load of a number of disparate maintainers to a one, or a few people that could solely focus on getting it correct. Most of the groundwork for adaptation in Boost already exists thanks to Reuben so it shouldn't require a huge amount of discovery. Matt
On Tue, Dec 17, 2024 at 2:35 PM Matt Borland via Boost < boost@lists.boost.org> wrote:
. Most of the groundwork for adaptation in Boost already exists thanks to Reuben so it shouldn't require a huge amount of discovery.
Hi, do you know if this is tracked somewhere? I know are this 2 links https://anarthal.github.io/cppblog/modules https://anarthal.github.io/cppblog/modules2 If I mixed up people here my apologies, I presumed that by Reuben you meant Ruben Perez.
Hi, do you know if this is tracked somewhere? I know are this 2 links https://anarthal.github.io/cppblog/modules https://anarthal.github.io/cppblog/modules2
If I mixed up people here my apologies, I presumed that by Reuben you meant Ruben Perez.
Yes, Reuben Perez with those posts and associated repos. He may have more that I am not aware of, but he'll likely respond to this thread. As John Maddock said he has an experimental branch on Regex and there's also one he put on Math. See: https://github.com/boostorg/math/pull/783 Matt
On Tue, 17 Dec 2024 at 15:27, Matt Borland via Boost <boost@lists.boost.org> wrote:
Hi, do you know if this is tracked somewhere? I know are this 2 links https://anarthal.github.io/cppblog/modules https://anarthal.github.io/cppblog/modules2
If I mixed up people here my apologies, I presumed that by Reuben you meant Ruben Perez.
Yes, Reuben Perez with those posts and associated repos. He may have more that I am not aware of, but he'll likely respond to this thread. As John Maddock said he has an experimental branch on Regex and there's also one he put on Math. See: https://github.com/boostorg/math/pull/783
Yes, it's actually just me - Ruben Perez (anarthal in slack). There are just these two posts. No follow ups yet. Reuben is actually a misspelling of my name. Hope this helps solve the confusion.
> This seems like a good candidate for a GSOC project. It would shift the load of a number of disparate maintainers to a one, or a few people that could solely focus on getting it correct. Most of the groundwork for adaptation in Boost already exists thanks to Reuben so it shouldn't require a huge amount of discovery. Yeah, it sounds really nice. Would you like to plan one? I feel anarshal is qualified to take the mentor's position. And I like to help if needed. > Would have been nice if the `export` keyword was accepted and silently ignored when outside of module. Yes, but we have missed the train. > but in the general case you'd have something like > > #ifdef BOOST_USE_MODULE_CORE > import boost.core; > #else > #include <boost/core/this.hpp> > #include <boost/core/that.hpp> > #endif > > repeated for every library you #include things from. Yes, exactly. And in fact, this pattern is relatively trivial. I wrote a tool to do such conversions (https://github.com/ChuanqiXu9/clang-modules-converter <https://github.com/ChuanqiXu9/clang-modules-converter >). This tool would only participate the process of rewriting so it won’t be put into boost’s ecosystem. People can also make their own tool since the problem in source level is trivial, simple and common actually. > And the end user will have to do this too. Maybe not for all. See the list of github links I sent. I believe in these links users can replace `#include` to import for boost simply. (I know what you’re saying. Just to say not all users need to do that.) > I'm also not quite clear on how things will work on the CMake side. To be honest, I don’t have an insight here. Our use experience was based on a modified bazel, not cmake. And also our code is in a monorepo in a closed world. It might be different in the open world. I CCed the CMake dev for modules. I guess he is willing to help. > Also, we're basically talking Clang-only here, right? GCC doesn't work and MSVC... may or may not work? From the discussion thread, I think we’re talking about standard C++20 modules. We’re not talking about any extension or compiler specific behaviors. So I believe what we plan to do here is fine. Even if some compilers are not able to make it now, I believe it will someday. > Some compilers do *not* implement GMF entity pruning correctly, or at all. This will lead to disappointed and/or frustrated users, or - even worse - to users who are misled to assume seeing correct module BMIs, while these are - in fact - incorrect. This was one the topics in my modules workshop this fall (hint: stay away from Clang at least up to 18. Clang 19 *might* have improved in this regard). The recommended remedy is to avoid #includes of standard library headers at all and turn to import std; instead. But this is just the starting point. My take away for boost guys is, when we make boost modular, we’d better to assume `import std;` is available. Or boost don’t need to care the case: let’s provide boost modules without std module. Thanks, Chuanqi ------------------------------------------------------------------ From:Matt Borland <matt@mattborland.com> Send Time:2024 Dec. 17 (Tue.) 21:35 To:boost<boost@lists.boost.org> Cc:Chuanqi<chuanqi.xcq@alibaba-inc.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially > Boost has a special position in the C++’s ecosystem. The most simple and impressive answer I’ve heard for the question “when can we start to use modules” is, “when boost makes it”. > This seems like a good candidate for a GSOC project. It would shift the load of a number of disparate maintainers to a one, or a few people that could solely focus on getting it correct. Most of the groundwork for adaptation in Boost already exists thanks to Reuben so it shouldn't require a huge amount of discovery. Matt
On Tue, Dec 17, 2024 at 9:19 PM Chuanqi Xu via Boost <boost@lists.boost.org> wrote:
Would have been nice if the `export` keyword was accepted and silently ignored when outside of module. Yes, but we have missed the train.
There are more trains. And there's apparently still time for the C++23 train. This is something that would be entirely possible to propose now as an improvement for adoptability of modules. Who wants to write a proposal? Actually, we don't even need a train. Implementations could do that now without any ill effects (and even backported). And use it as implementation experience. What do you think Chuanqi of implementing that in clang? -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supongas Nada -- Robot Dreams - http://robot-dreams.net
Hi René, Interesting, I didn’t think about it actually. On the one hand, it is surprising to me to heard we don’t miss C++23 train. On the other hand, the idea to implement it in Clang without the proposal in WG21 looks like pandora’s box to me. If we did the second point, the code accepted by clang may not be accepted by other compilers. Although it happens now, we don’t want it to be the case. Further more, I feel it makes the position of WG21 to be in a pretty embrassive position. Thanks, Chuanqi ------------------------------------------------------------------ From:René Ferdinand Rivera Morell <grafikrobot@gmail.com> Send Time:2024 Dec. 19 (Thu.) 05:07 To:boost<boost@lists.boost.org> Cc:Matt Borland<matt@mattborland.com>; Chuanqi<chuanqi.xcq@alibaba-inc.com>; Ben Boeckel<ben.boeckel@kitware.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially On Tue, Dec 17, 2024 at 9:19 PM Chuanqi Xu via Boost <boost@lists.boost.org <mailto:boost@lists.boost.org >> wrote:
Would have been nice if the `export` keyword was accepted and silently ignored when outside of module. Yes, but we have missed the train. There are more trains. And there's apparently still time for the C++23 train. This is something that would be entirely possible to propose now as an improvement for adoptability of modules. Who wants to write a proposal? Actually, we don't even need a train. Implementations could do that now without any ill effects (and even backported). And use it as implementation experience. What do you think Chuanqi of implementing that in clang? -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supongas Nada -- Robot Dreams - http://robot-dreams.net <http://robot-dreams.net/ >
On Wed, Dec 18, 2024 at 7:26 PM Chuanqi Xu <chuanqi.xcq@alibaba-inc.com> wrote:
Hi René,
Interesting, I didn’t think about it actually. On the one hand, it is surprising to me to heard we don’t miss C++23 train.
Sorry, that was a fast typing mistake on my part. I meant the C++26 train.
On the other hand, the idea to implement it in Clang without the proposal in WG21 looks like pandora’s box to me.
If we did the second point, the code accepted by clang may not be accepted by other compilers. Although it happens now, we don’t want it to be the case. Further more, I feel it makes the position of WG21 to be in a pretty embrassive position.
That is a good point. And it would end up back in using macros to resolve portability in that case. Still worth thinking about in wg21 though. :-) -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supongas Nada -- Robot Dreams - http://robot-dreams.net
On 2024-12-19 at 02:26, Chuanqi Xu via Boost wrote:
On the other hand, the idea to implement it in Clang without the proposal in WG21 looks like pandora’s box to me.
The committee is sometimes accused of inventing a new language, instead of only standardizing. Giving them some proven "existing practice" to standardize could be a good idea. Especially if it has to do with the "export" keyword. :-)
If we did the second point, the code accepted by clang may not be accepted by other compilers. Although it happens now, we don’t want it to be the case. Further more, I feel it makes the position of WG21 to be in a pretty embrassive position.
The idea of ignorable ‘export’ receives much more interests than I thought surprisingly. From the implementor’s position, it looks simple to implement and it looks not super bad to me to downgrade the existing error for non-sense export to warning and ignore the ‘export’ keyword. It makes sense more or less. But another concern is, given boost (or any other library) should be accepted by other compilers and older clang. Then how could we put `export` in the sources directly? Didn’t we have to use yet another macro to handle this? Thanks, Chuanqi ------------------------------------------------------------------ From:Bo Persson via Boost <boost@lists.boost.org> Send Time:2024 Dec. 19 (Thu.) 16:30 To:boost<boost@lists.boost.org> Cc:Bo Persson<bo@bo-persson.se> Subject:Re: [boost] Interest for C++20 modules support of boost officially On 2024-12-19 at 02:26, Chuanqi Xu via Boost wrote:
On the other hand, the idea to implement it in Clang without the proposal in WG21 looks like pandora’s box to me. The committee is sometimes accused of inventing a new language, instead of only standardizing. Giving them some proven "existing practice" to standardize could be a good idea. Especially if it has to do with the "export" keyword. :-) If we did the second point, the code accepted by clang may not be accepted by other compilers. Although it happens now, we don’t want it to be the case. Further more, I feel it makes the position of WG21 to be in a pretty embrassive position.
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost <http://lists.boost.org/mailman/listinfo.cgi/boost >
Chuanqi Xu wrote:
But another concern is, given boost (or any other library) should be accepted by other compilers and older clang. Then how could we put `export` in the sources directly? Didn’t we have to use yet another macro to handle this?
That's a good point, yes. Since in pre-20 mode we have to remove the `export` anyway, there's perhaps not much point in having ignorable `export`. Or... maybe there still is, because we could at least use a single Config macro BOOST_CXX20_EXPORT, instead of separate ones per library.
On Thu, Dec 19, 2024 at 10:27 AM Peter Dimov via Boost <boost@lists.boost.org> wrote:
Chuanqi Xu wrote:
But another concern is, given boost (or any other library) should be accepted by other compilers and older clang. Then how could we put `export` in the sources directly? Didn’t we have to use yet another macro to handle this?
That's a good point, yes. Since in pre-20 mode we have to remove the `export` anyway, there's perhaps not much point in having ignorable `export`.
Or... maybe there still is, because we could at least use a single Config macro BOOST_CXX20_EXPORT, instead of separate ones per library.
Just a thought from the peanut gallery here... Was it ever discussed to partially preprocess the sources to generate a modules-only distribution without #includes and with only imports? Still single-source, with macros and such (which is a burden for maintainers), but generating a "cleaned-up" modules-only source package for those who want to opt in to modules. Kind of like a module-based Boost 2.0 but without it being a fork. And pushing it a little further, preprocess out pre-C++20 compiler workarounds too—why not? If this was discussed, I missed it on the mailing list. But perhaps it's too preposterous to even be discussed. :) --DD
Was it ever discussed to partially preprocess the sources to generate a modules-only distribution without #includes and with only imports? Still single-source, with macros and such (which is a burden for maintainers), but generating a "cleaned-up" modules-only source package for those who want to opt in to modules. Kind of like a module-based Boost 2.0 but without it being a fork. And pushing it a little further, preprocess out pre-C++20 compiler workarounds too—why not? If this was discussed, I missed it on the mailing list. But perhaps it's too preposterous to even be discussed. :) I thought such idea roughly. But I never saw such discussion anywhere. I guess the reason maybe it sounds hard? I wrote a tool to help refactoring projects to modules (https://github.com/ChuanqiXu9/clang-modules-converter <https://github.com/ChuanqiXu9/clang-modules-converter >). But given the complexity of C++, it is only a semi-automated process. There was a only a paper to do such things (https://onlinelibrary.wiley.com/doi/full/10.1002/smr.2736 <https://onlinelibrary.wiley.com/doi/full/10.1002/smr.2736 >). But all of this are semi-auto. It means more burden to maintainers. And from what we proposed so far, actually you can find the maintain burden won’t be so large. It is basically to #ifdef some headers and add a macro to a decl. It is trivial. I guess people may be more frustrated to fight for the auto genetated mess. But as you said, these are merely ideas. No one ever try to do it in the long way. Thanks, Chuanqi
Chuanqi Xu wrote:
But another concern is, given boost (or any other library) should be accepted by other compilers and older clang. Then how could we put `export` in the sources directly? Didn’t we have to use yet another macro to handle this?
That's a good point, yes. Since in pre-20 mode we have to remove the `export` anyway, there's perhaps not much point in having ignorable `export`.
Or... maybe there still is, because we could at least use a single Config macro BOOST_CXX20_EXPORT, instead of separate ones per library. Just a thought from the peanut gallery here... Was it ever discussed to partially preprocess the sources to generate a modules-only distribution without #includes and with only imports? Still single-source, with macros and such (which is a burden for
From:Dominique Devienne via Boost <boost@lists.boost.org> Send Time:2024 Dec. 19 (Thu.) 19:51 To:boost<boost@lists.boost.org> Cc:Dominique Devienne<ddevienne@gmail.com>; Peter Dimov<pdimov@gmail.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially On Thu, Dec 19, 2024 at 10:27 AM Peter Dimov via Boost <boost@lists.boost.org> wrote: maintainers), but generating a "cleaned-up" modules-only source package for those who want to opt in to modules. Kind of like a module-based Boost 2.0 but without it being a fork. And pushing it a little further, preprocess out pre-C++20 compiler workarounds too—why not? If this was discussed, I missed it on the mailing list. But perhaps it's too preposterous to even be discussed. :) --DD _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost <http://lists.boost.org/mailman/listinfo.cgi/boost >
Bo Persson via Boost <boost@lists.boost.org> hat am 19.12.2024 09:29 CET geschrieben:
On 2024-12-19 at 02:26, Chuanqi Xu via Boost wrote:
On the other hand, the idea to implement it in Clang without the proposal in WG21 looks like pandora’s box to me.
The committee is sometimes accused of inventing a new language, instead of only standardizing. Giving them some proven "existing practice" to standardize could be a good idea.
The "existing practice" was established long before modules and the related rules became standardized with C++20 (remember the Modules TS? It was - at least partially - implemented in MSVC and Clang). I like the remarkable stability of that language feature. Changing that requires a proposal with a better motivation than just perceived convenience, and a champion who's willing to die on that hill.
Especially if it has to do with the "export" keyword. :-)
If we did the second point, the code accepted by clang may not be accepted by other compilers. Although it happens now, we don’t want it to be the case. Further more, I feel it makes the position of WG21 to be in a pretty embrassive position.
On 17/12/2024 02:37, Chuanqi Xu via Boost wrote:
Hi Boost Developers, I am Chuanqi Xu, the maintainer for modules & serialization in clang and also a fan of C++20 modules. The post itself was motivated by a discussion (https://cpplang.slack.com/archives/C27KZLB0X/p1734335425348809 <https://cpplang.slack.com/archives/C27KZLB0X/p1734335425348809 >). You don’t need to read it if you’re not interested. I’ll try to make the post itself self contained.
I *think* at present I would prefer to see library-level support - ie clients would do a: import boost.whatever; rather than a single monolithic: import boost; In any case multiple libraries can be always be grouped together in a "super-module" that import-exports all of them. There are some advantages to this: * it's modular and can grow gradually based on demand. * We don't get blocked by if library X explodes all current compilers (there's some complex code in Boost after all). FYI regex currently has an experimental branch which does this - I really should have got it into the last release, basically I just need to document things. I would also add that it's not really as straightforward as you might think - compilers are certainly getting better in this regard - but there are all sorts of traps especially if client code mixes imports with #includes of regex's std lib dependencies. I also had to do quite a bit of work (real code changes) to ensure that module builds were actually faster in practice than a simple header-only approach. Given the template-heavy nature of Boost I would imagine that many libraries would be in this situation. And finally.... build systems *currently* are not up to the job.... if I merged the regex module support now that would leave users with a very 1980's "here's some code files, build them yourself" kind of situation. Best, John.
On Tue, Dec 17, 2024 at 8:17 AM John Maddock via Boost < boost@lists.boost.org> wrote:
On 17/12/2024 02:37, Chuanqi Xu via Boost wrote:
Hi Boost Developers, I am Chuanqi Xu, the maintainer for modules & serialization in clang and also a fan of C++20 modules. The post itself was motivated by a discussion (https://cpplang.slack.com/archives/C27KZLB0X/p1734335425348809 <https://cpplang.slack.com/archives/C27KZLB0X/p1734335425348809 >). You don’t need to read it if you’re not interested. I’ll try to make the post itself self contained.
I *think* at present I would prefer to see library-level support - ie clients would do a:
import boost.whatever;
+1 -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supongas Nada -- Robot Dreams - http://robot-dreams.net
Chuanqi Xu wrote:
For authors of libraries, after boost introduce modules, you only need to do 2 extra things: - In headers, add a macro (assume its name as BOOST_MODULE_EXPORT, which will be ‘export’ conditionally) to the entities that need to be exported.
Would have been nice if the `export` keyword was accepted and silently ignored when outside of module.
Am 17.12.2024 um 16:29 schrieb Peter Dimov via Boost:
Chuanqi Xu wrote:
For authors of libraries, after boost introduce modules, you only need to do 2 extra things: - In headers, add a macro (assume its name as BOOST_MODULE_EXPORT, which will be ‘export’ conditionally) to the entities that need to be exported. Would have been nice if the `export` keyword was accepted and silently ignored when outside of module.
Totally so! Whoever is considering to introduce module support into some Boost library should be acutely aware of, and familiar with, the following sections in the C++ standard and its implications: * http://eel.is/c++draft/module.reach#3 * http://eel.is/c++draft/module.reach in general and in particular * http://eel.is/c++draft/module.global.frag#3 * http://eel.is/c++draft/module.global.frag#4 This applies to all modules which are *not* leafs in the entire dependency tree and contain so-called global module fragments (http://eel.is/c++draft/module.global.frag) in user-facing TUs. Some compilers do *not* implement GMF entity pruning correctly, or at all. This will lead to disappointed and/or frustrated users, or - even worse - to users who are misled to assume seeing correct module BMIs, while these are - in fact - incorrect. This was one the topics in my modules workshop this fall (hint: stay away from Clang at least up to 18. Clang 19 *might* have improved in this regard). The recommended remedy is to avoid #includes of standard library headers at all and turn to import std; instead. But this is just the starting point. Dani -- PGP/GPG: 2CCB 3ECB 0954 5CD3 B0DB 6AA0 BA03 56A1 2C4638C5
Whoever is considering to introduce module support into some Boost library should be acutely aware of, and familiar with, the following sections in the C++ standard and its implications:
* http://eel.is/c++draft/module.reach#3 * http://eel.is/c++draft/module.reach in general
and in particular
* http://eel.is/c++draft/module.global.frag#3 * http://eel.is/c++draft/module.global.frag#4
Do you know of any material explaining these concepts in a more approachable way? I'm no language lawyer, and tend to have a hard time understanding some of these.
On Tue, 17 Dec 2024 at 16:30, Peter Dimov via Boost <boost@lists.boost.org> wrote:
Chuanqi Xu wrote:
For authors of libraries, after boost introduce modules, you only need to do 2 extra things: - In headers, add a macro (assume its name as BOOST_MODULE_EXPORT, which will be ‘export’ conditionally) to the entities that need to be exported.
Would have been nice if the `export` keyword was accepted and silently ignored when outside of module.
Wouldn't you need to still ifdef-out your includes?
Wouldn't you need to still ifdef-out your includes?
Yes. While it seems like it would be nice for 'export' to be ignorable, I don't really see that it would make much of a difference to how easily a library can be modularized. Especially given that we have export blocks, so it's not necessary to put a macro in front of every individual declaration anyway. On Thu, 19 Dec 2024 at 20:36, Ruben Perez via Boost <boost@lists.boost.org> wrote:
On Tue, 17 Dec 2024 at 16:30, Peter Dimov via Boost <boost@lists.boost.org> wrote:
Chuanqi Xu wrote:
For authors of libraries, after boost introduce modules, you only
need to do 2
extra things: - In headers, add a macro (assume its name as BOOST_MODULE_EXPORT, which will be ‘export’ conditionally) to the entities that need to be exported.
Would have been nice if the `export` keyword was accepted and silently ignored when outside of module.
Wouldn't you need to still ifdef-out your includes?
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Chuanqi Xu wrote:
For authors of libraries, after boost introduce modules, you only need to do 2 extra things: - In headers, add a macro (assume its name as BOOST_MODULE_EXPORT, which will be ‘export’ conditionally) to the entities that need to be exported. - In headers, include standard headers and other dependent headers conditionally so that we don’t include anything if the headers are in a module unit. e.g., https://github.com/boostorg/pfr/blob/f09e6aeae9d050897fff72b93d5f5e866cc5e11...
For a header-only library with no dependencies, you can basically do whatever you like, but in the general case you'd have something like #ifdef BOOST_USE_MODULE_CORE import boost.core; #else #include <boost/core/this.hpp> #include <boost/core/that.hpp> #endif repeated for every library you #include things from. And the end user will have to do this too. I'm also not quite clear on how things will work on the CMake side. I can envisage the add_subdirectory case working, more or less, but what would (or needs to) happen when Boost is _installed_ via CMake, with module support enabled? And how will it then be consumed on the user side, after find_package? Also, we're basically talking Clang-only here, right? GCC doesn't work and MSVC... may or may not work?
I'm also not quite clear on how things will work on the CMake side. I can envisage the add_subdirectory case working, more or less, but what would (or needs to) happen when Boost is _installed_ via CMake, with module support enabled?
And how will it then be consumed on the user side, after find_package? As far as I understood the BMIs are "just precompiled headers". So those must not be installed because they basically depend on everything related to the compiler invocation.
I guess the installed find module could just add target_sources(boost_core PUBLIC FILE_SET modules TYPE CXX_MODULES FILES file.cppm) Not sure how much this can be automated.
I'm also not quite clear on how things will work on the CMake side. I can envisage the add_subdirectory case working, more or less, but what would (or needs to) happen when Boost is _installed_ via CMake, with module support enabled?
And how will it then be consumed on the user side, after find_package?
I think this may be the most difficult part right now. Last time I checked (as described in the cited articles), the workflow with CMake was something like: a) You add_library() and target_souces() using the new FILE_SET CXX_MODULES feature. This builds a library (e.g. libboost_asio.a) and a BMI. Note that even previously header-only libraries end up generating compiled libraries in the modules world. The compiled library contains code for global initializers, and is emitted even if there are no global initializers [1]. b) You install() the library with its module bindings. This means that you end up installing libboost_asio.a and any other files required to generate the BMI for the Asio module (in this case, this would be Asio headers). If the module contained implementation units (somehow akin to .cpp files in today's world - apologies if the comparison is not exact), these shouldn't get installed. c) The user calls find_package() on the installed package, and calls target_link_libraries() linking to the library installed in the previous step. When doing this, CMake rebuilds the BMI, but not the object files (libboost_asio.a). The BMI will be built with the same cxxstd (I don't know about other properties) as the original library. This is the point that generates questions for me, as it imposes many more restrictions than the traditional header model [2]. On the other hand, the CMake team seems to be working on enhancing this, but without an ETA [3].
Also, we're basically talking Clang-only here, right? GCC doesn't work and MSVC... may or may not work?
The approach where you #ifdef out standard includes and conditionally export entities applies to both MSVC and clang. gcc has had experimental support for modules for a couple of releases. libstdc++ has added support for the std module recently [4], but it hasn't been released yet. I'm going to experiment with Boost.CMake and modules to see if we could produce an installed Boost containing only the libraries that support modules. And see how much trouble point c) would cause us. I'm particularly worried about libraries like Asio, that radically change based on macros. [1] https://clang.llvm.org/docs/StandardCPlusPlusModules.html#module-initializer... [2] https://crascit.com/2024/04/04/cxx-modules-cmake-shared-libraries/ [3] https://discourse.cmake.org/t/advice-on-c-20-modules-boost/10641/5 [4] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106852
On 19/12/2024 11:32, Ruben Perez via Boost wrote:
I'm also not quite clear on how things will work on the CMake side. I can envisage the add_subdirectory case working, more or less, but what would (or needs to) happen when Boost is _installed_ via CMake, with module support enabled?
And how will it then be consumed on the user side, after find_package? I think this may be the most difficult part right now. Last time I checked (as described in the cited articles), the workflow with CMake was something like:
a) You add_library() and target_souces() using the new FILE_SET CXX_MODULES feature. This builds a library (e.g. libboost_asio.a) and a BMI. Note that even previously header-only libraries end up generating compiled libraries in the modules world. The compiled library contains code for global initializers, and is emitted even if there are no global initializers [1]. b) You install() the library with its module bindings. This means that you end up installing libboost_asio.a and any other files required to generate the BMI for the Asio module (in this case, this would be Asio headers). If the module contained implementation units (somehow akin to .cpp files in today's world - apologies if the comparison is not exact), these shouldn't get installed. c) The user calls find_package() on the installed package, and calls target_link_libraries() linking to the library installed in the previous step. When doing this, CMake rebuilds the BMI, but not the object files (libboost_asio.a). The BMI will be built with the same cxxstd (I don't know about other properties) as the original library. This is the point that generates questions for me, as it imposes manyyou need to assume tha more restrictions than the traditional header model [2]. On the other hand, the CMake team seems to be working on enhancing this, but without an ETA [3].
Correct me if I'm wrong, but in the modules world I don't think you can really install anything - you need to think of them as more like precompiled headers that are project specific and built as part of each individual project, you need to assume that changing *any* compiler option will break the module unless it is re-built. That's why the msvc std module is built on a per-project basis. Boost modules will be just the same I assume. Meanwhile, I've cleaned up and merged Boost.Regex module support to develop if anyone wants to try it out. There are rudimentary instructions here: https://github.com/boostorg/regex/blob/develop/doc/modules.qbk, conceivably this could be enhanced by providing CMake and visual studio projects. Best, John.
Correct me if I'm wrong, but in the modules world I don't think you can really install anything - you need to think of them as more like precompiled headers that are project specific and built as part of each individual project, you need to assume that changing *any* compiler option will break the module unless it is re-built. That's why the msvc std module is built on a per-project basis. Boost modules will be just the same I assume. When we say installing modules, we’re saying installing the source files for the module interfaces, not the immediate products. And for recompiling, it is the job of build systems. The story is, in a program, a module interfaces will only be compiled into an object once (and be linked). However, the module interfaces can be compiled into several pcms (BMIs) by the build systems with different configurations. So it likes headers in some aspect. This talk (https://www.youtube.com/watch?v=_LGR0U5Opdg&t=1662s <https://www.youtube.com/watch?v=_LGR0U5Opdg&t=1662s >) have some explanations Meanwhile, I've cleaned up and merged Boost.Regex module support to develop if anyone wants to try it out. There are rudimentary instructions here: https://github.com/boostorg/regex/blob/develop/doc/modules.qbk <https://github.com/boostorg/regex/blob/develop/doc/modules.qbk >, conceivably this could be enhanced by providing CMake and visual studio projects. Excellent! I can’t believe it is so fast! Thanks, Chuanqi
I'm also not quite clear on how things will work on the CMake side. I can envisage the add_subdirectory case working, more or less, but what would (or needs to) happen when Boost is _installed_ via CMake, with module support enabled?
And how will it then be consumed on the user side, after find_package? I think this may be the most difficult part right now. Last time I checked (as described in the cited articles), the workflow with CMake was something like:
a) You add_library() and target_souces() using the new FILE_SET CXX_MODULES feature. This builds a library (e.g. libboost_asio.a) and a BMI. Note that even previously header-only libraries end up generating compiled libraries in the modules world. The compiled library contains code for global initializers, and is emitted even if there are no global initializers [1]. b) You install() the library with its module bindings. This means that you end up installing libboost_asio.a and any other files required to generate the BMI for the Asio module (in this case, this would be Asio headers). If the module contained implementation units (somehow akin to .cpp files in today's world - apologies if the comparison is not exact), these shouldn't get installed. c) The user calls find_package() on the installed package, and calls target_link_libraries() linking to the library installed in the previous step. When doing this, CMake rebuilds the BMI, but not the object files (libboost_asio.a). The BMI will be built with the same cxxstd (I don't know about other properties) as the original library. This is the point that generates questions for me, as it imposes manyyou need to assume tha more restrictions than the traditional header model [2]. On the other hand, the CMake team seems to be working on enhancing this, but without an ETA [3]. Correct me if I'm wrong, but in the modules world I don't think you can really install anything - you need to think of them as more like
From:John Maddock via Boost <boost@lists.boost.org> Send Time:2024 Dec. 19 (Thu.) 20:01 To:Ruben Perez via Boost<boost@lists.boost.org> Cc:John Maddock<jz.maddock@googlemail.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially On 19/12/2024 11:32, Ruben Perez via Boost wrote: precompiled headers that are project specific and built as part of each individual project, you need to assume that changing *any* compiler option will break the module unless it is re-built. That's why the msvc std module is built on a per-project basis. Boost modules will be just the same I assume. Meanwhile, I've cleaned up and merged Boost.Regex module support to develop if anyone wants to try it out. There are rudimentary instructions here: https://github.com/boostorg/regex/blob/develop/doc/modules.qbk <https://github.com/boostorg/regex/blob/develop/doc/modules.qbk >, conceivably this could be enhanced by providing CMake and visual studio projects. Best, John. _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost <http://lists.boost.org/mailman/listinfo.cgi/boost >
a) You add_library() and target_souces() using the new FILE_SET CXX_MODULES feature. This builds a library (e.g. libboost_asio.a) and a BMI. Note that even previously header-only libraries end up generating compiled libraries in the modules world. The compiled library contains code for global initializers, and is emitted even if there are no global initializers [1]. b) You install() the library with its module bindings. This means that you end up installing libboost_asio.a and any other files required to generate the BMI for the Asio module (in this case, this would be Asio headers). If the module contained implementation units (somehow akin to .cpp files in today's world - apologies if the comparison is not exact), these shouldn't get installed. c) The user calls find_package() on the installed package, and calls target_link_libraries() linking to the library installed in the previous step. When doing this, CMake rebuilds the BMI, but not the object files (libboost_asio.a). The BMI will be built with the same cxxstd (I don't know about other properties) as the original library. This is the point that generates questions for me, as it imposes manyyou need to assume tha more restrictions than the traditional header model [2]. On the other hand, the CMake team seems to be working on enhancing this, but without an ETA [3].
Correct me if I'm wrong, but in the modules world I don't think you can really install anything - you need to think of them as more like precompiled headers that are project specific and built as part of each individual project, you need to assume that changing *any* compiler option will break the module unless it is re-built. That's why the msvc std module is built on a per-project basis. Boost modules will be just the same I assume.
If I understood the CMake developers and Chuanqi correctly (please let me know if I didn't), you do install a library with the generated object files, but you do not install the BMIs, as they're like pre-compiled headers. Let me explain how I understood it with two concrete examples. Take Boost.Url as an example of a compiled library, and Boost.Regex as an example of a header-only library. 1. Boost.Url today installs: * libboost_url.a, containing function definitions. * Headers, containing declarations. 2. Boost.Url in the module world would install: * libboost_url.a, containing the same function definitions as above. * A boost_url.cppm file, akin to what headers are today. CMake would build a BMI from this file when a user needs to import boost.url. In practice, this cppm file will likely be implemented in terms of the header files that we have today. 3. Boost.Regex today installs: * Headers, containing declarations and definitions. 4. Boost.Regex in the module world would install: * A libboost_regex.a object, containing module initializers [1]. For instance, if you have a singleton defining a Boost.System error category, initialization code for the singleton would be emitted here. From the docs, it seems this object gets created even if no initialization code is required. * A boost_regex.cppm file, containing the bulk of the library, akin to what we provide using headers today. My intention is doing some testing to see how much trouble would this new libboost_regex.a library cause.
Meanwhile, I've cleaned up and merged Boost.Regex module support to develop if anyone wants to try it out. There are rudimentary instructions here: https://github.com/boostorg/regex/blob/develop/doc/modules.qbk, conceivably this could be enhanced by providing CMake and visual studio projects.
What I described above is what you get if you use CMake's install() function for a target containing C++20 modules. Ironically, this doesn't apply for the standard library, which is always built by the user. Unfortunately, CMake doesn't seem to have first-class support for this, meaning that the user would need to figure out dependencies manually. I think the best model for our header-only libraries would be the one used for the std module: we distribute the cppm files, and the module gets built by the user. I will ping the CMake developers to see if there is a way to handle this. Thanks for your efforts here, John. Regards, Ruben.
Ruben Perez wrote:
If I understood the CMake developers and Chuanqi correctly (please let me know if I didn't), you do install a library with the generated object files, but you do not install the BMIs, as they're like pre-compiled headers.
Let me explain how I understood it with two concrete examples. Take Boost.Url as an example of a compiled library, and Boost.Regex as an example of a header-only library.
1. Boost.Url today installs: * libboost_url.a, containing function definitions. * Headers, containing declarations. 2. Boost.Url in the module world would install: * libboost_url.a, containing the same function definitions as above. * A boost_url.cppm file, akin to what headers are today. CMake would build a BMI from this file when a user needs to import boost.url. In practice, this cppm file will likely be implemented in terms of the header files that we have today.
Since this .cppm file will include the header files, the header files should also be installed in the module case. Where does the .cppm file go by convention? In which directory?
On Thu, 19 Dec 2024, 15:43 Peter Dimov via Boost, <boost@lists.boost.org> wrote:
If I understood the CMake developers and Chuanqi correctly (please let me know if I didn't), you do install a library with the generated object files, but you do not install the BMIs, as they're like pre-compiled headers.
Let me explain how I understood it with two concrete examples. Take Boost.Url as an example of a compiled library, and Boost.Regex as an example of a header-only library.
1. Boost.Url today installs: * libboost_url.a, containing function definitions. * Headers, containing declarations. 2. Boost.Url in the module world would install: * libboost_url.a, containing the same function definitions as above. * A boost_url.cppm file, akin to what headers are today. CMake would build a BMI from this file when a user needs to import boost.url. In practice, this cppm file will likely be implemented in terms of
Ruben Perez wrote: the
header files that we have today.
Since this .cppm file will include the header files, the header files should also be installed in the module case.
Yes. And some libraries also need to export macros. These need to be provided by traditional headers. For example, Asio needs to provide BOOST_ASIO_HAS_LOCAL_SOCKETS.
Where does the .cppm file go by convention? In which directory?
I don't think there is a convention for it yet.
Where does the .cppm file go by convention? In which directory? No convention yet. Some people suggests to `<install-dir>/modules`. Some people suggests `<install-dir>/share`. There are some people suggests `<install-dir>/include` too. libc++ chose `<install-dir>/share`. And if I read correctly, libstdc++ chose `<install-dir>/include`. So this is more or less a non-consensus topic now. Thanks, Chuanqi
From:Peter Dimov via Boost <boost@lists.boost.org> Send Time:2024 Dec. 19 (Thu.) 22:43 To:boost<boost@lists.boost.org> Cc:Peter Dimov<pdimov@gmail.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially Ruben Perez wrote:
If I understood the CMake developers and Chuanqi correctly (please let me know if I didn't), you do install a library with the generated object files, but you do not install the BMIs, as they're like pre-compiled headers.
Let me explain how I understood it with two concrete examples. Take Boost.Url as an example of a compiled library, and Boost.Regex as an example of a header-only library.
1. Boost.Url today installs: * libboost_url.a, containing function definitions. * Headers, containing declarations. 2. Boost.Url in the module world would install: * libboost_url.a, containing the same function definitions as above. * A boost_url.cppm file, akin to what headers are today. CMake would build a BMI from this file when a user needs to import boost.url. In practice, this cppm file will likely be implemented in terms of the header files that we have today. Since this .cppm file will include the header files, the header files should also be installed in the module case. Where does the .cppm file go by convention? In which directory?
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost <http://lists.boost.org/mailman/listinfo.cgi/boost >
Debian has started putting google test in <prefix>/src, since it has similar issues as modules do around sensitivity to compiler flags. Getting to the understanding that the sources for the interface must be an installed component has taken more work than expected. Figuring out where in a LHS-ish tree they go has been complicated with differing understandings of what <prefix>/include is for, vs other locations in the hierarchy. Fortunately we've also got some agreement on a file that describes what is needed to build the module interface, so there's an indirection to give the flexibility needed. On Thu, Dec 19, 2024 at 9:23 PM Chuanqi Xu via Boost <boost@lists.boost.org> wrote:
Where does the .cppm file go by convention? In which directory? No convention yet. Some people suggests to `<install-dir>/modules`. Some people suggests `<install-dir>/share`. There are some people suggests `<install-dir>/include` too. libc++ chose `<install-dir>/share`. And if I read correctly, libstdc++ chose `<install-dir>/include`. So this is more or less a non-consensus topic now. Thanks, Chuanqi
From:Peter Dimov via Boost <boost@lists.boost.org> Send Time:2024 Dec. 19 (Thu.) 22:43 To:boost<boost@lists.boost.org> Cc:Peter Dimov<pdimov@gmail.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially Ruben Perez wrote:
If I understood the CMake developers and Chuanqi correctly (please let me know if I didn't), you do install a library with the generated object files, but you do not install the BMIs, as they're like pre-compiled headers.
Let me explain how I understood it with two concrete examples. Take Boost.Url as an example of a compiled library, and Boost.Regex as an example of a header-only library.
1. Boost.Url today installs: * libboost_url.a, containing function definitions. * Headers, containing declarations. 2. Boost.Url in the module world would install: * libboost_url.a, containing the same function definitions as above. * A boost_url.cppm file, akin to what headers are today. CMake would build a BMI from this file when a user needs to import boost.url. In practice, this cppm file will likely be implemented in terms of the header files that we have today. Since this .cppm file will include the header files, the header files should also be installed in the module case. Where does the .cppm file go by convention? In which directory?
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost < http://lists.boost.org/mailman/listinfo.cgi/boost >
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Hi Ruben, I think we’re in the same page except a few wording issues. e.g, I won’t call it as pre-compiled headers. But this doesn’t matter.
2. Boost.Url in the module world would install: * libboost_url.a, containing the same function definitions as above. * A boost_url.cppm file, akin to what headers are today. CMake would build a BMI from this file when a user needs to import boost.url. In practice, this cppm file will likely be implemented in terms of the header files that we have today. The libboost_url.a with modules support may have the same entities with original libboost_url.a **plus** the module initializer. And also, as discussed in other mail, the headers will be installed too. Unfortunately, CMake doesn't seem to have first-class support for this, meaning that the user would need to figure out dependencies manually. What do you mean by figuring out dependencies manually? Do you mean, in cmake, the dependencies will be calculated if the libraries are linked? If yes, I feel it might be fine. Or did I misunderstand it? Thanks, Chuanqi
a) You add_library() and target_souces() using the new FILE_SET CXX_MODULES feature. This builds a library (e.g. libboost_asio.a) and a BMI. Note that even previously header-only libraries end up generating compiled libraries in the modules world. The compiled library contains code for global initializers, and is emitted even if there are no global initializers [1]. b) You install() the library with its module bindings. This means that you end up installing libboost_asio.a and any other files required to generate the BMI for the Asio module (in this case, this would be Asio headers). If the module contained implementation units (somehow akin to .cpp files in today's world - apologies if the comparison is not exact), these shouldn't get installed. c) The user calls find_package() on the installed package, and calls target_link_libraries() linking to the library installed in the previous step. When doing this, CMake rebuilds the BMI, but not the object files (libboost_asio.a). The BMI will be built with the same cxxstd (I don't know about other properties) as the original library. This is the point that generates questions for me, as it imposes manyyou need to assume tha more restrictions than the traditional header model [2]. On the other hand, the CMake team seems to be working on enhancing this, but without an ETA [3].
Correct me if I'm wrong, but in the modules world I don't think you can really install anything - you need to think of them as more like precompiled headers that are project specific and built as part of each individual project, you need to assume that changing *any* compiler option will break the module unless it is re-built. That's why the msvc std module is built on a per-project basis. Boost modules will be just the same I assume. If I understood the CMake developers and Chuanqi correctly (please let me know if I didn't), you do install a library with the generated object files, but you do not install the BMIs, as they're like
From:Ruben Perez via Boost <boost@lists.boost.org> Send Time:2024 Dec. 19 (Thu.) 21:46 To:boost<boost@lists.boost.org> Cc:Ruben Perez<rubenperez038@gmail.com>; John Maddock<jz.maddock@googlemail.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially pre-compiled headers. Let me explain how I understood it with two concrete examples. Take Boost.Url as an example of a compiled library, and Boost.Regex as an example of a header-only library. 1. Boost.Url today installs: * libboost_url.a, containing function definitions. * Headers, containing declarations. 2. Boost.Url in the module world would install: * libboost_url.a, containing the same function definitions as above. * A boost_url.cppm file, akin to what headers are today. CMake would build a BMI from this file when a user needs to import boost.url. In practice, this cppm file will likely be implemented in terms of the header files that we have today. 3. Boost.Regex today installs: * Headers, containing declarations and definitions. 4. Boost.Regex in the module world would install: * A libboost_regex.a object, containing module initializers [1]. For instance, if you have a singleton defining a Boost.System error category, initialization code for the singleton would be emitted here. From the docs, it seems this object gets created even if no initialization code is required. * A boost_regex.cppm file, containing the bulk of the library, akin to what we provide using headers today. My intention is doing some testing to see how much trouble would this new libboost_regex.a library cause.
Meanwhile, I've cleaned up and merged Boost.Regex module support to develop if anyone wants to try it out. There are rudimentary instructions here: https://github.com/boostorg/regex/blob/develop/doc/modules.qbk <https://github.com/boostorg/regex/blob/develop/doc/modules.qbk >, conceivably this could be enhanced by providing CMake and visual studio projects.
What I described above is what you get if you use CMake's install() function for a target containing C++20 modules. Ironically, this doesn't apply for the standard library, which is always built by the user. Unfortunately, CMake doesn't seem to have first-class support for this, meaning that the user would need to figure out dependencies manually. I think the best model for our header-only libraries would be the one used for the std module: we distribute the cppm files, and the module gets built by the user. I will ping the CMake developers to see if there is a way to handle this. Thanks for your efforts here, John. Regards, Ruben. _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost <http://lists.boost.org/mailman/listinfo.cgi/boost >
On Fri, 20 Dec 2024 at 03:15, Chuanqi Xu <chuanqi.xcq@alibaba-inc.com> wrote:
Hi Ruben,
I think we’re in the same page except a few wording issues. e.g, I won’t call it as pre-compiled headers. But this doesn’t matter.
2. Boost.Url in the module world would install: * libboost_url.a, containing the same function definitions as above. * A boost_url.cppm file, akin to what headers are today. CMake would build a BMI from this file when a user needs to import boost.url. In practice, this cppm file will likely be implemented in terms of the header files that we have today.
The libboost_url.a with modules support may have the same entities with original libboost_url.a **plus** the module initializer.
And also, as discussed in other mail, the headers will be installed too.
Yes, completely. I'm building a small prototype to show how things would work, as discussing with concrete code is much more fruitful.
Unfortunately, CMake doesn't seem to have first-class support for this, meaning that the user would need to figure out dependencies manually.
What do you mean by figuring out dependencies manually? Do you mean, in cmake, the dependencies will be calculated if the libraries are linked? If yes, I feel it might be fine. Or did I misunderstand it?
Consider Boost.variant2. It's a header-only library that depends on Boost.assert, Boost.config and Boost.mp11, which are also header-only. This is what the CMake code to build these as modules could look like (simplified - I will post the entire prototype when ready): add_library(boost_variant2) target_sources(boost_variant2 PUBLIC FILE_SET CXX_MODULES FILES modules/variant2.cxx) target_include_directories(boost_variant2 PUBLIC include) target_link_libraries(boost_variant2 PUBLIC Boost::assert Boost::config Boost::mp11 ) Here, Boost::assert and Boost::mp11 have been created with similar code as the one shown above. Boost::config is an interface library with only headers (as it only exports macros). The targets are installed by the Boost.CMake infrastructure, with a call akin to: install(TARGETS ${LIB} EXPORT ${LIB}-targets RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}" LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}" ARCHIVE DESTINATION "${CMAKE_INSTALL_LIBDIR}" PRIVATE_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" PUBLIC_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" FILE_SET CXX_MODULES DESTINATION . ) When building Boost with CMake, for each module (Boost::assert, Boost::mp11 and Boost::variant2), a BMI and a static library with the module initializer is built. When installing Boost (with CMake), for each module (Boost::assert, Boost::mp11 and Boost::variant2), the headers, module code and static library is installed. When consuming Boost from another project, e.g. using find_package(Boost COMPONENTS Boost::variant2), the headers, module code and static library for Boost::variant2 and its dependencies are found and used. The BMIs are built in the consumer project. Right now, the flags used originally to build Boost are used to re-build the BMIs. This causes trouble, but is intended to change in the future, with an unknown time frame [1]. Following this approach, dependencies are automatic. But this has a number of downsides: 1. At the moment, BMIs are built with the settings used to build Boost in the first place, which causes a lot of trouble. 2. The installation generates many more binaries than in the headers world. It is uncertain whether these binaries will be compatible if build flags change. 3. Some libraries support configuration macros. For instance, Boost::throw_exception supports BOOST_NO_EXCEPTIONS. This is used by other Boost libraries to support builds that disable exceptions. In the current world, the end user defines BOOST_NO_EXCEPTIONS in their code, includes the library, and everything works. Using a scheme like the one I suggested requires the user to build all the Boost modules using BOOST_NO_EXCEPTIONS, which is more involved than what we have today. For this reason, it makes sense to consider whether there is a way to completely avoid the generation and installation of these binaries for header-only libraries, and follow a model similar to standard modules. In this ideal model, installing Boost with CMake would only install the headers and module code for each header-only library, and no binary artifacts. The consuming project would build both the BMIs and the objects containing the module initializers with the adequate build flags, as many times as required by the different targets. Ideally, the following syntax would work and do what I'm suggesting: # This does NOT work as of today add_library(boost_variant2 INTERFACE) # Don't build the library in the current project target_sources(boost_variant2 INTERFACE FILE_SET CXX_MODULES FILES modules/variant2.cxx) target_include_directories(boost_variant2 INTERFACE include) target_link_libraries(boost_variant2 INTERFACE Boost::assert Boost::config Boost::mp11 ) In the ideal world, this wouldn't build any binary library or BMI per se. When linking to the boost_variant2 target, a BMI and a static library would be built with the flags required by the consuming target. Note that this does NOT work as of today [2]. I've raised a feature request to the CMake team with this suggestion [3], but I don't think this is happening in the short-term. Given that this does not work, the other option would be installing the headers and the module code manually. For instance: # This is what we have today. Note that we have removed the target_sources add_library(boost_variant2 INTERFACE) target_include_directories(boost_variant2 INTERFACE include) target_link_libraries(boost_variant2 INTERFACE Boost::assert Boost::config Boost::mp11 ) # Install the module code separately install(DIRECTORY modules DESTINATION .) With this setup, it is the user's responsibility to issue the target_sources calls required to build the boost_variant2 module. There is also nothing telling CMake that the boost_variant2 module requires building boost_assert and boost_mp11, though. Thus, it becomes the user's responsibility to know this dependency chain and add the relevant target_sources calls. This is what I mean by "figuring out dependencies manually". While it may not seem that hard in this case, things get much worse as you study libraries with more dependencies. I'm the author of Boost.MySQL, and I wouldn't be able to spell you the current dependency tree without an automated tool to help. I hope this helps clarify what I meant. If I have missed anything obvious, or anyone has ideas on how to make this better, please let me know.
Thanks, Chuanqi
Regards, Ruben. [1] https://discourse.cmake.org/t/advice-on-c-20-modules-boost/10641/13 [2] https://cmake.org/cmake/help/latest/command/target_sources.html#file-sets [3] https://discourse.cmake.org/t/distributing-c-20-modules-as-source/13246
The targets are installed by the Boost.CMake infrastructure
As mentioned in the Slack discussion I'm more of a build2 user lately so my CMake knowledge is a bit out of date. It's possible there is also some Windows/Linux difference here. However, I'm sure that in CMake projects I've worked on in the past, we've always used boost without an install step. The boost CMake scripts would simply be pulled (one way or another, CMake seems to have many ways of doing things, and I guess package managers are another way of doing this) into the project, and boost targets linked against as needed. With this approach, boost components are simply built as part of the consuming project so there are none of the issues that come with prebuilt binaries. I realise that installation is something that needs to be solved eventually, but I'd be surprised if the above approach wasn't fairly common, so if it works I think it would be the most straightforward and a good first goal to aim for.
On Sun, 22 Dec 2024 at 14:29, Cameron Angus via Boost <boost@lists.boost.org> wrote:
The targets are installed by the Boost.CMake infrastructure
As mentioned in the Slack discussion I'm more of a build2 user lately so my CMake knowledge is a bit out of date. It's possible there is also some Windows/Linux difference here. However, I'm sure that in CMake projects I've worked on in the past, we've always used boost without an install step. The boost CMake scripts would simply be pulled (one way or another, CMake seems to have many ways of doing things, and I guess package managers are another way of doing this) into the project, and boost targets linked against as needed. With this approach, boost components are simply built as part of the consuming project so there are none of the issues that come with prebuilt binaries.
I realise that installation is something that needs to be solved eventually, but I'd be surprised if the above approach wasn't fairly common, so if it works I think it would be the most straightforward and a good first goal to aim for.
I think we usually refer to this as the "add_subdirectory workflow". Maybe I'm worrying about the installation workflow too much. We don't maintain the CMake scripts that package managers use AFAIK, so I can't speak about them.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Sun, 22 Dec 2024 at 16:47, Peter Dimov via Boost <boost@lists.boost.org> wrote:
Ruben Perez wrote:
I think we usually refer to this as the "add_subdirectory workflow". Maybe I'm worrying about the installation workflow too much.
You aren't. Users definitely care about the installation workflow.
This is somehow reconfortant to hear.
Following this approach, dependencies are automatic. But this has a number of downsides: 1. At the moment, BMIs are built with the settings used to build Boost in the first place, which causes a lot of trouble. 2. The installation generates many more binaries than in the headers world. It is uncertain whether these binaries will be compatible if build flags change. 3. Some libraries support configuration macros. For instance, Boost::throw_exception supports BOOST_NO_EXCEPTIONS. This is used by other Boost libraries to support builds that disable exceptions. In the current world, the end user defines BOOST_NO_EXCEPTIONS in their code, includes the library, and everything works. Using a scheme like the one I suggested requires the user to build all the Boost modules using BOOST_NO_EXCEPTIONS, which is more involved than what we have today.
Yup, my feeling is that this puts us back where we were before auto-linking with msvc - which is to say with a huge number of hard to diagnose bug reports from folks with some obscure binary incompatibility. OK, maybe not that huge yet, given modules niche usage, but hopefully growing over time. At the risk of going off at a tangent, I would also suggest a "core" boost module containing all the commonly used small libraries - the lesson from the std module appears to be that bigger is better, at least up to some point.
For this reason, it makes sense to consider whether there is a way to completely avoid the generation and installation of these binaries for header-only libraries, and follow a model similar to standard modules. In this ideal model, installing Boost with CMake would only install the headers and module code for each header-only library, and no binary artifacts. The consuming project would build both the BMIs and the objects containing the module initializers with the adequate build flags, as many times as required by the different targets.
Ideally, the following syntax would work and do what I'm suggesting:
# This does NOT work as of today add_library(boost_variant2 INTERFACE) # Don't build the library in the current project target_sources(boost_variant2 INTERFACE FILE_SET CXX_MODULES FILES modules/variant2.cxx) target_include_directories(boost_variant2 INTERFACE include) target_link_libraries(boost_variant2 INTERFACE Boost::assert Boost::config Boost::mp11 )
Having CMake do it all for us would be great, thinking out loud here, is there a way to publish a CMake code fragment that users could cut-and-paste into their own CMake file? Still not ideal I know... And finally... if anyone wants to experiment/road test a module with source, then Regex in develop is all module friendly now (but has no module build files). Thanks for this! Best, John.
On Sun, 22 Dec 2024 at 12:02, John Maddock via Boost <boost@lists.boost.org> wrote:
Following this approach, dependencies are automatic. But this has a number of downsides: 1. At the moment, BMIs are built with the settings used to build Boost in the first place, which causes a lot of trouble. 2. The installation generates many more binaries than in the headers world. It is uncertain whether these binaries will be compatible if build flags change. 3. Some libraries support configuration macros. For instance, Boost::throw_exception supports BOOST_NO_EXCEPTIONS. This is used by other Boost libraries to support builds that disable exceptions. In the current world, the end user defines BOOST_NO_EXCEPTIONS in their code, includes the library, and everything works. Using a scheme like the one I suggested requires the user to build all the Boost modules using BOOST_NO_EXCEPTIONS, which is more involved than what we have today.
Yup, my feeling is that this puts us back where we were before auto-linking with msvc - which is to say with a huge number of hard to diagnose bug reports from folks with some obscure binary incompatibility.
OK, maybe not that huge yet, given modules niche usage, but hopefully growing over time.
I think this is a completely valid concern. Once I have a minimal prototype working, I will try to investigate how much of a problem this is.
At the risk of going off at a tangent, I would also suggest a "core" boost module containing all the commonly used small libraries - the lesson from the std module appears to be that bigger is better, at least up to some point.
For now, I'm doing a single module per library, because it maps much easier to the current CMake infrastructure. Given the current state of modules, I don't want to waste too much time.
For this reason, it makes sense to consider whether there is a way to completely avoid the generation and installation of these binaries for header-only libraries, and follow a model similar to standard modules. In this ideal model, installing Boost with CMake would only install the headers and module code for each header-only library, and no binary artifacts. The consuming project would build both the BMIs and the objects containing the module initializers with the adequate build flags, as many times as required by the different targets.
Ideally, the following syntax would work and do what I'm suggesting:
# This does NOT work as of today add_library(boost_variant2 INTERFACE) # Don't build the library in the current project target_sources(boost_variant2 INTERFACE FILE_SET CXX_MODULES FILES modules/variant2.cxx) target_include_directories(boost_variant2 INTERFACE include) target_link_libraries(boost_variant2 INTERFACE Boost::assert Boost::config Boost::mp11 )
Having CMake do it all for us would be great, thinking out loud here, is there a way to publish a CMake code fragment that users could cut-and-paste into their own CMake file? Still not ideal I know...
I've had the same thoughts, too. Best I can think of is a CMake function. For instance: # In the user's CMakeLists.txt # This means "I want to build the Boost.Variant2 module and its dependencies. # Create a target called my_variant2_module boost_add_cxx20_module(variant2 TARGET_NAME my_variant2_module ) # Now use the module add_executable(main main.cpp) target_link_libraries(main PRIVATE my_variant2_module) There are at least 2 unsolved problems here: 1) How does the user specify build options for the module? That is, how do they specify that they want C++23, a certain visibility, and BOOST_DISABLE_EXCEPTIONS? This is a problem because options must be specified not only to the my_variant2_module, but to all the dependent Boost modules (e.g. Boost::assert and Boost::mp11, in this case). 2) How does this work for compiled libraries? A possibility I can think for 1) is making boost_add_cxx20_module accept a CMake interface target with the user options. All targets created by boost_add_cxx20_module would be linked to this target: # An interface target to specify that we want BOOST_DISABLE_EXCEPTIONS add_library(my_options_lib INTERFACE) target_compile_definitions(my_options_lib INTERFACE BOOST_DISABLE_EXCEPTIONS) # Create the Boost modules, defining BOOST_DISABLE_EXCEPTIONS boost_add_cxx20_module(variant2 TARGET_NAME my_variant2_module LINK_TO my_options_lib ) I'm pretty sure this approach has problems I'm not seeing right now. I'm reluctant to this kind of ad-hoc approach - it'd be best if CMake did it. I haven't thought about 2) yet.
And finally... if anyone wants to experiment/road test a module with source, then Regex in develop is all module friendly now (but has no module build files).
If I have the time, I will include it in the prototype, although I promise nothing yet. I'm trying to get some of the foundational libraries (assert, throw_exception, config, core and mp11) and their tests building for both compilers before anything else.
Thanks for this! Best, John.
Thanks, Ruben.
Ruben Perez wrote:
3. Some libraries support configuration macros. For instance, Boost::throw_exception supports BOOST_NO_EXCEPTIONS. This is used by other Boost libraries to support builds that disable exceptions. In the current world, the end user defines BOOST_NO_EXCEPTIONS in their code, includes the library, and everything works. Using a scheme like the one I suggested requires the user to build all the Boost modules using BOOST_NO_EXCEPTIONS, which is more involved than what we have today.
BOOST_NO_EXCEPTIONS is not defined by the user. It's defined by Boost.Config when -fno-exceptions or an equivalent is used. Differing -std levels pose similar issues. E.g. mp11 defines some things only when -std=c++17 or higher. Not exported: https://github.com/boostorg/mp11/blob/43fd186a8eb5d2b3dcc04cde49092b5be2bc40... Would be exported: https://github.com/boostorg/mp11/blob/43fd186a8eb5d2b3dcc04cde49092b5be2bc40... The user doesn't define any macros to enable this; as with -fno-exceptions, it's detected automatically. https://github.com/boostorg/mp11/blob/43fd186a8eb5d2b3dcc04cde49092b5be2bc40...
On Sun, 22 Dec 2024 at 13:10, Peter Dimov via Boost <boost@lists.boost.org> wrote:
Ruben Perez wrote:
3. Some libraries support configuration macros. For instance, Boost::throw_exception supports BOOST_NO_EXCEPTIONS. This is used by other Boost libraries to support builds that disable exceptions. In the current world, the end user defines BOOST_NO_EXCEPTIONS in their code, includes the library, and everything works. Using a scheme like the one I suggested requires the user to build all the Boost modules using BOOST_NO_EXCEPTIONS, which is more involved than what we have today.
BOOST_NO_EXCEPTIONS is not defined by the user. It's defined by Boost.Config when -fno-exceptions or an equivalent is used.
Thanks for the clarification. I think the same considerations still apply: if the user wants -fno-exceptions, they would need to build the Boost modules with that flag if we follow what the CMake guys recommended. Thanks, Ruben.
Ruben Perez wrote:
On Sun, 22 Dec 2024 at 13:10, Peter Dimov via Boost <boost@lists.boost.org> wrote:
Ruben Perez wrote:
3. Some libraries support configuration macros. For instance, Boost::throw_exception supports BOOST_NO_EXCEPTIONS. This is used
by
other Boost libraries to support builds that disable exceptions. In the current world, the end user defines BOOST_NO_EXCEPTIONS in their code, includes the library, and everything works. Using a scheme like the one I suggested requires the user to build all the Boost modules using BOOST_NO_EXCEPTIONS, which is more involved than what we have today.
BOOST_NO_EXCEPTIONS is not defined by the user. It's defined by Boost.Config when -fno-exceptions or an equivalent is used.
Thanks for the clarification.
I think the same considerations still apply: if the user wants -fno-exceptions, they would need to build the Boost modules with that flag if we follow what the CMake guys recommended.
I was going to say that the same applies to the -std level, but it occurs to me that in order to build and use modules, you need at least -std=c++20 anyway so most such issues won't manifest for now, until we get conditional uses of C++23 or C++26. We also have an interesting problem with -fno-exceptions; the function boost::throw_exception in that case is only declared on our side, but defined by the user. This (I think) means that the declaration can't be exported by the module, it needs to appear in the GMF. Maybe. At the beginning I'd probably not bother with modularizing Assert (which mostly exports macros anyway) or ThrowException. By the way, I posted about the optional `export` idea to the committee mailing list, and Michael Spencer replied and basically said "don't". "My suggestion is to do either `export import <boost/foo.hpp>;` in the module if you are ok with getting extra declarations, or `export using ...` like libc++ does. Trying to textually include a header into the purview of a named module is fraught with issues." Since everything that's textually included becomes attached to the named module, it's too easy to create ODR violations. But, apparently, Clang doesn't like header units, so the above is also not quite optimal.
I was going to say that the same applies to the -std level, but it occurs to me that in order to build and use modules, you need at least -std=c++20 anyway so most such issues won't manifest for now, until we get conditional uses of C++23 or C++26.
In the prototype I'm writing, it's C++23, since everything uses import std. As you have mentioned many times, that's the only way to cut build times.
We also have an interesting problem with -fno-exceptions; the function boost::throw_exception in that case is only declared on our side, but defined by the user. This (I think) means that the declaration can't be exported by the module, it needs to appear in the GMF. Maybe.
I haven't reached this yet, but my idea was to declare the function as extern "C++", which AFAIK attaches it to the global module, even if it's declared outside of it.
By the way, I posted about the optional `export` idea to the committee mailing list, and Michael Spencer replied and basically said "don't".
"My suggestion is to do either `export import <boost/foo.hpp>;` in the module if you are ok with getting extra declarations, or `export using ...` like libc++ does. Trying to textually include a header into the purview of a named module is fraught with issues."
Since everything that's textually included becomes attached to the named module, it's too easy to create ODR violations.
But, apparently, Clang doesn't like header units, so the above is also not quite optimal.
I'd like to understand more about this, since this is the approach I have been following (and AFAIK, the one that John Maddock and Matt Borland followed, too). What would be the possible ODR violations here? Does he refer to the case where we forget to #ifdef-out an include for a dependency? It looks like `export import <boost/foo.hpp>;` could be done easily by users (if it worked). At the minute, `export using` has problems: 1) It doesn't work properly under MSVC because of an apparent language bug regarding specializations [2] [3]. 2) It ends up placing *a lot* of declarations in the global module fragment. clang docs suggest to avoid this [1], and I recall someone on the ML argued in this direction, too, but I can't find the relevant message. This is what I currently have. It's incomplete and will get some more work today: https://github.com/anarthal/boost-cmake/tree/feature/cxx20-modules https://github.com/anarthal/config/tree/feature/cxx20-modules https://github.com/anarthal/assert/tree/feature/cxx20-modules https://github.com/anarthal/throw_exception/tree/feature/cxx20-modules https://github.com/anarthal/mp11/tree/feature/cxx20-modules https://github.com/anarthal/core/tree/feature/cxx20-modules Thanks, Ruben.
[1] https://clang.llvm.org/docs/StandardCPlusPlusModules.html#reduce-duplication... [2] https://anarthal.github.io/cppblog/modules2#template-spec [3] https://github.com/anarthal/cppblog/issues/1
Ruben Perez wrote:
By the way, I posted about the optional `export` idea to the committee mailing list, and Michael Spencer replied and basically said "don't".
"My suggestion is to do either `export import <boost/foo.hpp>;` in the module if you are ok with getting extra declarations, or `export using ...` like libc++ does. Trying to textually include a header into the purview of a named module is fraught with issues."
Since everything that's textually included becomes attached to the named module, it's too easy to create ODR violations.
But, apparently, Clang doesn't like header units, so the above is also not quite optimal.
I'd like to understand more about this, since this is the approach I have been following (and AFAIK, the one that John Maddock and Matt Borland followed, too). What would be the possible ODR violations here? Does he refer to the case where we forget to #ifdef-out an include for a dependency?
Yes. Assume for a moment that we don't have `import std;` available. In that case, an #include <algorithm> from one of our headers would attach all its declarations and definitions to our named module. When a user imports our module and includes <algorithm>, that's an ODR violation. The way to avoid this is to include <algorithm>, along with everything else, in the GMF of our named module, so that the includes inside the module become no-ops. But we could easily miss such an include. This would also apply to Boost libraries that aren't modules, such as Config. This problem seems to make modules an all or nothing proposition. If one Boost library is a module, everything else also must be. We can't really mix and match modules and includes on a per-library basis as I envisaged. And things would be simplified considerably if we can also extend this to the standard library. It's either both import boost.* and import std, or neither.
It looks like `export import <boost/foo.hpp>;` could be done easily by users (if it worked).
At the source level yes, but not on the CMake level.
At the minute, `export using` has problems: 1) It doesn't work properly under MSVC because of an apparent language bug regarding specializations [2] [3]. 2) It ends up placing *a lot* of declarations in the global module fragment. clang docs suggest to avoid this [1], and I recall someone on the ML argued in this direction, too, but I can't find the relevant message.
IIUC, this is still better for performance than mixing named modules and includes, even if the extern "C++" trick is used to avoid the ODR violations. And I'm not sure that we can realistically avoid mixing imports and includes.
On Sun, 22 Dec 2024 at 16:45, Peter Dimov <pdimov@gmail.com> wrote:
Ruben Perez wrote:
By the way, I posted about the optional `export` idea to the committee mailing list, and Michael Spencer replied and basically said "don't".
"My suggestion is to do either `export import <boost/foo.hpp>;` in the module if you are ok with getting extra declarations, or `export using ...` like libc++ does. Trying to textually include a header into the purview of a named module is fraught with issues."
Since everything that's textually included becomes attached to the named module, it's too easy to create ODR violations.
But, apparently, Clang doesn't like header units, so the above is also not quite optimal.
I'd like to understand more about this, since this is the approach I have been following (and AFAIK, the one that John Maddock and Matt Borland followed, too). What would be the possible ODR violations here? Does he refer to the case where we forget to #ifdef-out an include for a dependency?
Yes. Assume for a moment that we don't have `import std;` available. In that case, an #include <algorithm> from one of our headers would attach all its declarations and definitions to our named module. When a user imports our module and includes <algorithm>, that's an ODR violation.
The way to avoid this is to include <algorithm>, along with everything else, in the GMF of our named module, so that the includes inside the module become no-ops. But we could easily miss such an include.
I see. I don't know if it's the same thing, but this looks similar to what happens when you mix imports and includes for the standard library, at least in MSVC. I know it's not supposed to be like this, but it is (at least now). I wonder whether there is an (automated) way to check for these, as it's definitely error-prone. My guess is that once export using works well, we can replace what I'm writing by export using (so we gain resilience), while keeping all the conditional includes I've introduced (to reduce overhead). So it wouldn't be wasted work, after all.
This would also apply to Boost libraries that aren't modules, such as Config.
This problem seems to make modules an all or nothing proposition. If one Boost library is a module, everything else also must be. We can't really mix and match modules and includes on a per-library basis as I envisaged. And things would be simplified considerably if we can also extend this to the standard library. It's either both import boost.* and import std, or neither.
What I've written until now assumes exactly this. It may be a lot to assume, but if you do need to mix, you can just use headers. I may be completely wrong, and mixing includes and imports will become important in the future. It felt fair to make this assumption (at least now) because the standard library seems to do it, too.
It looks like `export import <boost/foo.hpp>;` could be done easily by users (if it worked).
At the source level yes, but not on the CMake level.
At the minute, `export using` has problems: 1) It doesn't work properly under MSVC because of an apparent language bug regarding specializations [2] [3]. 2) It ends up placing *a lot* of declarations in the global module fragment. clang docs suggest to avoid this [1], and I recall someone on the ML argued in this direction, too, but I can't find the relevant message.
IIUC, this is still better for performance than mixing named modules and includes, even if the extern "C++" trick is used to avoid the ODR violations. And I'm not sure that we can realistically avoid mixing imports and includes.
Ruben Perez wrote:
This problem seems to make modules an all or nothing proposition. If one Boost library is a module, everything else also must be. We can't really mix and match modules and includes on a per-library basis as I envisaged. And things would be simplified considerably if we can also extend this to the standard library. It's either both import boost.* and import std, or neither.
What I've written until now assumes exactly this. It may be a lot to assume, but if you do need to mix, you can just use headers. I may be completely wrong, and mixing includes and imports will become important in the future. It felt fair to make this assumption (at least now) because the standard library seems to do it, too.
In the general case, the user will have to build all of his dependencies using a consistent setting of the "use Boost modules" macro, because otherwise a prebuilt library may well include something Boost. Which is going to be a problem if that prebuilt library is installed by a package manager, especially the system package manager. I suppose that's already a problem a theory, but modules might be more sensitive to it.
Ruben Perez wrote:
This is what I currently have. It's incomplete and will get some more work today:
https://github.com/anarthal/boost-cmake/tree/feature/cxx20-modules https://github.com/anarthal/config/tree/feature/cxx20-modules https://github.com/anarthal/assert/tree/feature/cxx20-modules https://github.com/anarthal/throw_exception/tree/feature/cxx20-modules https://github.com/anarthal/mp11/tree/feature/cxx20-modules https://github.com/anarthal/core/tree/feature/cxx20-modules
I don't think BOOST_CXX20_MODULE is the right name. Macros prefixed with BOOST_CXXnn_ indicate feature availability. The modules macro would be called BOOST_CXX20_NO_MODULES, because we use negative macros. (And BOOST_CXX23_NO_STD_MODULE, respectively.) What we need here is not an availability macro (are C++20 modules supported by the compiler?) but a user configuration macro (I want to use Boost as modules.) It should be something like BOOST_USE_MODULES, for instance.
On Sun, 22 Dec 2024 at 17:06, Peter Dimov <pdimov@gmail.com> wrote:
Ruben Perez wrote:
This is what I currently have. It's incomplete and will get some more work today:
https://github.com/anarthal/boost-cmake/tree/feature/cxx20-modules https://github.com/anarthal/config/tree/feature/cxx20-modules https://github.com/anarthal/assert/tree/feature/cxx20-modules https://github.com/anarthal/throw_exception/tree/feature/cxx20-modules https://github.com/anarthal/mp11/tree/feature/cxx20-modules https://github.com/anarthal/core/tree/feature/cxx20-modules
I don't think BOOST_CXX20_MODULE is the right name. Macros prefixed with BOOST_CXXnn_ indicate feature availability. The modules macro would be called BOOST_CXX20_NO_MODULES, because we use negative macros.
(And BOOST_CXX23_NO_STD_MODULE, respectively.)
What we need here is not an availability macro (are C++20 modules supported by the compiler?) but a user configuration macro (I want to use Boost as modules.)
It should be something like BOOST_USE_MODULES, for instance.
Completely agree. I will change it.
Ruben Perez wrote:
This is what I currently have. It's incomplete and will get some more work today:
https://github.com/anarthal/boost-cmake/tree/feature/cxx20-modules https://github.com/anarthal/config/tree/feature/cxx20-modules https://github.com/anarthal/assert/tree/feature/cxx20-modules https://github.com/anarthal/throw_exception/tree/feature/cxx20-modules https://github.com/anarthal/mp11/tree/feature/cxx20-modules https://github.com/anarthal/core/tree/feature/cxx20-modules
Regarding boost_set_cxx20_module_settings(boost_assert) You can't rely on this function being available in a library's CMakeLists.txt file. Libraries (except those by Andrey) can be used without the superproject.
On Sun, 22 Dec 2024, 17:10 Peter Dimov, <pdimov@gmail.com> wrote:
Ruben Perez wrote:
This is what I currently have. It's incomplete and will get some more work today:
https://github.com/anarthal/boost-cmake/tree/feature/cxx20-modules https://github.com/anarthal/config/tree/feature/cxx20-modules https://github.com/anarthal/assert/tree/feature/cxx20-modules https://github.com/anarthal/throw_exception/tree/feature/cxx20-modules https://github.com/anarthal/mp11/tree/feature/cxx20-modules https://github.com/anarthal/core/tree/feature/cxx20-modules
Regarding
boost_set_cxx20_module_settings(boost_assert)
You can't rely on this function being available in a library's CMakeLists.txt file. Libraries (except those by Andrey) can be used without the superproject.
My assumption here was "modules only work with the superproject, either via add_subdirectory or via install + find_package". I was likely too naive, and the restriction should probably be lifted. How about providing the function in a helper cmake script which must be included by code using modules? We can always copy-paste the settings, but it can be a little bit of a mess.
Ruben Perez wrote:
On Sun, 22 Dec 2024, 17:10 Peter Dimov, <pdimov@gmail.com <mailto:pdimov@gmail.com> > wrote:
Regarding
boost_set_cxx20_module_settings(boost_assert)
You can't rely on this function being available in a library's CMakeLists.txt file. Libraries (except those by Andrey) can be used without the superproject.
My assumption here was "modules only work with the superproject, either via add_subdirectory or via install + find_package". I was likely too naive, and the restriction should probably be lifted. How about providing the function in a helper cmake script which must be included by code using modules? We can always copy-paste the settings, but it can be a little bit of a mess.
At some point, once this stabilizes, we'll probably put it in the `boostdep --cmake` output. But I agree that while developing it's more convenient to only have to change it in one place.
On Sun, Dec 22, 2024 at 9:30 AM Ruben Perez via Boost <boost@lists.boost.org> wrote:
I was going to say that the same applies to the -std level, but it occurs to me that in order to build and use modules, you need at least -std=c++20 anyway so most such issues won't manifest for now, until we get conditional uses of C++23 or C++26.
In the prototype I'm writing, it's C++23, since everything uses import std. As you have mentioned many times, that's the only way to cut build times.
Also, vendors have made the std module available in C++20. -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supongas Nada -- Robot Dreams - http://robot-dreams.net
> Also, vendors have made the std module available in C++20. Yes. Given libstdc++ has std module too now. I think it makes sense to only import std in modules code. As someone mentioned somewhere else, it is slightly harder to maintain 3 modes: 1. traditional header modes. 2. modules modes with import std available. 3. modules modes with import std not available. Given how important std module is, I think it makes sense to assume import std is available to reduce some maintain burden. Thanks, Chuanqi ------------------------------------------------------------------ From:René Ferdinand Rivera Morell via Boost <boost@lists.boost.org> Send Time:2024 Dec. 23 (Mon.) 03:21 To:boost<boost@lists.boost.org> Cc:"René Ferdinand Rivera Morell"<grafikrobot@gmail.com>; Peter Dimov<pdimov@gmail.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially On Sun, Dec 22, 2024 at 9:30 AM Ruben Perez via Boost <boost@lists.boost.org> wrote: > > I was going to say that the same applies to the -std level, but it > occurs to me > > that in order to build and use modules, you need at least -std=c++20 > anyway > > so most such issues won't manifest for now, until we get conditional uses > > of C++23 or C++26. > > In the prototype I'm writing, it's C++23, since everything uses import > std. As you have mentioned many times, that's the only way to cut > build times. > Also, vendors have made the std module available in C++20. -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supongas Nada -- Robot Dreams - http://robot-dreams.net <http://robot-dreams.net > _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost <http://lists.boost.org/mailman/listinfo.cgi/boost >
> I'd like to understand more about this, since this is the approach I have been following (and AFAIK, the one that John Maddock and Matt Borland followed, too). What would be the possible ODR violations here? Does he refer to the case where we forget to #ifdef-out an include for a dependency? The ODR violation is a misunderstanding or a miscommunication. The ODR violation can go away if it was wrapped by `extern “C++”`. > Does he refer to the case where we forget to #ifdef-out an include for a dependency? And yes, it seems like they mislooked this. I talked this privately with Mathias. (He mentioned this first in that mailing list). We can achieve the most ideal performance by #ifdef-out all the third party headers as the we do now. > It looks like `export import <boost/foo.hpp>;` could be done easily by users (if it worked). Paste some reply from EWG’s mailing list: The key point for the efficiency of modules is, we should avoid redeclarations in different BMIs. For example, for, ``` import <string>; import <vector>; ... ``` For a naive support, that may compiles `string` to `string.pcm` and compiles `vector` to `vector.pcm` and load these 2 pcms to the current TU. This is not efficient enough. Since `string.pcm` and `vector.pcm` may contain duplicated redeclarations in common included headers. e.g., declarations in allocators. To avoid this, the tools must be able to replace the inclusion of every importable header to the import of the corresponding header units. So that `string.pcm` and `vector.pcm` can share the declarations in `allocator.pcm`. In another word, when tools see `import <header.h>`, the tools need to convert **every** indirectly included headers to header units to achieve the best efficiency. So the conclusion is, for header units to work efficiently enough, the tools must be able to replace every (importable) textual inclusion to import for header units if possible. But the tools are far away from that state. CMake even doesn’t support the most trivial case. And other build systems which claim supporting header units may only support the trivial case too. My personal opinion is, to not care about it until the tools support it. And the reason why I suggest #ifdef-out is, it actually did the above process to avoid redeclarations manually. And its refactoring costs and maintaining costs is low. So I feel the current method is fine and workable. > https://github.com/anarthal/boost-cmake/tree/feature/cxx20-modules <https://github.com/anarthal/boost-cmake/tree/feature/cxx20-modules > > https://github.com/anarthal/config/tree/feature/cxx20-modules <https://github.com/anarthal/config/tree/feature/cxx20-modules > > https://github.com/anarthal/assert/tree/feature/cxx20-modules <https://github.com/anarthal/assert/tree/feature/cxx20-modules > > https://github.com/anarthal/throw_exception/tree/feature/cxx20-modules <https://github.com/anarthal/throw_exception/tree/feature/cxx20-modules > > https://github.com/anarthal/mp11/tree/feature/cxx20-modules <https://github.com/anarthal/mp11/tree/feature/cxx20-modules > > https://github.com/anarthal/core/tree/feature/cxx20-modules <https://github.com/anarthal/core/tree/feature/cxx20-modules > Quick action! And some of my suggestion is: 1. Use `.cppm` or even `.ixx` for the module interface units. Although build systems don’t force it, the suffix makes much easier to read by human beings. e.g., we can perform simple search for the use of modules by searching `*.cppm` in search engines. 2. Wrap the inclusion of headers in the module purview with `extern “C++”`. https://github.com/anarthal/throw_exception/blob/d32c0f3961934d15bbb938bb39d01bfce2653266/modules/throw_exception.cxx#L9 <https://github.com/anarthal/throw_exception/blob/d32c0f3961934d15bbb938bb39d01bfce2653266/modules/throw_exception.cxx#L9 > 3. Don’t import in headers: https://github.com/anarthal/throw_exception/blob/d32c0f3961934d15bbb938bb39d01bfce2653266/include/boost/throw_exception.hpp#L23-L24 <https://github.com/anarthal/throw_exception/blob/d32c0f3961934d15bbb938bb39d01bfce2653266/include/boost/throw_exception.hpp#L23-L24 > but import them in module interface. Thanks, Chuanqi ------------------------------------------------------------------ From:Ruben Perez via Boost <boost@lists.boost.org> Send Time:2024 Dec. 22 (Sun.) 23:30 To:Peter Dimov<pdimov@gmail.com> Cc:Ruben Perez<rubenperez038@gmail.com>; boost<boost@lists.boost.org> Subject:Re: [boost] Interest for C++20 modules support of boost officially > I was going to say that the same applies to the -std level, but it occurs to me > that in order to build and use modules, you need at least -std=c++20 anyway > so most such issues won't manifest for now, until we get conditional uses > of C++23 or C++26. In the prototype I'm writing, it's C++23, since everything uses import std. As you have mentioned many times, that's the only way to cut build times. > > We also have an interesting problem with -fno-exceptions; the function > boost::throw_exception in that case is only declared on our side, but defined > by the user. This (I think) means that the declaration can't be exported by the > module, it needs to appear in the GMF. Maybe. I haven't reached this yet, but my idea was to declare the function as extern "C++", which AFAIK attaches it to the global module, even if it's declared outside of it. > > By the way, I posted about the optional `export` idea to the committee mailing > list, and Michael Spencer replied and basically said "don't". > > "My suggestion is to do either `export import <boost/foo.hpp>;` in the module > if you are ok with getting extra declarations, or `export using ...` like libc++ does. > Trying to textually include a header into the purview of a named module is > fraught with issues." > > Since everything that's textually included becomes attached to the named > module, it's too easy to create ODR violations. > > But, apparently, Clang doesn't like header units, so the above is also not quite > optimal. I'd like to understand more about this, since this is the approach I have been following (and AFAIK, the one that John Maddock and Matt Borland followed, too). What would be the possible ODR violations here? Does he refer to the case where we forget to #ifdef-out an include for a dependency? It looks like `export import <boost/foo.hpp>;` could be done easily by users (if it worked). At the minute, `export using` has problems: 1) It doesn't work properly under MSVC because of an apparent language bug regarding specializations [2] [3]. 2) It ends up placing *a lot* of declarations in the global module fragment. clang docs suggest to avoid this [1], and I recall someone on the ML argued in this direction, too, but I can't find the relevant message. This is what I currently have. It's incomplete and will get some more work today: https://github.com/anarthal/boost-cmake/tree/feature/cxx20-modules <https://github.com/anarthal/boost-cmake/tree/feature/cxx20-modules > https://github.com/anarthal/config/tree/feature/cxx20-modules <https://github.com/anarthal/config/tree/feature/cxx20-modules > https://github.com/anarthal/assert/tree/feature/cxx20-modules <https://github.com/anarthal/assert/tree/feature/cxx20-modules > https://github.com/anarthal/throw_exception/tree/feature/cxx20-modules <https://github.com/anarthal/throw_exception/tree/feature/cxx20-modules > https://github.com/anarthal/mp11/tree/feature/cxx20-modules <https://github.com/anarthal/mp11/tree/feature/cxx20-modules > https://github.com/anarthal/core/tree/feature/cxx20-modules <https://github.com/anarthal/core/tree/feature/cxx20-modules > Thanks, Ruben. > > [1] https://clang.llvm.org/docs/StandardCPlusPlusModules.html#reduce-duplications <https://clang.llvm.org/docs/StandardCPlusPlusModules.html#reduce-duplications > [2] https://anarthal.github.io/cppblog/modules2#template-spec <https://anarthal.github.io/cppblog/modules2#template-spec > [3] https://github.com/anarthal/cppblog/issues/1 <https://github.com/anarthal/cppblog/issues/1 > _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost <http://lists.boost.org/mailman/listinfo.cgi/boost >
2. Wrap the inclusion of headers in the module purview with `extern “C++”`.
It seems this solves a lot of issues and as such has been used quite widely up to now (e.g. MS STL, fmtlib's WIP module support, I believe the prototype module for Boost.Regex also provided this as a configuration option). And indeed, it would be a requirement if there was a desire to support the case of consuming boost modules in a project that also had other third party dependencies which themselves used boost via headers. So my question is, are you aware what the drawbacks are of attaching everything to the global module? Clearly it feels like a hack and I assume maybe there are downsides relating to implementation efficiency, but as yet I've not heard anything concrete.
Hi all, I finally have something workable regarding Boost and C++20 modules. I've described it in detail in this article [1], but here's a rough summary: I've modularized Boost.Mp11 [2] (as an example of a header-only library) and Boost.Charconv [3] (as an example of a compiled library). I know that Charconv is not very useful in C++20, but it has few dependencies and is relatively simple. I've also changed all tests in these libraries to consume them as C++20 modules in modular builds, and added CI jobs. In the process, I've found a couple of what I think are MSVC bugs: * An excess in pruning template specializations declared in the GMF, even when they are (I think) decl-reachable from the purview [4]. * A problem when an exported using declaration targets another using declaration declared in the GMF [5]. (I've reported the bugs this past weekend, so they might be pending Microsoft's approval when you read this. I've pasted the bug report descriptions at the bottom of the email, just in case). Could any module expert confirm that these are indeed bugs? Other than that, I'd like to get some feedback on this, and whether you think this proposal is worth the effort. Many thanks, Ruben. [1] https://anarthal.github.io/cppblog/modules3 [2] https://github.com/boostorg/mp11/pull/104 [3] https://github.com/boostorg/charconv/pull/255 [4] https://developercommunity.visualstudio.com/t/C20-modules:-specialzations-in... Declarations in the global module fragment should not be discarded if they are decl-reachable from an exported entity within the module (http://eel.is/c++draft/module.global.frag). MSVC doesn't seem to apply this rule to template specializations. Minimal reproduction follows. Project layout: /mylib/include/third_party.hpp /mylib/include/mylib.hpp /mylib/modules/mylib.cppm /main.cpp Contents of /mylib/include/third_party.hpp: namespace third_party { template <class T> struct is_error_code_enum { inline static constexpr bool value = false; }; } Contents of /mylib/include/mylib.hpp: #include <third_party.hpp> namespace mylib { enum class myenum { v = 1 }; } template<> struct third_party::is_error_code_enum<mylib::myenum> { inline static constexpr bool value = true; }; Contents of /mylib/modules/mylib.cppm: module; #include <mylib.hpp> export module mylib; export namespace mylib { using mylib::myenum; void f(third_party::is_error_code_enum<mylib::myenum>) {} // specialization is decl-rechable } Contents of /main.cpp: #include <mylib/include/third_party.hpp> import mylib; int main(){ static_assert(third_party::is_error_code_enum<mylib::myenum::value>); // fails } [5] https://developercommunity.visualstudio.com/t/MSVC-C20-modules:-export-using... When a template alias is defined in the global module fragment, then exported with the export using technique, and finally imported in a source file, the compiler fails to parse the importing source file. Project setup: /mylib/include/mylib.hpp /mylib/modules/mylib.cppm /main.cpp Contents of /mylib/include/mylib.hpp: namespace mylib { template<typename T, T N> struct some_constant {}; template int N using mp_size_t = some_constant<int, N>; } Contents of /mylib/modules/mylib.cppm module; #include <mylib.hpp> export module mylib; export namespace mylib { using mylib::mp_size_t; } Contents of /main.cpp: import mylib; int main(){ mylib::mp_size_t<0> var; } Compiler error: [redacted]\main.cpp(4): error C2059: syntax error: '<'
Ruben Perez wrote:
Hi all,
I finally have something workable regarding Boost and C++20 modules. I've described it in detail in this article [1], but here's a rough summary:
I've modularized Boost.Mp11 [2] (as an example of a header-only library) and Boost.Charconv [3] (as an example of a compiled library). ... Other than that, I'd like to get some feedback on this, and whether you think this proposal is worth the effort.
I looked at the Mp11 patch, and I think I don't quite like this approach. At present, user code needs to look like this #ifdef BOOST_USE_MODULES import boost.mp11; import boost.core; import std; #else #include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <standard_header> #endif and I'm not sure that this is what we want. Rather, my current thinking is that we want this for user code: #include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <standard_header> with boost/mp11/this.hpp being #ifdef BOOST_USE_MODULES import boost.mp11; #else #include <boost/mp11/detail/this.hpp> #endif This would presume that <standard_header> does something similar, which is in fact one suggested implementation strategy. If not, we'll need #include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <boost/std/standard_header.hpp> which is not ideal, but is still better than what we started with. In the general case (when a header also defines macros), the public header will contain both import boost.mp11; and the part that defines the macros, which in Mp11's case is #include <boost/mp11/version.hpp> User code wouldn't need to know that in addition to the import, it also needs to include a separate macro header.
On Sun, 12 Jan 2025, 22:10 Peter Dimov via Boost, <boost@lists.boost.org> wrote:
Hi all,
I finally have something workable regarding Boost and C++20 modules. I've described it in detail in this article [1], but here's a rough summary:
I've modularized Boost.Mp11 [2] (as an example of a header-only library) and Boost.Charconv [3] (as an example of a compiled library). ... Other than that, I'd like to get some feedback on this, and whether you
Ruben Perez wrote: think
this proposal is worth the effort.
I looked at the Mp11 patch, and I think I don't quite like this approach.
At present, user code needs to look like this
#ifdef BOOST_USE_MODULES import boost.mp11; import boost.core; import std; #else #include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <standard_header> #endif
and I'm not sure that this is what we want.
Rather, my current thinking is that we want this for user code:
#include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <standard_header>
with boost/mp11/this.hpp being
#ifdef BOOST_USE_MODULES import boost.mp11; #else #include <boost/mp11/detail/this.hpp> #endif
This would presume that <standard_header> does something similar, which is in fact one suggested implementation strategy.
If not, we'll need
#include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <boost/std/standard_header.hpp>
which is not ideal, but is still better than what we started with.
In the general case (when a header also defines macros), the public header will contain both
import boost.mp11;
and the part that defines the macros, which in Mp11's case is
#include <boost/mp11/version.hpp>
User code wouldn't need to know that in addition to the import, it also needs to include a separate macro header.
This is an interesting idea. Standard library headers don't do this (although making them do it would be... relieving). I will have a try to your approach next week. Thanks, Ruben.
While benchmarks seem promising, the technology still looks very experimental. I think it makes sense for us to wait until the bugs I’ve found are fixed, and CMake support for import std become stable, before merging any of my work. (Assuming the boost reviewers feel good about the approach). I am wondering if it is possible to release it as experimental. So that people who loves to live at the bleeding edge can get the benefit early and provide feedbacks early. Thanks, Chuanqi
From:Ruben Perez via Boost <boost@lists.boost.org> Send Time:2025 Jan. 13 (Mon.) 05:15 To:"boost@lists.boost.org List"<boost@lists.boost.org> CC:Ruben Perez<rubenperez038@gmail.com>; Peter Dimov<pdimov@gmail.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially On Sun, 12 Jan 2025, 22:10 Peter Dimov via Boost, <boost@lists.boost.org> wrote:
Hi all,
I finally have something workable regarding Boost and C++20 modules. I've described it in detail in this article [1], but here's a rough summary:
I've modularized Boost.Mp11 [2] (as an example of a header-only library) and Boost.Charconv [3] (as an example of a compiled library). ... Other than that, I'd like to get some feedback on this, and whether you
Ruben Perez wrote: think
this proposal is worth the effort.
I looked at the Mp11 patch, and I think I don't quite like this approach.
At present, user code needs to look like this
#ifdef BOOST_USE_MODULES import boost.mp11; import boost.core; import std; #else #include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <standard_header> #endif
and I'm not sure that this is what we want.
Rather, my current thinking is that we want this for user code:
#include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <standard_header>
with boost/mp11/this.hpp being
#ifdef BOOST_USE_MODULES import boost.mp11; #else #include <boost/mp11/detail/this.hpp> #endif
This would presume that <standard_header> does something similar, which is in fact one suggested implementation strategy.
If not, we'll need
#include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <boost/std/standard_header.hpp>
which is not ideal, but is still better than what we started with.
In the general case (when a header also defines macros), the public header will contain both
import boost.mp11;
and the part that defines the macros, which in Mp11's case is
#include <boost/mp11/version.hpp>
User code wouldn't need to know that in addition to the import, it also needs to include a separate macro header.
This is an interesting idea. Standard library headers don't do this (although making them do it would be... relieving). I will have a try to your approach next week. Thanks, Ruben. _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost <http://lists.boost.org/mailman/listinfo.cgi/boost >
#ifdef BOOST_USE_MODULES import boost.mp11; import boost.core; import std; #else #include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <standard_header> #endif
and I'm not sure that this is what we want. I feel this may not be too bad from a user’s perspective. Rather, my current thinking is that we want this for user code: #include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <standard_header> On the one hand, I feel the readability is not so good. On the other hand, the standard wording may not want this: In a module unit, all module-import-declarations and export-declarations exporting module-import-declarations shall appear before all other declarations in the declaration-seq of the translation-unit and of the private-module-fragment (if any). from https://eel.is/c++draft/module.import#1 <https://eel.is/c++draft/module.import#1 >https://eel.is/c++draft/module.import#1 <https://eel.is/c++draft/module.import#1 > Thanks, Chuanqi
Hi all,
I finally have something workable regarding Boost and C++20 modules. I've described it in detail in this article [1], but here's a rough summary:
I've modularized Boost.Mp11 [2] (as an example of a header-only library) and Boost.Charconv [3] (as an example of a compiled library). ... Other than that, I'd like to get some feedback on this, and whether you think this proposal is worth the effort. I looked at the Mp11 patch, and I think I don't quite like this approach. At present, user code needs to look like this #ifdef BOOST_USE_MODULES import boost.mp11; import boost.core; import std; #else #include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <standard_header> #endif and I'm not sure that this is what we want. Rather, my current thinking is that we want this for user code: #include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <standard_header> with boost/mp11/this.hpp being #ifdef BOOST_USE_MODULES import boost.mp11; #else #include <boost/mp11/detail/this.hpp> #endif This would presume that <standard_header> does something similar, which is in fact one suggested implementation strategy. If not, we'll need #include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <boost/std/standard_header.hpp> which is not ideal, but is still better than what we started with. In the general case (when a header also defines macros), the public
From:Peter Dimov via Boost <boost@lists.boost.org> Send Time:2025 Jan. 13 (Mon.) 05:10 To:boost<boost@lists.boost.org> CC:Peter Dimov<pdimov@gmail.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially Ruben Perez wrote: header will contain both import boost.mp11; and the part that defines the macros, which in Mp11's case is #include <boost/mp11/version.hpp> User code wouldn't need to know that in addition to the import, it also needs to include a separate macro header. _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost <http://lists.boost.org/mailman/listinfo.cgi/boost >
Chuanqi Xu wrote:
Rather, my current thinking is that we want this for user code:
#include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <standard_header>
On the one hand, I feel the readability is not so good. On the other hand, the standard wording may not want this:
In a module unit, all module-import-declarations and export-declarations exporting module-import-declarations shall appear before all other declarations in the declaration-seq of the translation-unit and of the private- module-fragment (if any).
That's in a module unit, which the above isn't. In a module unit, you'd just write import boost.mp11; import boost.core; import std; because you know you're using modules. Of course if you do that you're no longer insulated from which libraries are modular and which aren't. If the abovementioned headers are written "correctly", the requirement can be met even with the include form, provided that the includes are sorted so that all modular libraries come before all non-modular ones. This is the reverse of the usual order though.
Of course if you do that you're no longer insulated from which libraries are modular and which aren’t. Yeah, this is my point. As a user, I don’t mind some additional typings but prefer to see things explicitly and clearly. Thanks, Chuanqi
From:Peter Dimov <pdimov@gmail.com> Send Time:2025 Jan. 13 (Mon.) 10:29 To:Chuanqi<chuanqi.xcq@alibaba-inc.com>; Peter Dimov via Boost<boost@lists.boost.org> Subject:RE: [boost] Interest for C++20 modules support of boost officially Chuanqi Xu wrote:
Rather, my current thinking is that we want this for user code:
#include <boost/mp11/this.hpp> #include <boost/core/that.hpp> #include <standard_header>
On the one hand, I feel the readability is not so good. On the other hand, the standard wording may not want this:
In a module unit, all module-import-declarations and export-declarations exporting module-import-declarations shall appear before all other declarations in the declaration-seq of the translation-unit and of the private- module-fragment (if any). That's in a module unit, which the above isn't. In a module unit, you'd just write import boost.mp11; import boost.core; import std; because you know you're using modules. Of course if you do that you're no longer insulated from which libraries are modular and which aren't. If the abovementioned headers are written "correctly", the requirement can be met even with the include form, provided that the includes are sorted so that all modular libraries come before all non-modular ones. This is the reverse of the usual order though.
In the general case (when a header also defines macros), the public header will contain both
import boost.mp11;
and the part that defines the macros, which in Mp11's case is
#include <boost/mp11/version.hpp>
User code wouldn't need to know that in addition to the import, it also needs to include a separate macro header.
This assumes that macros are "standalone" and don't require including any stdlib headers. This is the case for mp11/version.hpp and lightweight test, but is not for Boost.Config or Boost.Charconv headers, for instance. These may require including <cstddef> or <cstdint> for macros, which need to happen before any import std. Wouldn't this cause trouble with your scheme? Thanks, Rubén.
Ruben Perez wrote:
In the general case (when a header also defines macros), the public header will contain both
import boost.mp11;
and the part that defines the macros, which in Mp11's case is
#include <boost/mp11/version.hpp>
User code wouldn't need to know that in addition to the import, it also needs to include a separate macro header.
This assumes that macros are "standalone" and don't require including any stdlib headers. This is the case for mp11/version.hpp and lightweight test, but is not for Boost.Config or Boost.Charconv headers, for instance. These may require including <cstddef> or <cstdint> for macros, which need to happen before any import std. Wouldn't this cause trouble with your scheme?
Possibly, yes. If we want to keep the public headers usable in a module unit, we can't include <cstddef> there because it will introduce declarations. Although whether we need to keep the headers usable in a module unit is up for debate. Fixing this properly can only be done on the stdlib/implementation side.
On Mon, 13 Jan 2025 at 14:01, Peter Dimov <pdimov@gmail.com> wrote:
Ruben Perez wrote:
In the general case (when a header also defines macros), the public header will contain both
import boost.mp11;
and the part that defines the macros, which in Mp11's case is
#include <boost/mp11/version.hpp>
User code wouldn't need to know that in addition to the import, it also needs to include a separate macro header.
This assumes that macros are "standalone" and don't require including any stdlib headers. This is the case for mp11/version.hpp and lightweight test, but is not for Boost.Config or Boost.Charconv headers, for instance. These may require including <cstddef> or <cstdint> for macros, which need to happen before any import std. Wouldn't this cause trouble with your scheme?
Possibly, yes. If we want to keep the public headers usable in a module unit, we can't include <cstddef> there because it will introduce declarations.
Although whether we need to keep the headers usable in a module unit is up for debate.
Fixing this properly can only be done on the stdlib/implementation side.
I think this applies for non module units, too. I've just tested that, under MSVC, the following errors: // File: main.cpp import std; #include <cstddef> int main() {} This does not seem to happen with all the headers, though. * version, cstdint, climits, cassert, cfloat, cstdio, cstdlib, cerrno seem to work fine * cmath and cstddef seem to error. So if we have in a program: // File: main.cpp #include <boost/mp11.hpp> #include <boost/charconv.hpp> int main() {} Under modular builds, this would be roughly equivalent to: // From boost/mp11.hpp #include <cassert> // insert here any headers required by mp11 macros #include <boost/mp11/detail/config.hpp> // the actual macro definitions import std; // generated by the standard header proxies you propose import boost.mp11; // generated by mp11 own headers // From boost/charconv.hpp #include <cmath> // insert here any headers required by charconv macros - this errors #include <boost/charconv/detail/config.hpp> // the actual macro definitions import std; // generated by the standard header proxies you propose import boost.charconv; // generated by charconv own headers int main() {} We can probably trim down most of the required stdlib headers so this is not a problem. Or just ifdef-out standard headers without the proxy headers that expand to import std.
Ruben Perez wrote:
I think this applies for non module units, too. I've just tested that, under MSVC, the following errors:
// File: main.cpp import std; #include <cstddef> int main() {}
This does not seem to happen with all the headers, though. * version, cstdint, climits, cassert, cfloat, cstdio, cstdlib, cerrno seem to work fine * cmath and cstddef seem to error.
So if we have in a program:
// File: main.cpp #include <boost/mp11.hpp> #include <boost/charconv.hpp> int main() {}
Yeah, that's going to be a problem. Interestingly, though, import std; #include <math.h> #include <stddef.h> int main() {} works fine. If we only need these for the macros, this might well work. (It's probably std::byte from cstddef that breaks things.)
On Mon, 13 Jan 2025 at 20:37, Peter Dimov <pdimov@gmail.com> wrote:
Ruben Perez wrote:
I think this applies for non module units, too. I've just tested that, under MSVC, the following errors:
// File: main.cpp import std; #include <cstddef> int main() {}
This does not seem to happen with all the headers, though. * version, cstdint, climits, cassert, cfloat, cstdio, cstdlib, cerrno seem to work fine * cmath and cstddef seem to error.
So if we have in a program:
// File: main.cpp #include <boost/mp11.hpp> #include <boost/charconv.hpp> int main() {}
Yeah, that's going to be a problem.
Interestingly, though,
import std; #include <math.h> #include <stddef.h> int main() {}
works fine.
If we only need these for the macros, this might well work.
(It's probably std::byte from cstddef that breaks things.)
I think it has to do with where entities are declared. Your snippet errors if you import std.compat (which can safely be avoided altogether in all our code without problems): import std.compat; #include <math.h> #include <stddef.h> int main() {} [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(399): error C2995: 'bool isfinite(_Ty) noexcept': function template has already been defined [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(399): note: see declaration of 'isfinite' [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(405): error C2995: 'bool isinf(_Ty) noexcept': function template has already been defined [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(405): note: see declaration of 'isinf' [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(411): error C2995: 'bool isnan(_Ty) noexcept': function template has already been defined [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(411): note: see declaration of 'isnan' [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(417): error C2995: 'bool isnormal(_Ty) noexcept': function template has already been defined [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(417): note: see declaration of 'isnormal' [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(423): error C2995: 'bool isgreater(_Ty1,_Ty2) noexcept': function template has already been defined [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(423): note: see declaration of 'isgreater' [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(429): error C2995: 'bool isgreaterequal(_Ty1,_Ty2) noexcept': function template has already been defined [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(429): note: see declaration of 'isgreaterequal' [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(435): error C2995: 'bool isless(_Ty1,_Ty2) noexcept': function template has already been defined [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(435): note: see declaration of 'isless' [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(441): error C2995: 'bool islessequal(_Ty1,_Ty2) noexcept': function template has already been defined [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(441): note: see declaration of 'islessequal' [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(447): error C2995: 'bool islessgreater(_Ty1,_Ty2) noexcept': function template has already been defined [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(447): note: see declaration of 'islessgreater' [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(453): error C2995: 'bool isunordered(_Ty1,_Ty2) noexcept': function template has already been defined [build] C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt\corecrt_math.h(453): note: see declaration of 'isunordered'
Hi, the patch looks pretty good to me. Except I think we shouldn’t disable the warnings in the CMakeLists.txt and we should disable the warning in the .cppm files. Thanks, Chuanqi ------------------------------------------------------------------ From:Ruben Perez <rubenperez038@gmail.com> Send Time:2025 Jan. 13 (Mon.) 04:51 To:boost<boost@lists.boost.org> CC:Daniela Engert<dani@ngrt.de>; Chuanqi<chuanqi.xcq@alibaba-inc.com>; "Stephan T. Lavavej"<stl@exchange.microsoft.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially Hi all, I finally have something workable regarding Boost and C++20 modules. I've described it in detail in this article [1], but here's a rough summary: I've modularized Boost.Mp11 [2] (as an example of a header-only library) and Boost.Charconv [3] (as an example of a compiled library). I know that Charconv is not very useful in C++20, but it has few dependencies and is relatively simple. I've also changed all tests in these libraries to consume them as C++20 modules in modular builds, and added CI jobs. In the process, I've found a couple of what I think are MSVC bugs: * An excess in pruning template specializations declared in the GMF, even when they are (I think) decl-reachable from the purview [4]. * A problem when an exported using declaration targets another using declaration declared in the GMF [5]. (I've reported the bugs this past weekend, so they might be pending Microsoft's approval when you read this. I've pasted the bug report descriptions at the bottom of the email, just in case). Could any module expert confirm that these are indeed bugs? Other than that, I'd like to get some feedback on this, and whether you think this proposal is worth the effort. Many thanks, Ruben. [1] https://anarthal.github.io/cppblog/modules3 <https://anarthal.github.io/cppblog/modules3 > [2] https://github.com/boostorg/mp11/pull/104 <https://github.com/boostorg/mp11/pull/104 > [3] https://github.com/boostorg/charconv/pull/255 <https://github.com/boostorg/charconv/pull/255 > [4] https://developercommunity.visualstudio.com/t/C20-modules:-specialzations-in... <https://developercommunity.visualstudio.com/t/C20-modules:-specialzations-in... > Declarations in the global module fragment should not be discarded if they are decl-reachable from an exported entity within the module (http://eel.is/c++draft/module.global.frag <http://eel.is/c++draft/module.global.frag >). MSVC doesn't seem to apply this rule to template specializations. Minimal reproduction follows. Project layout: /mylib/include/third_party.hpp /mylib/include/mylib.hpp /mylib/modules/mylib.cppm /main.cpp Contents of /mylib/include/third_party.hpp: namespace third_party { template <class T> struct is_error_code_enum { inline static constexpr bool value = false; }; } Contents of /mylib/include/mylib.hpp: #include <third_party.hpp> namespace mylib { enum class myenum { v = 1 }; } template<> struct third_party::is_error_code_enum<mylib::myenum> { inline static constexpr bool value = true; }; Contents of /mylib/modules/mylib.cppm: module; #include <mylib.hpp> export module mylib; export namespace mylib { using mylib::myenum; void f(third_party::is_error_code_enum<mylib::myenum>) {} // specialization is decl-rechable } Contents of /main.cpp: #include <mylib/include/third_party.hpp> import mylib; int main(){ static_assert(third_party::is_error_code_enum<mylib::myenum::value>); // fails } [5] https://developercommunity.visualstudio.com/t/MSVC-C20-modules:-export-using... <https://developercommunity.visualstudio.com/t/MSVC-C20-modules:-export-using... > When a template alias is defined in the global module fragment, then exported with the export using technique, and finally imported in a source file, the compiler fails to parse the importing source file. Project setup: /mylib/include/mylib.hpp /mylib/modules/mylib.cppm /main.cpp Contents of /mylib/include/mylib.hpp: namespace mylib { template<typename T, T N> struct some_constant {}; template int N using mp_size_t = some_constant<int, N>; } Contents of /mylib/modules/mylib.cppm module; #include <mylib.hpp> export module mylib; export namespace mylib { using mylib::mp_size_t; } Contents of /main.cpp: import mylib; int main(){ mylib::mp_size_t<0> var; } Compiler error: [redacted]\main.cpp(4): error C2059: syntax error: '<'
Build performance gains are higher when lots of translation units consume the same library. Building the Boost.Mp11 test suite (which has around 200 translation units) yields the following results: Headers: 2min 10s. Modules: 39s (this includes the time required to build the std and Boost modules). The numbers are even better than I thought! Thanks, Chuanqi
From:Ruben Perez <rubenperez038@gmail.com> Send Time:2025 Jan. 13 (Mon.) 04:51 To:boost<boost@lists.boost.org> CC:Daniela Engert<dani@ngrt.de>; Chuanqi<chuanqi.xcq@alibaba-inc.com>; "Stephan T. Lavavej"<stl@exchange.microsoft.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially Hi all, I finally have something workable regarding Boost and C++20 modules. I've described it in detail in this article [1], but here's a rough summary: I've modularized Boost.Mp11 [2] (as an example of a header-only library) and Boost.Charconv [3] (as an example of a compiled library). I know that Charconv is not very useful in C++20, but it has few dependencies and is relatively simple. I've also changed all tests in these libraries to consume them as C++20 modules in modular builds, and added CI jobs. In the process, I've found a couple of what I think are MSVC bugs: * An excess in pruning template specializations declared in the GMF, even when they are (I think) decl-reachable from the purview [4]. * A problem when an exported using declaration targets another using declaration declared in the GMF [5]. (I've reported the bugs this past weekend, so they might be pending Microsoft's approval when you read this. I've pasted the bug report descriptions at the bottom of the email, just in case). Could any module expert confirm that these are indeed bugs? Other than that, I'd like to get some feedback on this, and whether you think this proposal is worth the effort. Many thanks, Ruben. [1] https://anarthal.github.io/cppblog/modules3 <https://anarthal.github.io/cppblog/modules3 > [2] https://github.com/boostorg/mp11/pull/104 <https://github.com/boostorg/mp11/pull/104 > [3] https://github.com/boostorg/charconv/pull/255 <https://github.com/boostorg/charconv/pull/255 > [4] https://developercommunity.visualstudio.com/t/C20-modules:-specialzations-in... <https://developercommunity.visualstudio.com/t/C20-modules:-specialzations-in... > Declarations in the global module fragment should not be discarded if they are decl-reachable from an exported entity within the module (http://eel.is/c++draft/module.global.frag <http://eel.is/c++draft/module.global.frag >). MSVC doesn't seem to apply this rule to template specializations. Minimal reproduction follows. Project layout: /mylib/include/third_party.hpp /mylib/include/mylib.hpp /mylib/modules/mylib.cppm /main.cpp Contents of /mylib/include/third_party.hpp: namespace third_party { template <class T> struct is_error_code_enum { inline static constexpr bool value = false; }; } Contents of /mylib/include/mylib.hpp: #include <third_party.hpp> namespace mylib { enum class myenum { v = 1 }; } template<> struct third_party::is_error_code_enum<mylib::myenum> { inline static constexpr bool value = true; }; Contents of /mylib/modules/mylib.cppm: module; #include <mylib.hpp> export module mylib; export namespace mylib { using mylib::myenum; void f(third_party::is_error_code_enum<mylib::myenum>) {} // specialization is decl-rechable } Contents of /main.cpp: #include <mylib/include/third_party.hpp> import mylib; int main(){ static_assert(third_party::is_error_code_enum<mylib::myenum::value>); // fails } [5] https://developercommunity.visualstudio.com/t/MSVC-C20-modules:-export-using... <https://developercommunity.visualstudio.com/t/MSVC-C20-modules:-export-using... > When a template alias is defined in the global module fragment, then exported with the export using technique, and finally imported in a source file, the compiler fails to parse the importing source file. Project setup: /mylib/include/mylib.hpp /mylib/modules/mylib.cppm /main.cpp Contents of /mylib/include/mylib.hpp: namespace mylib { template<typename T, T N> struct some_constant {}; template int N using mp_size_t = some_constant<int, N>; } Contents of /mylib/modules/mylib.cppm module; #include <mylib.hpp> export module mylib; export namespace mylib { using mylib::mp_size_t; } Contents of /main.cpp: import mylib; int main(){ mylib::mp_size_t<0> var; } Compiler error: [redacted]\main.cpp(4): error C2059: syntax error: '<'
On 12/01/2025 20:51, Ruben Perez via Boost wrote:
Hi all,
I finally have something workable regarding Boost and C++20 modules. I've described it in detail in this article [1], but here's a rough summary:
Wow, thank you for this, that's a brilliant first step and proof of principle. I have a few comments, many of which are about things over which we/you have no control! 1) Are the CMake files usable with older CMake versions? 2) The lack of Visual Studio generators for CMake is annoying: by far the best user experience is likely to come from Visual Studio integration as that handles module dependencies very well. 3) The need for a separate object file even for header only libraries is annoying, it leads to all kind of issues: 3a) We're going to end up generating an awful lot of module import initializer libraries. 3b) Using Boost with modules may be untenable, without using CMake to handle the dependencies: given that Boost.Whatever may have no end of dependencies all of which will need to be manually specified on the command like otherwise. For a large project with multiple Boost dependencies things explode even further. We're between a rock and hard place here, as I don't see a single monolithic Boost module as a good solution either. But as things stand, building a trivial hello-world using-boost-as-module app from the command line could well be a no-no. 3c) I wonder what impact a dependency on lots of small modules will have on build times: both pre-compile module dependency-scanning phase, and linking to all those micro-libraries. Best, John.
On Mon, 13 Jan 2025, 17:05 John Maddock via Boost, <boost@lists.boost.org> wrote:
On 12/01/2025 20:51, Ruben Perez via Boost wrote:
Hi all,
I finally have something workable regarding Boost and C++20 modules. I've described it in detail in this article [1], but here's a rough summary:
Wow, thank you for this, that's a brilliant first step and proof of principle.
Thanks for your comments.
I have a few comments, many of which are about things over which we/you have no control!
1) Are the CMake files usable with older CMake versions?
As far as I know, C++20 module support starts with CMake 3.28. And support for import std (which the proposal uses) is still experimental. So I'd say the answer to this question would be no, unfortunately.
2) The lack of Visual Studio generators for CMake is annoying: by far the best user experience is likely to come from Visual Studio integration as that handles module dependencies very well.
Yes, definitely. Windows CIs were more involved to set up because of this.
3) The need for a separate object file even for header only libraries is annoying, it leads to all kind of issues:
3a) We're going to end up generating an awful lot of module import initializer libraries.
3b) Using Boost with modules may be untenable, without using CMake to handle the dependencies: given that Boost.Whatever may have no end of dependencies all of which will need to be manually specified on the command like otherwise. For a large project with multiple Boost dependencies things explode even further. We're between a rock and hard place here, as I don't see a single monolithic Boost module as a good solution either. But as things stand, building a trivial hello-world using-boost-as-module app from the command line could well be a no-no.
Yes, that's the case, unfortunately. Or, at least, without a build system that understands the dependencies. A positive point is that running all of our test suites in module mode should make us relatively confident in what we ship. This can then be consumed from maybe build2 or other build system.
3c) I wonder what impact a dependency on lots of small modules will have on build times: both pre-compile module dependency-scanning phase, and linking to all those micro-libraries.
This is definitely something to look at, but I can't do it without modularizing higher order libraries. If this becomes a problem, we could think of creating a big "boost.essentials" module that contains things like Core, ThrowException, Mp11, Variant2 and System. I think this has been proposed before in the context of headers. I'd defer this until we can measure this and the impact is significant. For now, I've set all initializer-only libraries to be STATIC, even when BUILD_SHARED_LIBS is ON.
Best, John.
Thanks Ruben.
Hi all, As you may know, I wrote a prototype to enable consuming Boost as C++20 modules (i.e. import boost.xyz) some weeks ago. After some rounds of helpful feedback from maintainers (specially from Peter Dimov), we've decided to temporarily halt the work until the module ecosystem becomes more stable. I plan to revisit this prototype once CMake's support for "import std" becomes stable, and MSVC fixes the bugs I reported. I've summarized all my findings and design decisions in this (updated) article: https://anarthal.github.io/cppblog/modules3 Many thanks, Ruben.
[Ruben Perez]
As you may know, I wrote a prototype to enable consuming Boost as C++20 modules (i.e. import boost.xyz) some weeks ago. After some rounds of helpful feedback from maintainers (specially from Peter Dimov), we've decided to temporarily halt the work until the module ecosystem becomes more stable. I plan to revisit this prototype once CMake's support for "import std" becomes stable, and MSVC fixes the bugs I reported.
This is awesome - thank you for trying out modules and reporting compiler bugs! This is exactly what's necessary to help modules become usable by larger and larger audiences. I've pinged our modules compiler dev about the second bug that's still active (the first is marked "fixed pending release"). STL
On Wed, 29 Jan 2025 at 21:40, Stephan T. Lavavej <stl@exchange.microsoft.com> wrote:
[Ruben Perez]
As you may know, I wrote a prototype to enable consuming Boost as C++20 modules (i.e. import boost.xyz) some weeks ago. After some rounds of helpful feedback from maintainers (specially from Peter Dimov), we've decided to temporarily halt the work until the module ecosystem becomes more stable. I plan to revisit this prototype once CMake's support for "import std" becomes stable, and MSVC fixes the bugs I reported.
This is awesome - thank you for trying out modules and reporting compiler bugs! This is exactly what's necessary to help modules become usable by larger and larger audiences.
I've pinged our modules compiler dev about the second bug that's still active (the first is marked "fixed pending release").
Great. Thanks a lot. Ruben.
STL
Ruben Perez wrote:
In particular, CMake under Windows doesn’t support the usual Visual Studio generators when building modules.
https://cmake.org/cmake/help/latest/manual/cmake-cxxmodules.7.html says that Visual Studio 17 2022 is supported.
On Mon, 13 Jan 2025, 19:20 Peter Dimov via Boost, <boost@lists.boost.org> wrote:
Ruben Perez wrote:
In particular, CMake under Windows doesn’t support the usual Visual Studio generators when building modules.
https://cmake.org/cmake/help/latest/manual/cmake-cxxmodules.7.html says that Visual Studio 17 2022 is supported.
It also says it doesn't support installing modules and building EXPORTED targets involving modules, including the std module, so I don't think it would be usable with the prototype I wrote, at least as it is now. I thought modules didn't work at all with the Visual Studio generators, so thanks for bringing that up.
Ruben Perez wrote:
On Mon, 13 Jan 2025, 19:20 Peter Dimov via Boost, <boost@lists.boost.org <mailto:boost@lists.boost.org> > wrote:
Ruben Perez wrote:
In particular, CMake under Windows doesn’t support the usual Visual Studio generators when building modules.
https://cmake.org/cmake/help/latest/manual/cmake- cxxmodules.7.html says that Visual Studio 17 2022 is supported.
It also says it doesn't support installing modules
What generator supports installing modules?
and building EXPORTED targets involving modules, including the std module,
IMPORTED. I'm not actually sure what this means.
On Mon, 13 Jan 2025 at 20:14, Peter Dimov <pdimov@gmail.com> wrote:
Ruben Perez wrote:
On Mon, 13 Jan 2025, 19:20 Peter Dimov via Boost, <boost@lists.boost.org <mailto:boost@lists.boost.org> > wrote:
Ruben Perez wrote: > [1] https://anarthal.github.io/cppblog/modules3
> In particular, CMake under Windows doesn’t support the usual Visual Studio generators when building modules.
https://cmake.org/cmake/help/latest/manual/cmake- cxxmodules.7.html says that Visual Studio 17 2022 is supported.
It also says it doesn't support installing modules
What generator supports installing modules?
I thought this meant that installing module interfaces (.cppm) was not supported. It is indeed supported. But see my next comments.
and building EXPORTED targets involving modules, including the std module,
IMPORTED. I'm not actually sure what this means.
I completely meant IMPORTED. This means that the find_package workflow doesn't work with C++20 modules and Visual Studio generators. For example: find_package(mylib REQUIRED) # contains a static library in the form of a C++20 module add_executable(main main.cpp) target_link_libraries(main PUBLIC mylib) Fails with: [cmake] CMake Error in CMakeLists.txt: [cmake] Target "mylib@synth_701dd1d1595e" contains C++ modules intended for [cmake] BMI-only compilation. This is not yet supported by the Visual Studio [cmake] generator. If I read the CMake docs correctly, mylib here is an IMPORTED target that contains a PUBLIC CXX_MODULES FILE_SET. When encountering these, CMake attempts to build a suitable binary module interface, which is required to consume the module. This step is the one that isn't supported by Visual Studio generators. Interestingly, import std seems to be built on this same concept, and thus doesn't work with Visual Studio generators: [cmake] CMake Error in libs/mp11/test/CMakeLists.txt: [cmake] The "CXX_MODULE_STD" property on the target "boost_mp11-mp_bind_front" [cmake] requires that the "__CMAKE::CXX23" target exist, but it was not provided by [cmake] the toolchain. Reason: [cmake] [cmake] Unsupported generator: Visual Studio 17 2022 Thanks, Ruben.
Since everything that's textually included becomes attached to the named module, it's too easy to create ODR violations. This may be some misunderstandings. It is fine if the inclusion was guarded in `extern “C++"` Thanks, Chuanqi
On Sun, 22 Dec 2024 at 13:10, Peter Dimov via Boost <boost@lists.boost.org> wrote:
Ruben Perez wrote:
3. Some libraries support configuration macros. For instance, Boost::throw_exception supports BOOST_NO_EXCEPTIONS. This is used
by
other Boost libraries to support builds that disable exceptions. In the current world, the end user defines BOOST_NO_EXCEPTIONS in their code, includes the library, and everything works. Using a scheme like the one I suggested requires the user to build all the Boost modules using BOOST_NO_EXCEPTIONS, which is more involved than what we have today.
BOOST_NO_EXCEPTIONS is not defined by the user. It's defined by Boost.Config when -fno-exceptions or an equivalent is used.
Thanks for the clarification.
I think the same considerations still apply: if the user wants -fno-exceptions, they would need to build the Boost modules with that flag if we follow what the CMake guys recommended. I was going to say that the same applies to the -std level, but it occurs to me
From:Peter Dimov via Boost <boost@lists.boost.org> Send Time:2024 Dec. 22 (Sun.) 20:39 To:Ruben Perez<rubenperez038@gmail.com>; boost<boost@lists.boost.org> Cc:Peter Dimov<pdimov@gmail.com> Subject:Re: [boost] Interest for C++20 modules support of boost officially Ruben Perez wrote: that in order to build and use modules, you need at least -std=c++20 anyway so most such issues won't manifest for now, until we get conditional uses of C++23 or C++26. We also have an interesting problem with -fno-exceptions; the function boost::throw_exception in that case is only declared on our side, but defined by the user. This (I think) means that the declaration can't be exported by the module, it needs to appear in the GMF. Maybe. At the beginning I'd probably not bother with modularizing Assert (which mostly exports macros anyway) or ThrowException. By the way, I posted about the optional `export` idea to the committee mailing list, and Michael Spencer replied and basically said "don't". "My suggestion is to do either `export import <boost/foo.hpp>;` in the module if you are ok with getting extra declarations, or `export using ...` like libc++ does. Trying to textually include a header into the purview of a named module is fraught with issues." Since everything that's textually included becomes attached to the named module, it's too easy to create ODR violations. But, apparently, Clang doesn't like header units, so the above is also not quite optimal. _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost <http://lists.boost.org/mailman/listinfo.cgi/boost >
Ruben Perez wrote:
install(TARGETS ${LIB} EXPORT ${LIB}-targets RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}" LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}" ARCHIVE DESTINATION "${CMAKE_INSTALL_LIBDIR}" PRIVATE_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" PUBLIC_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" FILE_SET CXX_MODULES DESTINATION . )
`DESTINATION .`? Where does this put the module files? In /usr?
On Sun, 22 Dec 2024 at 17:02, Peter Dimov via Boost <boost@lists.boost.org> wrote:
Ruben Perez wrote:
install(TARGETS ${LIB} EXPORT ${LIB}-targets RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}" LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}" ARCHIVE DESTINATION "${CMAKE_INSTALL_LIBDIR}" PRIVATE_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" PUBLIC_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" FILE_SET CXX_MODULES DESTINATION . )
`DESTINATION .`? Where does this put the module files? In /usr?
No. It places them under ${CMAKE_INSTALL_PREFIX}/./modules/${LIB}.cxx.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Ruben Perez wrote:
On Sun, 22 Dec 2024 at 17:02, Peter Dimov via Boost <boost@lists.boost.org> wrote:
Ruben Perez wrote:
install(TARGETS ${LIB} EXPORT ${LIB}-targets RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}" LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}" ARCHIVE DESTINATION "${CMAKE_INSTALL_LIBDIR}" PRIVATE_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" PUBLIC_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" FILE_SET CXX_MODULES DESTINATION . )
`DESTINATION .`? Where does this put the module files? In /usr?
No. It places them under ${CMAKE_INSTALL_PREFIX}/./modules/${LIB}.cxx.
If we assume the default prefix of /usr, this means /usr/modules/core.cxx, which doesn't seem quite right on a number of levels. My instinct was to call the source file boost_core.cppm, and it seems like it was correct. The current convention for modules seems to be either CMAKE_INSTALL_LIBDIR or CMAKE_INSTALL_DATADIR.
On Sun, 22 Dec 2024 at 17:27, Peter Dimov <pdimov@gmail.com> wrote:
Ruben Perez wrote:
On Sun, 22 Dec 2024 at 17:02, Peter Dimov via Boost <boost@lists.boost.org> wrote:
Ruben Perez wrote:
install(TARGETS ${LIB} EXPORT ${LIB}-targets RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}" LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}" ARCHIVE DESTINATION "${CMAKE_INSTALL_LIBDIR}" PRIVATE_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" PUBLIC_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" FILE_SET CXX_MODULES DESTINATION . )
`DESTINATION .`? Where does this put the module files? In /usr?
No. It places them under ${CMAKE_INSTALL_PREFIX}/./modules/${LIB}.cxx.
If we assume the default prefix of /usr, this means /usr/modules/core.cxx, which doesn't seem quite right on a number of levels.
My instinct was to call the source file boost_core.cppm, and it seems like it was correct.
The current convention for modules seems to be either CMAKE_INSTALL_LIBDIR or CMAKE_INSTALL_DATADIR.
True. Will change it.
Following this approach, dependencies are automatic. But this has a number of downsides: 1. At the moment, BMIs are built with the settings used to build Boost in the first place, which causes a lot of trouble. 2. The installation generates many more binaries than in the headers world. It is uncertain whether these binaries will be compatible if build flags change. 3. Some libraries support configuration macros. For instance, Boost::throw_exception supports BOOST_NO_EXCEPTIONS. This is used by other Boost libraries to support builds that disable exceptions. In
Hi Ruben, thanks for your detailed explanation and now I get your points and concerns. the current world, the end user defines BOOST_NO_EXCEPTIONS in their code, includes the library, and everything works. Using a scheme like the one I suggested requires the user to build all the Boost modules using BOOST_NO_EXCEPTIONS, which is more involved than what we have today. Understood your concerns. For 1, it is (or meant to be the responsibility) of build systems. They should take care of it. But as you mentioned, here is not ETA for it. For 2, IIRC, it won’t be a problem since all the installed binaries really contains an initializers for modules only. And if not, I guess it would be problematic already. For 3, I am wondering if this is similar with 1. And also, I think your suggestion to ask the customers to build and consume the object files for consumers make since too. Although there was opinions that this strategy may produce ODR violations, it should be fine for a lot of existing libraries that provide module interfaces as wrappers of headers. The above theory for ODR violations is, every *.cppm file should map to a single and unique object file. So that we won’t have multiple definitions for the same declarations in a program. But for interfaces of modules of header wrappers, the object file should contain a single initializer. So it may be fine for the specific case. So my conclusion is, we may have following alternatives: (1) Just install the binraries. And wait for CMake to handle the problem of different settings. Note that all the libraries including the standard library face the same issue now. (2) Ask the consumers to build and consume the object files. Achieve this by asking and waiting CMake to implement a new feature for this. (3) Ask the consumers to build and consume the object files.. Achieve this by writing some cmake scripts? Sorry I am not a Cmake expert so I can’t give a concrete suggestion. I am just wondering if it is possible to make it by some cmake scripts. (4) Do not install modules right now but provide a way for users to consume modules with local build. The libc++ has similar period: https://libcxx.llvm.org/Modules.html#using-the-local-build <https://libcxx.llvm.org/Modules.html#using-the-local-build > . (there was almost 1 year between libc++ support modules and cmake support import installed libc++ modules, if I remember correctly.) This sounds not ideal. And as pointed by other thread, a lot of users care about installed uses. But if our time are limited, and we don’t write cmake scripts again and again, maybe it is fine to refactor the source only and providing a way to use modules for local builds. e.g., we’re using bazel and it is good enough for users like us (and probably of build2 and cmake users who like fetch_contents). It is not ideal for sure, but it works for some user. And we can add the support of installataion when every thing has a solution. Thanks, Chuanqi ------------------------------------------------------------------ From:Ruben Perez <rubenperez038@gmail.com> Send Time:2024 Dec. 22 (Sun.) 17:39 To:Chuanqi<chuanqi.xcq@alibaba-inc.com> Cc:Ruben Perez via Boost<boost@lists.boost.org> Subject:Re: [boost] Interest for C++20 modules support of boost officially On Fri, 20 Dec 2024 at 03:15, Chuanqi Xu <chuanqi.xcq@alibaba-inc.com> wrote:
Hi Ruben,
I think we’re in the same page except a few wording issues. e.g, I won’t call it as pre-compiled headers. But this doesn’t matter.
2. Boost.Url in the module world would install: * libboost_url.a, containing the same function definitions as above. * A boost_url.cppm file, akin to what headers are today. CMake would build a BMI from this file when a user needs to import boost.url. In practice, this cppm file will likely be implemented in terms of the header files that we have today.
The libboost_url.a with modules support may have the same entities with original libboost_url.a **plus** the module initializer.
And also, as discussed in other mail, the headers will be installed too.
Yes, completely. I'm building a small prototype to show how things would work, as discussing with concrete code is much more fruitful.
Unfortunately, CMake doesn't seem to have first-class support for this, meaning that the user would need to figure out dependencies manually.
What do you mean by figuring out dependencies manually? Do you mean, in cmake, the dependencies will be calculated if the libraries are linked? If yes, I feel it might be fine. Or did I misunderstand it?
Consider Boost.variant2. It's a header-only library that depends on Boost.assert, Boost.config and Boost.mp11, which are also header-only. This is what the CMake code to build these as modules could look like (simplified - I will post the entire prototype when ready): add_library(boost_variant2) target_sources(boost_variant2 PUBLIC FILE_SET CXX_MODULES FILES modules/variant2.cxx) target_include_directories(boost_variant2 PUBLIC include) target_link_libraries(boost_variant2 PUBLIC Boost::assert Boost::config Boost::mp11 ) Here, Boost::assert and Boost::mp11 have been created with similar code as the one shown above. Boost::config is an interface library with only headers (as it only exports macros). The targets are installed by the Boost.CMake infrastructure, with a call akin to: install(TARGETS ${LIB} EXPORT ${LIB}-targets RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}" LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}" ARCHIVE DESTINATION "${CMAKE_INSTALL_LIBDIR}" PRIVATE_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" PUBLIC_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" FILE_SET CXX_MODULES DESTINATION . ) When building Boost with CMake, for each module (Boost::assert, Boost::mp11 and Boost::variant2), a BMI and a static library with the module initializer is built. When installing Boost (with CMake), for each module (Boost::assert, Boost::mp11 and Boost::variant2), the headers, module code and static library is installed. When consuming Boost from another project, e.g. using find_package(Boost COMPONENTS Boost::variant2), the headers, module code and static library for Boost::variant2 and its dependencies are found and used. The BMIs are built in the consumer project. Right now, the flags used originally to build Boost are used to re-build the BMIs. This causes trouble, but is intended to change in the future, with an unknown time frame [1]. Following this approach, dependencies are automatic. But this has a number of downsides: 1. At the moment, BMIs are built with the settings used to build Boost in the first place, which causes a lot of trouble. 2. The installation generates many more binaries than in the headers world. It is uncertain whether these binaries will be compatible if build flags change. 3. Some libraries support configuration macros. For instance, Boost::throw_exception supports BOOST_NO_EXCEPTIONS. This is used by other Boost libraries to support builds that disable exceptions. In the current world, the end user defines BOOST_NO_EXCEPTIONS in their code, includes the library, and everything works. Using a scheme like the one I suggested requires the user to build all the Boost modules using BOOST_NO_EXCEPTIONS, which is more involved than what we have today. For this reason, it makes sense to consider whether there is a way to completely avoid the generation and installation of these binaries for header-only libraries, and follow a model similar to standard modules. In this ideal model, installing Boost with CMake would only install the headers and module code for each header-only library, and no binary artifacts. The consuming project would build both the BMIs and the objects containing the module initializers with the adequate build flags, as many times as required by the different targets. Ideally, the following syntax would work and do what I'm suggesting: # This does NOT work as of today add_library(boost_variant2 INTERFACE) # Don't build the library in the current project target_sources(boost_variant2 INTERFACE FILE_SET CXX_MODULES FILES modules/variant2.cxx) target_include_directories(boost_variant2 INTERFACE include) target_link_libraries(boost_variant2 INTERFACE Boost::assert Boost::config Boost::mp11 ) In the ideal world, this wouldn't build any binary library or BMI per se. When linking to the boost_variant2 target, a BMI and a static library would be built with the flags required by the consuming target. Note that this does NOT work as of today [2]. I've raised a feature request to the CMake team with this suggestion [3], but I don't think this is happening in the short-term. Given that this does not work, the other option would be installing the headers and the module code manually. For instance: # This is what we have today. Note that we have removed the target_sources add_library(boost_variant2 INTERFACE) target_include_directories(boost_variant2 INTERFACE include) target_link_libraries(boost_variant2 INTERFACE Boost::assert Boost::config Boost::mp11 ) # Install the module code separately install(DIRECTORY modules DESTINATION .) With this setup, it is the user's responsibility to issue the target_sources calls required to build the boost_variant2 module. There is also nothing telling CMake that the boost_variant2 module requires building boost_assert and boost_mp11, though. Thus, it becomes the user's responsibility to know this dependency chain and add the relevant target_sources calls. This is what I mean by "figuring out dependencies manually". While it may not seem that hard in this case, things get much worse as you study libraries with more dependencies. I'm the author of Boost.MySQL, and I wouldn't be able to spell you the current dependency tree without an automated tool to help. I hope this helps clarify what I meant. If I have missed anything obvious, or anyone has ideas on how to make this better, please let me know.
Thanks, Chuanqi
Regards, Ruben. [1] https://discourse.cmake.org/t/advice-on-c-20-modules-boost/10641/13 <https://discourse.cmake.org/t/advice-on-c-20-modules-boost/10641/13 > [2] https://cmake.org/cmake/help/latest/command/target_sources.html#file-sets <https://cmake.org/cmake/help/latest/command/target_sources.html#file-sets > [3] https://discourse.cmake.org/t/distributing-c-20-modules-as-source/13246 <https://discourse.cmake.org/t/distributing-c-20-modules-as-source/13246 >
participants (15)
-
Alexander Grund -
Bo Persson -
Cameron Angus -
Chuanqi Xu -
Daniela Engert -
Dominique Devienne -
Ivan Matek -
Janko Dedic -
John Maddock -
Matt Borland -
Peter Dimov -
René Ferdinand Rivera Morell -
Ruben Perez -
Stephan T. Lavavej -
Steve Downey