A proposal for a modern build-system
Hi. Preface: C++20 modules and header-units are a great addition to C++. There are numerous benefits of these. One of them is a faster compilation. Conventional build-systems however are unable to benefit from C++20 modules or header-units. Boost current build-system b2 does not support C++20 modules or header-units. CMake does not support C++20 header-units https://gitlab.kitware.com/cmake/cmake/-/issues/25293. I developed the build-system HMake https://github.com/HassanSajjad-302/HMake. It is in C++ and MIT Licenced. It supports drop-in header-files to header-units replacement. While compilers support processing a header-file as header-unit, no other build-system supports this feature. With this I compiled SFML with C++20 header-units https://www.reddit.com/r/cpp/comments/1555g6b/hmake_build_system_02_compilin.... In the post, I mentioned that 2-3.5x speed-up was achieved but I did some more testing recently and an even better speed-up of 2.7-4.7x was achieved i.e., the speed of header-units is further improving with recent iterations. Proposal: I want to propose my build-system HMake for boost. Besides its already state-of-the-art C++20 modules and header-units support, it will be the first to support https://lists.isocpp.org/sg15/2023/11/2106.php or https://lists.isocpp.org/sg15/2023/11/2146.php if any of this gets implemented. These papers present ideas for faster module adoption and avoiding redundant module compilations. These need support from build-system. This way boost could be a breeding ground for C++20 modules adoption. Also, I have good experience with boost's current build-system b2 as a good portion of HMake's current API is inspired by it. Timeframe: I will complete this proposal in 3 months. Deliverables: 1) 2-months: Boost basic configuration compiled with C++20 header-units. This is more of a lone sprint. 2) 3-months: All boost current configurations + new with header-units and modules + tests and examples. This will require more active involvement by the current maintainers. I will give weekly updates here and will make a video after 2 months presenting the progress made. This will be to help other contributors so they can ensure that their library is fully supported. Cost: Standard C++ Developer Compensation for 3 months (Negotiable). Payments are to be issued monthly. But you can back out at any point in case you are unsatisfied with development speed or direction. Impact: While I don't see any reason for it to be a flop, it is an engineering endeavor so there is some inherent risk. The best-case scenario is that it improves Boost usability and 2.5x compilation speed improvement starts a domino effect for HMake and C++20 header-units and modules adoption. Best, Hassan Sajjad
1) 2-months: Boost basic configuration compiled with C++20 header-units. This is more of a lone sprint.
Can you please elaborate on what this means? Are you intending to replace the build mechanism we ship to users (as in when they build Boost by typing ./b2 --with-X)? If that's the case, does your system support pre C++20 builds, which we support? Or Boost internal test suite? 2) 3-months: All boost current configurations + new with header-units and
modules + tests and examples.
Bear in mind that we have many libraries and each one has really extensive suits. This is not trivial at all. Thanks, Ruben.
Yes. I intend to first coexist with b2 for a brief period and then replace it. Yes, it supports pre C++20 builds and it will support Boost internal test suite. This is not trivial at all.
I am highly confident. Please see that what this achieves is non-trivial
as-well.
https://github.com/HassanSajjad-302/SFML/blob/43a9a2af28f6ec9656f0294fa62c1e...
.
In header-unit configuration, t compiles 232 header-units ifc and 1308
module-files(consumer of these ifc). Eight DLLs are compiled
(automatically copied to dependent executables). 17 different targets are
compiled. A target is a group of files with same compile-command. targets
can have other targets as dependencies which impacts the dependent target's
compile and link command. These targets are compiled in 3 different
configurations. So, a total of 51 targets is compiled. And 24 library
targets are compiled (8 shared libs, 16 static libs), others are
header-only libs. More configurations could be defined very easily as-well.
And that's all in 200 lines.
I have experience with b2 which would be helpful as well.
On Fri, Mar 22, 2024 at 12:36 AM Ruben Perez
1) 2-months: Boost basic configuration compiled with C++20 header-units. This is more of a lone sprint.
Can you please elaborate on what this means? Are you intending to replace the build mechanism we ship to users (as in when they build Boost by typing ./b2 --with-X)? If that's the case, does your system support pre C++20 builds, which we support? Or Boost internal test suite?
2) 3-months: All boost current configurations + new with header-units and
modules + tests and examples.
Bear in mind that we have many libraries and each one has really extensive suits. This is not trivial at all.
Thanks, Ruben.
On 3/21/24 20:34, Hassan Sajjad via Boost wrote:
Hi.
Preface:
C++20 modules and header-units are a great addition to C++. There are numerous benefits of these. One of them is a faster compilation. Conventional build-systems however are unable to benefit from C++20 modules or header-units. Boost current build-system b2 does not support C++20 modules or header-units. CMake does not support C++20 header-units https://gitlab.kitware.com/cmake/cmake/-/issues/25293.
I developed the build-system HMake https://github.com/HassanSajjad-302/HMake. It is in C++ and MIT Licenced. It supports drop-in header-files to header-units replacement. While compilers support processing a header-file as header-unit, no other build-system supports this feature.
With this I compiled SFML with C++20 header-units https://www.reddit.com/r/cpp/comments/1555g6b/hmake_build_system_02_compilin.... In the post, I mentioned that 2-3.5x speed-up was achieved but I did some more testing recently and an even better speed-up of 2.7-4.7x was achieved i.e., the speed of header-units is further improving with recent iterations.
Proposal:
I want to propose my build-system HMake for boost.
I'm strongly opposed to supporting a third build system. Especially given that the proposed build system is not mature and hasn't gained wide adoption. In fact, from the GitHub page it looks like a prototype rather than something that was battle tested. I see no documentation besides the readme with a few examples. Also, personally, I don't think C++ is a good choice for a build system language. The build system files are unnecessarily verbose, and the build process seems too complex.
Besides its already state-of-the-art C++20 modules and header-units support, it will be the first to support https://lists.isocpp.org/sg15/2023/11/2106.php or https://lists.isocpp.org/sg15/2023/11/2146.php if any of this gets implemented. These papers present ideas for faster module adoption and avoiding redundant module compilations. These need support from build-system. This way boost could be a breeding ground for C++20 modules adoption. Also, I have good experience with boost's current build-system b2 as a good portion of HMake's current API is inspired by it.
Timeframe:
I will complete this proposal in 3 months.
This is a very ambitious time frame, IMO.
Deliverables:
1) 2-months: Boost basic configuration compiled with C++20 header-units. This is more of a lone sprint. 2) 3-months: All boost current configurations + new with header-units and modules + tests and examples. This will require more active involvement by the current maintainers. I will give weekly updates here and will make a video after 2 months presenting the progress made. This will be to help other contributors so they can ensure that their library is fully supported.
Cost:
Standard C++ Developer Compensation for 3 months (Negotiable). Payments are to be issued monthly. But you can back out at any point in case you are unsatisfied with development speed or direction.
I'm not the one to make funding decisions, but I don't see why Boost would pay for this work. It's not like Boost is in dire need of a new build system. If we do want to sponsor our build system advance, I think the money would be better spent on improving CMake support in Boost and in implementing features we need in CMake.
Also, personally, I don't think C++ is a good choice for a build system language. The build system files are unnecessarily verbose, and the build process seems too complex.
This is a subjective opinion. I appreciate it but would appreciate it more if you could provide actionable feedback. I believe mine is the easiest-to-use build system out there, especially for C++ development. I'm strongly opposed to supporting a third build system. Especially
given that the proposed build system is not mature and hasn't gained wide adoption. In fact, from the GitHub page it looks like a prototype rather than something that was battle tested. I see no documentation besides the readme with a few examples.
It is not battle-tested but ready for one. I feel it has now passed from the training academy. More documentation would be added as more features would be added. This is a very ambitious time frame, IMO. While converting SFML, more time was spent on understanding the current build-description compared to expressing it in C++/HMake. SFML basic configuration was achieved in 200 lines. It does not take much time to write 200 lines. And I have experience with Boost build-system b2. If we do want to sponsor our build system advance, I think the money
would be better spent on improving CMake support in Boost and in implementing features we need in CMake.
I feel that the return on investment that I offer is unmatched. That's due to a fundamentally better approach to my software. Many libraries have first come to boost and then come to C++ standard. You should sponsor me because if this succeeds, others might adopt it as as-well. For even bigger projects like UE5, the advantage of compiling with header-units would be bigger >4x compared to my estimate of >2.5x in clean-build for Boost. On Fri, Mar 22, 2024 at 2:05 AM Andrey Semashev via Boost < boost@lists.boost.org> wrote:
Hi.
Preface:
C++20 modules and header-units are a great addition to C++. There are numerous benefits of these. One of them is a faster compilation. Conventional build-systems however are unable to benefit from C++20 modules or header-units. Boost current build-system b2 does not support C++20 modules or header-units. CMake does not support C++20 header-units https://gitlab.kitware.com/cmake/cmake/-/issues/25293.
I developed the build-system HMake https://github.com/HassanSajjad-302/HMake. It is in C++ and MIT Licenced. It supports drop-in header-files to header-units replacement. While compilers support processing a
On 3/21/24 20:34, Hassan Sajjad via Boost wrote: header-file
as header-unit, no other build-system supports this feature.
With this I compiled SFML with C++20 header-units
https://www.reddit.com/r/cpp/comments/1555g6b/hmake_build_system_02_compilin... .
In the post, I mentioned that 2-3.5x speed-up was achieved but I did some more testing recently and an even better speed-up of 2.7-4.7x was achieved i.e., the speed of header-units is further improving with recent iterations.
Proposal:
I want to propose my build-system HMake for boost.
I'm strongly opposed to supporting a third build system. Especially given that the proposed build system is not mature and hasn't gained wide adoption. In fact, from the GitHub page it looks like a prototype rather than something that was battle tested. I see no documentation besides the readme with a few examples.
Also, personally, I don't think C++ is a good choice for a build system language. The build system files are unnecessarily verbose, and the build process seems too complex.
Besides its already state-of-the-art C++20 modules and header-units support, it will be the first to support https://lists.isocpp.org/sg15/2023/11/2106.php or https://lists.isocpp.org/sg15/2023/11/2146.php if any of this gets implemented. These papers present ideas for faster module adoption and avoiding redundant module compilations. These need support from build-system. This way boost could be a breeding ground for C++20 modules adoption. Also, I have good experience with boost's current build-system b2 as a good portion of HMake's current API is inspired by it.
Timeframe:
I will complete this proposal in 3 months.
This is a very ambitious time frame, IMO.
Deliverables:
1) 2-months: Boost basic configuration compiled with C++20 header-units. This is more of a lone sprint. 2) 3-months: All boost current configurations + new with header-units and modules + tests and examples. This will require more active involvement by the current maintainers. I will give weekly updates here and will make a video after 2 months presenting the progress made. This will be to help other contributors so they can ensure that their library is fully supported.
Cost:
Standard C++ Developer Compensation for 3 months (Negotiable). Payments are to be issued monthly. But you can back out at any point in case you are unsatisfied with development speed or direction.
I'm not the one to make funding decisions, but I don't see why Boost would pay for this work. It's not like Boost is in dire need of a new build system.
If we do want to sponsor our build system advance, I think the money would be better spent on improving CMake support in Boost and in implementing features we need in CMake.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Please, don't top-post. https://www.boost.org/community/policy.html#quoting On 3/22/24 12:59, Hassan Sajjad wrote:
Also, personally, I don't think C++ is a good choice for a build system language. The build system files are unnecessarily verbose, and the build process seems too complex.
This is a subjective opinion. I appreciate it but would appreciate it more if you could provide actionable feedback.
This is the opinion I got from reading through the readme. For example: <quote> Now, to build the Example you need to run hhelper once and hbuild twice. hhelper will create the cache.json file. cache.json file provides an opportunity to select a different toolset. It has array indices to the arrays of different tools in toolsCache.json. These tools must exist in toolsCache.json for HMake to work. cache.json file also has the commands that hbuild will use to build the configure dll. Running hbuild first time will create the configure dll, linking hconfigure static-lib using whole-archive linker feature. Running hbuild again will load the configure dll and call the func2 with BSMode::BUILD. This will create the app executable in {buildDir}/app. </quote> So one needs to run the build system three times, and possibly edit a json in between to build his project. This is compared to running b2 once, or cmake twice (first to configure, second to build the project), and no editing in between. The editing part is actually a big problem if the building process is to be wrapped in any external tooling, such as package building tools or CI. Then the build system language. Here's an example from the docs, which presumably is the simplest way to build an application: <quote> #include "Configure.hpp" void buildSpecification() { GetCppExeDSC("app").getSourceTarget().SOURCE_FILES("main.cpp"); } MAIN_FUNCTION </quote> Compare this to b2: <quote> exe app : main.cpp ; </quote> or CMake: <quote> add_executable(app main.cpp) </quote> or Meson: <quote> project('simple', 'cpp') executable('app', 'main.cpp') </quote> I don't know how the more advanced stuff like iterating the filesystem or compiling configure-time checks would look like in HMake, but I suspect those will be more verbose and complicated as well. I can see you're explicitly using std::filesystem in some examples, and that reinforces my suspicion.
I'm strongly opposed to supporting a third build system. Especially given that the proposed build system is not mature and hasn't gained wide adoption. In fact, from the GitHub page it looks like a prototype rather than something that was battle tested. I see no documentation besides the readme with a few examples.
It is not battle-tested but ready for one.
It is not, without proper documentation. Another potential issue that comes to mind is, what is the minimum C++ version HMake requires for itself, and what is the minimum C++ version it supports to compile the project? E.g. is it possible to build HMake and the user's project on a system that only supports C++11? Does it support languages other than C++? C and assembler, in particular? I'm asking because Boost is not limited to just C++.
This is a very ambitious time frame, IMO.
While converting SFML, more time was spent on understanding the current build-description compared to expressing it in C++/HMake. SFML basic configuration was achieved in 200 lines. It does not take much time to write 200 lines.
I don't see HMake files in SFML (https://github.com/SFML/SFML), so I presume this was your internal benchmark, and I can't judge how close to the original the translation was. I can believe it was easier *for you* to implement the build scenario in HMake - because you know HMake. It won't be necessarily the case for someone new to HMake, especially without documentation. Also, SFML looks like a much smaller library than Boost.
If we do want to sponsor our build system advance, I think the money would be better spent on improving CMake support in Boost and in implementing features we need in CMake.
I feel that the return on investment that I offer is unmatched. That's due to a fundamentally better approach to my software.
Sorry, I don't see the benefit. I'm not very familiar with modules, but it looks to me that this technology is very far from wide adoption. More than that, it looks like it is very demanding on the developers and, apparently, the ecosystem (case in point - do we now need a whole new build system to leverage modules??) while providing benefits that can be achieved through different means (precompiled headers, ccache, more fine-grained headers, more compiled libraries). More than that, AFAIK, it has fundamental issues with macros. So yeah, modules are a mixed bag, and I don't see support for modules as a worthy goal in the immediate future. If modules are the only benefit HMake promises to provide, that's a weak selling point, IMO. And to reiterate, I'm just not going to support a third build system, whether it is HMake or something else. We're spending enough time on the existing two.
Please, don't top-post.
Sorry, what do you mean by that? I use Reply All button in Gmail. So one needs to run the build system three times, and possibly edit a
json in between to build his project. This is compared to running b2 once, or cmake twice (first to configure, second to build the project), and no editing in between.
No, except the first time, the build-system is very simple to setup and use. You need to run 3 commands to build a project from scratch. It is just one more compared to CMake. Also, build-system is designed such that you need not edit your hmake.cpp again and again. It supports wild cards. I don't know how the more advanced stuff like iterating the filesystem
or compiling configure-time checks would look like in HMake, but I suspect those will be more verbose and complicated as well.
No. Advanced stuff is much easier in hmake. I can see
you're explicitly using std::filesystem in some examples, and that reinforces my suspicion.
std::filesystem is much better than CMake filesystem API or any other build-system's filesystem API. And for anything advanced, much less verbose as-well. I don't see HMake files in SFML (https://github.com/SFML/SFML), so I
presume this was your internal benchmark, and I can't judge how close to the original the translation was.
I did not submit it for production use. I compiled just one configuration. I think it was successful. You can run the Example.exe which depends on 3 DLLs. Anyway, it was just for demonstration and not for production use. I can believe it was easier *for you* to implement the build scenario in
HMake - because you know HMake. It won't be necessarily the case for someone new to HMake, especially without documentation.
I would love to address if you have any specific complaints regarding documentation. Also, SFML looks like a much smaller library than Boost. But it could be considered a middle-sized project. And I don't see a reason for my build-system to not scale. So yeah, modules are a mixed bag, and I don't see support for modules as
a worthy goal in the immediate future. If modules are the only benefit HMake promises to provide, that's a weak selling point, IMO.
I feel that you are in the minority group. Most devs see modules as the future. And with this paper https://lists.isocpp.org/sg15/2023/11/2106.php, their adoption will be much quicker. Modules are not the only benefit of HMake. By the way, what features you will like in a futuristic build-system? HMake might support them. And to reiterate, I'm just not going to support a third build system,
whether it is HMake or something else. We're spending enough time on the existing two.
If successful, then we might deprecate at least one of them. On Fri, Mar 22, 2024 at 5:20 PM Andrey Semashev via Boost < boost@lists.boost.org> wrote:
Please, don't top-post.
https://www.boost.org/community/policy.html#quoting
On 3/22/24 12:59, Hassan Sajjad wrote:
Also, personally, I don't think C++ is a good choice for a build
system
language. The build system files are unnecessarily verbose, and the build process seems too complex.
This is a subjective opinion. I appreciate it but would appreciate it more if you could provide actionable feedback.
This is the opinion I got from reading through the readme. For example:
<quote> Now, to build the Example you need to run hhelper once and hbuild twice. hhelper will create the cache.json file. cache.json file provides an opportunity to select a different toolset. It has array indices to the arrays of different tools in toolsCache.json. These tools must exist in toolsCache.json for HMake to work. cache.json file also has the commands that hbuild will use to build the configure dll. Running hbuild first time will create the configure dll, linking hconfigure static-lib using whole-archive linker feature. Running hbuild again will load the configure dll and call the func2 with BSMode::BUILD. This will create the app executable in {buildDir}/app. </quote>
So one needs to run the build system three times, and possibly edit a json in between to build his project. This is compared to running b2 once, or cmake twice (first to configure, second to build the project), and no editing in between.
The editing part is actually a big problem if the building process is to be wrapped in any external tooling, such as package building tools or CI.
Then the build system language. Here's an example from the docs, which presumably is the simplest way to build an application:
<quote> #include "Configure.hpp"
void buildSpecification() { GetCppExeDSC("app").getSourceTarget().SOURCE_FILES("main.cpp"); }
MAIN_FUNCTION </quote>
Compare this to b2:
<quote> exe app : main.cpp ; </quote>
or CMake:
<quote> add_executable(app main.cpp) </quote>
or Meson:
<quote> project('simple', 'cpp') executable('app', 'main.cpp') </quote>
I don't know how the more advanced stuff like iterating the filesystem or compiling configure-time checks would look like in HMake, but I suspect those will be more verbose and complicated as well. I can see you're explicitly using std::filesystem in some examples, and that reinforces my suspicion.
I'm strongly opposed to supporting a third build system. Especially given that the proposed build system is not mature and hasn't gained wide adoption. In fact, from the GitHub page it looks like a
prototype
rather than something that was battle tested. I see no documentation besides the readme with a few examples.
It is not battle-tested but ready for one.
It is not, without proper documentation.
Another potential issue that comes to mind is, what is the minimum C++ version HMake requires for itself, and what is the minimum C++ version it supports to compile the project? E.g. is it possible to build HMake and the user's project on a system that only supports C++11?
Does it support languages other than C++? C and assembler, in particular? I'm asking because Boost is not limited to just C++.
This is a very ambitious time frame, IMO.
While converting SFML, more time was spent on understanding the current build-description compared to expressing it in C++/HMake. SFML basic configuration was achieved in 200 lines. It does not take much time to write 200 lines.
I don't see HMake files in SFML (https://github.com/SFML/SFML), so I presume this was your internal benchmark, and I can't judge how close to the original the translation was.
I can believe it was easier *for you* to implement the build scenario in HMake - because you know HMake. It won't be necessarily the case for someone new to HMake, especially without documentation.
Also, SFML looks like a much smaller library than Boost.
If we do want to sponsor our build system advance, I think the money would be better spent on improving CMake support in Boost and in implementing features we need in CMake.
I feel that the return on investment that I offer is unmatched. That's due to a fundamentally better approach to my software.
Sorry, I don't see the benefit.
I'm not very familiar with modules, but it looks to me that this technology is very far from wide adoption. More than that, it looks like it is very demanding on the developers and, apparently, the ecosystem (case in point - do we now need a whole new build system to leverage modules??) while providing benefits that can be achieved through different means (precompiled headers, ccache, more fine-grained headers, more compiled libraries). More than that, AFAIK, it has fundamental issues with macros.
So yeah, modules are a mixed bag, and I don't see support for modules as a worthy goal in the immediate future. If modules are the only benefit HMake promises to provide, that's a weak selling point, IMO.
And to reiterate, I'm just not going to support a third build system, whether it is HMake or something else. We're spending enough time on the existing two.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 3/22/24 16:03, Hassan Sajjad wrote:
Please, don't top-post.
Sorry, what do you mean by that? I use Reply All button in Gmail.
I've included a link to our discussion policies that, among other things, explains what top-posting is. I'm including the link again, for your convenience: https://www.boost.org/community/policy.html#quoting
I can see you're explicitly using std::filesystem in some examples, and that reinforces my suspicion.
std::filesystem is much better than CMake filesystem API or any other build-system's filesystem API. And for anything advanced, much less verbose as-well.
It is better for some things and worse for others. Where in b2 or CMake you would glob the filesystem using a wildcard, std::filesystem would require one to write a loop with directory iterators and applying a wildcard (which is not available in the std::filesystem). It would also require one to handle any errors that might arise in the process.
I can believe it was easier *for you* to implement the build scenario in HMake - because you know HMake. It won't be necessarily the case for someone new to HMake, especially without documentation.
I would love to address if you have any specific complaints regarding documentation.
My first complaint would be that it doesn't exist. Have a look at b2 docs or CMake reference for a few examples of what I have in mind.
Also, SFML looks like a much smaller library than Boost.
But it could be considered a middle-sized project. And I don't see a reason for my build-system to not scale.
It's not so much about scaling (although scaling is not a given, too) but about various features and corner cases. I mentioned configure-time checks earlier. Then there are non-trivial dependencies between libraries. There is support for configuration (e.g. which Boost libraries to build, release/debug, single/multi-threaded, which C++ runtime to link, etc.). Then there are tests and documentation, which also use a variety of tools to build.
It is better for some things and worse for others
We can design API on top of std::filesystem in case something is missed. My first complaint would be that it doesn't exist.
Why do you say this? Because there is very detailed documentation. I would appreciate it if you could give one example of the limitation of HMake. I feel all of what you are describing is either supported or can be supported. On Fri, Mar 22, 2024 at 6:22 PM Andrey Semashev via Boost < boost@lists.boost.org> wrote:
On 3/22/24 16:03, Hassan Sajjad wrote:
Please, don't top-post.
Sorry, what do you mean by that? I use Reply All button in Gmail.
I've included a link to our discussion policies that, among other things, explains what top-posting is. I'm including the link again, for your convenience:
https://www.boost.org/community/policy.html#quoting
I can see you're explicitly using std::filesystem in some examples, and that reinforces my suspicion.
std::filesystem is much better than CMake filesystem API or any other build-system's filesystem API. And for anything advanced, much less verbose as-well.
It is better for some things and worse for others. Where in b2 or CMake you would glob the filesystem using a wildcard, std::filesystem would require one to write a loop with directory iterators and applying a wildcard (which is not available in the std::filesystem). It would also require one to handle any errors that might arise in the process.
I can believe it was easier *for you* to implement the build
scenario in
HMake - because you know HMake. It won't be necessarily the case for someone new to HMake, especially without documentation.
I would love to address if you have any specific complaints regarding documentation.
My first complaint would be that it doesn't exist. Have a look at b2 docs or CMake reference for a few examples of what I have in mind.
Also, SFML looks like a much smaller library than Boost.
But it could be considered a middle-sized project. And I don't see a reason for my build-system to not scale.
It's not so much about scaling (although scaling is not a given, too) but about various features and corner cases. I mentioned configure-time checks earlier. Then there are non-trivial dependencies between libraries. There is support for configuration (e.g. which Boost libraries to build, release/debug, single/multi-threaded, which C++ runtime to link, etc.). Then there are tests and documentation, which also use a variety of tools to build.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Andrey Semashev via Boost
Compare this to b2:
<quote> exe app : main.cpp ; </quote>
or CMake:
<quote> add_executable(app main.cpp) </quote>
or Meson:
<quote> project('simple', 'cpp') executable('app', 'main.cpp') </quote>
or build2: <quote> exe{app}: cxx{main} </quote>
I'm not very familiar with modules, but it looks to me that this technology is very far from wide adoption. More than that, it looks like it is very demanding on the developers [...]
I think `import std;`, which I found surprisingly usable[1] in Clang 18 and libc++, could provide a lot of benefit while demanding little form developers.
So yeah, modules are a mixed bag, and I don't see support for modules as a worthy goal in the immediate future. If modules are the only benefit HMake promises to provide, that's a weak selling point, IMO.
What would be a strong selling point of a new build system for Boost, in your opinion (or anyone else's, for that matter)? [1] https://github.com/boris-kolpackov/async_simple/tree/CXX20Modules-build2
On 3/22/24 16:43, Boris Kolpackov wrote:
Andrey Semashev via Boost
writes: I'm not very familiar with modules, but it looks to me that this technology is very far from wide adoption. More than that, it looks like it is very demanding on the developers [...]
I think `import std;`, which I found surprisingly usable[1] in Clang 18 and libc++, could provide a lot of benefit while demanding little form developers.
If you intend your library to require C++20 with modules as a minimum, you might get away with a simple `import std;`. But you're still including headers from other libraries, and your own headers as well (which is what I'm seeing in your linked project's headers), so the benefit from importing std is questionable. The code becomes more messy if you want to support older C++ versions. And let's not forget that in order for the `import std;` to work, including in mixed usage scenario when part of the code includes headers and part uses modules, compiler and standard library developers had to do their part.
So yeah, modules are a mixed bag, and I don't see support for modules as a worthy goal in the immediate future. If modules are the only benefit HMake promises to provide, that's a weak selling point, IMO.
What would be a strong selling point of a new build system for Boost, in your opinion (or anyone else's, for that matter)?
I'm not sure what that might be. Both b2 and CMake get the job done, which is they support configuring, building and testing code. That's pretty much the extent of what I expect from a build system. There are, of course, things other than that, like fetching dependencies or building distro packages or deploying to a server, but in my opinion, these things are out of scope of what I call a build system.
Andrey Semashev via Boost
But you're still including headers from other libraries, and your own headers as well (which is what I'm seeing in your linked project's headers), so the benefit from importing std is questionable.
The linked project is a bit confusing in that it includes both
header-based and modular implementations side-by-side. The modular
one is here:
https://github.com/boris-kolpackov/async_simple/tree/CXX20Modules-build2/asy...
There are very few header includes AFAICS, which are C headers plus
And let's not forget that in order for the `import std;` to work, including in mixed usage scenario when part of the code includes headers and part uses modules, compiler and standard library developers had to do their part.
Yes, Clang already supports this, albeit only in the header first, module second order. So this works: #include <iostream> import std;
I'm not sure what that might be. Both b2 and CMake get the job done, which is they support configuring, building and testing code. That's pretty much the extent of what I expect from a build system.
Well, there is also install/uninstall (with the ability to distinguish between for-install/for-development during update) and preparation of source distributions. But I agree, the ones you have listed are the primary functions of a build system.
There are, of course, things other than that, like fetching dependencies or building distro packages or deploying to a server, but in my opinion, these things are out of scope of what I call a build system.
True, though they are closely related and some of them you won't be able to implement without adequate support in the build system. For example, building distro packages without install support would be pretty painful. Likewise, fetching dependencies would often involve conditions (as in, conditional dependencies) with the input for the conditions (information about target platform, compiler, etc) best supplied by the build system. So I think when looking for a new build system for Boost it would be strange not to ask how it improve the dependency management.
Hassan Sajjad via Boost
It supports drop-in header-files to header-units replacement. While compilers support processing a header-file as header-unit, no other build-system supports this feature.
That's incorrect: build2 had support for header units (with include translation) from 2021: https://build2.org/blog/build2-cxx20-modules-gcc.xhtml (Funny enough that support was announced in Feb 2021 while I see the first commit in the HMake repository was also made in Feb 2021, so build2 had header unit support for as long as HMake existed ;-)). And speaking of compilers, currently only GCC provides enough build system integration (in the form of the module mapper) to allow accurately supporting header units without making the user specify the dependencies manually (or imposing equivalent requirements like pre-compiling all the header units used by the project upfront). For background, see: https://developercommunity.visualstudio.com/t/scanDependencies-does-not-take...
That's incorrect: build2 had support for header units (with include translation) from 2021:
Thank you for informing me. I did not know about it. Can you give bigger
examples? Is it used in production anywhere?
On Fri, Mar 22, 2024 at 11:00 AM Boris Kolpackov
Hassan Sajjad via Boost
writes: It supports drop-in header-files to header-units replacement. While compilers support processing a header-file as header-unit, no other build-system supports this feature.
That's incorrect: build2 had support for header units (with include translation) from 2021:
https://build2.org/blog/build2-cxx20-modules-gcc.xhtml
(Funny enough that support was announced in Feb 2021 while I see the first commit in the HMake repository was also made in Feb 2021, so build2 had header unit support for as long as HMake existed ;-)).
And speaking of compilers, currently only GCC provides enough build system integration (in the form of the module mapper) to allow accurately supporting header units without making the user specify the dependencies manually (or imposing equivalent requirements like pre-compiling all the header units used by the project upfront). For background, see:
https://developercommunity.visualstudio.com/t/scanDependencies-does-not-take...
Hassan Sajjad
That's incorrect: build2 had support for header units (with include translation) from 2021:
Thank you for informing me. I did not know about it. Can you give bigger examples? Is it used in production anywhere?
I will assume your questions refer to build2 in general rather than to its header units support specifically (which is not very usable at the moment due to compiler deficiencies). Yes, you can see more substantial examples on cppget.org (packages) or the build2-packaging GitHub org (package repositories), which, in particular, include both Boost and Qt (and I agree with Andrey, SFML, which is also there, is not a particularly complex library). And yes, build2 is used in production, including in large (many hundreds of libraries), successful commercial projects. And speaking of Boost (not to make this reply completely off-topic), in Hmake's README you say:
HMake 1.0 will only be released when, few of the mega projects like UE5, AOSP, Qt etc. could be supported.
I would suggest that you include Boost on this list. While libraries like Qt give good "depth-wise" reality check (i.e., relatively few libraries with very complex builds), Boost gives a unique "breadth-wise" challenge with its 140+ heavily interdependent libraries with dependency chains at times 20+ levels deep. If you want to see what kind of issues a build system may run into while trying to deal with something like this, take a looks at this bug report: https://github.com/build2/build2/issues/184
I will assume your questions refer to build2 in general rather than to its header units support specifically (which is not very usable at the moment due to compiler deficiencies).
No. my comment was in header-units and modules support context.
On Fri, Mar 22, 2024 at 7:04 PM Boris Kolpackov
Hassan Sajjad
writes: That's incorrect: build2 had support for header units (with include translation) from 2021:
Thank you for informing me. I did not know about it. Can you give bigger examples? Is it used in production anywhere?
I will assume your questions refer to build2 in general rather than to its header units support specifically (which is not very usable at the moment due to compiler deficiencies).
Yes, you can see more substantial examples on cppget.org (packages) or the build2-packaging GitHub org (package repositories), which, in particular, include both Boost and Qt (and I agree with Andrey, SFML, which is also there, is not a particularly complex library). And yes, build2 is used in production, including in large (many hundreds of libraries), successful commercial projects.
And speaking of Boost (not to make this reply completely off-topic), in Hmake's README you say:
HMake 1.0 will only be released when, few of the mega projects like UE5, AOSP, Qt etc. could be supported.
I would suggest that you include Boost on this list. While libraries like Qt give good "depth-wise" reality check (i.e., relatively few libraries with very complex builds), Boost gives a unique "breadth-wise" challenge with its 140+ heavily interdependent libraries with dependency chains at times 20+ levels deep. If you want to see what kind of issues a build system may run into while trying to deal with something like this, take a looks at this bug report:
Hassan Sajjad
I will assume your questions refer to build2 in general rather than to its header units support specifically (which is not very usable at the moment due to compiler deficiencies).
No. my comment was in header-units and modules support context.
Alright, though in the future you may want to ask clearer questions ("it" can be notoriously ambiguous). To briefly answer your actual question, I don't believe header units are used in production by anyone except maybe in some controlled, "we pre-compile all header units before our project" cases (I believe Microsoft does something like this in their internal projects). The reason being compiler deficiencies (lack of build system support in Clang and MSVC, and general bugginess of modules support in GCC, though I saw a lot of related bugs being fixed for GCC 14 so things might be changing). For named modules, yes, they are being used (including with support for `import std;`). I have tried this on async_simple library: https://github.com/boris-kolpackov/async_simple/tree/CXX20Modules-build2/asy... And I know of a few build2 users who are using named modules in their greenfield projects.
Alright, though in the future you may want to ask clearer questions ("it" can be notoriously ambiguous).
Sorry for the confusion. I don't believe header units
are used in production by anyone except maybe in some controlled, "we pre-compile all header units before our project" cases (I believe Microsoft does something like this in their internal projects).
I compiled SFML with C++20 header-units with little to no changes in the
SFML codebase. The clean build of drop-in configuration was 16s on my
system compared to conventional 20s. Can build2 support such usage at such
a scale? If not then I feel that my build-system has a plus point in module
support. I would also like to see the syntax for such an example to do a
comparison as I feel that C++ is a better choice (but that is rather
subjective).
I have best wishes for your project. But I am a bit opinionated about
build-systems.
On Fri, Mar 22, 2024 at 8:38 PM Boris Kolpackov
Hassan Sajjad
writes: I will assume your questions refer to build2 in general rather than to its header units support specifically (which is not very usable at the moment due to compiler deficiencies).
No. my comment was in header-units and modules support context.
Alright, though in the future you may want to ask clearer questions ("it" can be notoriously ambiguous).
To briefly answer your actual question, I don't believe header units are used in production by anyone except maybe in some controlled, "we pre-compile all header units before our project" cases (I believe Microsoft does something like this in their internal projects). The reason being compiler deficiencies (lack of build system support in Clang and MSVC, and general bugginess of modules support in GCC, though I saw a lot of related bugs being fixed for GCC 14 so things might be changing).
For named modules, yes, they are being used (including with support for `import std;`). I have tried this on async_simple library:
https://github.com/boris-kolpackov/async_simple/tree/CXX20Modules-build2/asy...
And I know of a few build2 users who are using named modules in their greenfield projects.
On 21/03/2024 17:34, Hassan Sajjad via Boost wrote:
I want to propose my build-system HMake for boost. Besides its already state-of-the-art C++20 modules and header-units support, it will be the first to support https://lists.isocpp.org/sg15/2023/11/2106.php or https://lists.isocpp.org/sg15/2023/11/2146.php if any of this gets implemented. These papers present ideas for faster module adoption and avoiding redundant module compilations. These need support from build-system. This way boost could be a breeding ground for C++20 modules adoption. Also, I have good experience with boost's current build-system b2 as a good portion of HMake's current API is inspired by it.
Unless it's improved cmake build support for Boost, I cannot say I am interested in yet another build system. I also think few would consume Boost as modules for many, many years yet to come. Precompiled headers if they work provide most of the build speed improvements, and everything else Modules v1 supplies has marginal utility for the vast majority of C++ use cases if I am blunt. I am still hoping for a Modules v2 from the committee which has tangible benefits for the majority of users. Niall
I am still hoping for a Modules v2 from the committee which has tangible benefits for the majority of users.
I am interested in what v1 misses that you want in v2. I believe because of https://lists.isocpp.org/sg15/2023/11/2106.php, the adoption will be much quicker than you think. On Fri, Mar 22, 2024 at 3:47 PM Niall Douglas via Boost < boost@lists.boost.org> wrote:
On 21/03/2024 17:34, Hassan Sajjad via Boost wrote:
I want to propose my build-system HMake for boost. Besides its already state-of-the-art C++20 modules and header-units support, it will be the first to support https://lists.isocpp.org/sg15/2023/11/2106.php or https://lists.isocpp.org/sg15/2023/11/2146.php if any of this gets implemented. These papers present ideas for faster module adoption and avoiding redundant module compilations. These need support from build-system. This way boost could be a breeding ground for C++20 modules adoption. Also, I have good experience with boost's current build-system b2 as a good portion of HMake's current API is inspired by it.
Unless it's improved cmake build support for Boost, I cannot say I am interested in yet another build system.
I also think few would consume Boost as modules for many, many years yet to come. Precompiled headers if they work provide most of the build speed improvements, and everything else Modules v1 supplies has marginal utility for the vast majority of C++ use cases if I am blunt.
I am still hoping for a Modules v2 from the committee which has tangible benefits for the majority of users.
Niall
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 21.03.24 18:34, Hassan Sajjad via Boost wrote:
I developed the build-system HMake https://github.com/HassanSajjad-302/HMake. It is in C++ and MIT Licenced. It supports drop-in header-files to header-units replacement. While compilers support processing a header-file as header-unit, no other build-system supports this feature.
My experience with existing build systems is that they all suck. I currently have my own ad-hoc build system written in Python, and I'd love to be able to drop it, especially since my own system is currently also unable to build C++ modules, but none of the existing build systems I tried were able to meet my needs. I am therefore interested in a new build system in principle, but also extremely picky about which build systems I am interesting in. In terms of what my build system does, actually compiling the C++ source files is the easiest 1%. The hard parts are building libraries, processing data files, and packaging the complete project for distribution. Some more specific requirements: - Builds should be hermetic. Compiling the same project with the same command-line arguments on different computers should result in bit-for-bit identical results. (I currently handle this by compiling in a Docker container.) - I want to build for multiple target operating systems on one computer, from one source directory, with one invocation of the build system. - I don't like to edit my build specification files every time I add or remove a file, so I use wildcards extensively in specifying which source files to build. - Some executables need to be built for the target platforms. Some only need to be built for the build platform, so that I can run them as part of the build process. - The set of artifacts produced from one (data) source file is not known until the source file is processed. (Example: unzipping a zip file as part of the process of building the library contained in the zip file.) -- Rainer Deyke (rainerd@eldwood.com)
In terms of what my build system does, actually compiling the C++ source files is the easiest 1%. The hard parts are building libraries, processing data files, and packaging the complete project for distribution. Some more specific requirements:
HMake has APIs for some of these and is in the pipeline for some others. HMake is at its core a dependency resolver. So, anything you can do in make, you can do in HMake. In the future, high-end APIs will be available for most languages and common tasks. A common task during package development is compression and build-system generally first checks whether such commands as gzip or bz2 are available. With HMake, a compression API will be in-built that works on all operating systems. I want to build for multiple target operating systems on one
computer, from one source directory, with one invocation of the build system.
Check. Build-system has very good support for this. - I don't like to edit my build specification files every time I add
or remove a file, so I use wildcards extensively in specifying which source files to build.
Check. Build-system has very good support for this. I don't understand other requirements. I can comment if you could give some examples. C++20 modules and header-units is a complex maneuver and build-system is extendable so most likely build-system supports / can support other features as well. On Fri, Mar 22, 2024 at 5:34 PM Rainer Deyke via Boost < boost@lists.boost.org> wrote:
I developed the build-system HMake https://github.com/HassanSajjad-302/HMake. It is in C++ and MIT Licenced. It supports drop-in header-files to header-units replacement. While compilers support processing a
On 21.03.24 18:34, Hassan Sajjad via Boost wrote: header-file
as header-unit, no other build-system supports this feature.
My experience with existing build systems is that they all suck. I currently have my own ad-hoc build system written in Python, and I'd love to be able to drop it, especially since my own system is currently also unable to build C++ modules, but none of the existing build systems I tried were able to meet my needs. I am therefore interested in a new build system in principle, but also extremely picky about which build systems I am interesting in.
In terms of what my build system does, actually compiling the C++ source files is the easiest 1%. The hard parts are building libraries, processing data files, and packaging the complete project for distribution. Some more specific requirements:
- Builds should be hermetic. Compiling the same project with the same command-line arguments on different computers should result in bit-for-bit identical results. (I currently handle this by compiling in a Docker container.) - I want to build for multiple target operating systems on one computer, from one source directory, with one invocation of the build system. - I don't like to edit my build specification files every time I add or remove a file, so I use wildcards extensively in specifying which source files to build. - Some executables need to be built for the target platforms. Some only need to be built for the build platform, so that I can run them as part of the build process. - The set of artifacts produced from one (data) source file is not known until the source file is processed. (Example: unzipping a zip file as part of the process of building the library contained in the zip file.)
-- Rainer Deyke (rainerd@eldwood.com)
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Rainer Deyke via Boost
- Builds should be hermetic. Compiling the same project with the same command-line arguments on different computers should result in bit-for-bit identical results.
I believe what you are describing is commonly referred to as "reproducible builds". Hermeticity (broadly) means having a complete track of everything that contributes into the build and being able to prevent (or at least detect) uncontrolled changes to any of it. Hermeticity makes it easier to achieve reproducible builds but is neither required nor sufficient. True hermeticity is also very difficult to achieve in C/C++ projects.
(I currently handle this by compiling in a Docker container.)
Well, that doesn't really qualify as "different computers", does it? It's always "your (virtualized) computer" ;-).
- I want to build for multiple target operating systems on one computer, from one source directory, with one invocation of the build system. - I don't like to edit my build specification files every time I add or remove a file, so I use wildcards extensively in specifying which source files to build. - Some executables need to be built for the target platforms. Some only need to be built for the build platform, so that I can run them as part of the build process. - The set of artifacts produced from one (data) source file is not known until the source file is processed. (Example: unzipping a zip file as part of the process of building the library contained in the zip file.)
All covered by build2, if you are interested. We also have a notion of hermetic build configurations (though they are no magic, unfortunately): https://build2.org/release/0.14.0.xhtml#hermetic
Hi. Sorry but I'm against it. We already have the actual build sistem that's ok. If some feature is missing, like support for modules, we can update it if needed. Besides, boost is already build with cmake in package managers like vcpkg (I don't have experience with other package managers like conan but I assume that they are not reinventing the wheel). Adding another build system (that's not widely used) improves only the entropy. For me it's ok to add some issue for current build system. For me it's ok to give wider support to already supported (officially or unofficially) tools like cmake, even if one day the main group will decide to switch to one one them. But switching to a build system not widely used and/or tested it has no advantages and a lot of disadvantages and potential problems (cmake and b2 are tested over all kind of computer configuration in the world, even a little percentage of problem can afflict many, many users). Regards Daniele Lupo
If some feature is missing, like support for modules, we can update it if needed
I think this will need some effort. A better alternative could be to use a newer build-system maybe. But switching to a build system not widely used and/or tested it has no
advantages and a lot of disadvantages and potential problems (cmake and b2 are tested over all kind of computer configuration in the world, even a little percentage of problem can afflict many, many users).
HMake testing is quite extensive but at the moment it is 8/10. I know of 2 bugs. These would be fixed using test test-driven approach i.e. first a failing test-case would be added and then fixed. A production build-system cannot afford any bugs. I am confident that I can deliver such a system in the timeframe quoted and that the transition will be smooth sailing. Again, the biggest advantage compared to the competition is drop-in header-units support which could result in 2.5x speed-up. If this could be added in b2 for cheaper then well and good, otherwise you can consider mine. On Fri, Mar 22, 2024 at 7:49 PM Daniele Lupo via Boost < boost@lists.boost.org> wrote:
Hi.
Sorry but I'm against it. We already have the actual build sistem that's ok. If some feature is missing, like support for modules, we can update it if needed. Besides, boost is already build with cmake in package managers like vcpkg (I don't have experience with other package managers like conan but I assume that they are not reinventing the wheel). Adding another build system (that's not widely used) improves only the entropy.
For me it's ok to add some issue for current build system.
For me it's ok to give wider support to already supported (officially or unofficially) tools like cmake, even if one day the main group will decide to switch to one one them.
But switching to a build system not widely used and/or tested it has no advantages and a lot of disadvantages and potential problems (cmake and b2 are tested over all kind of computer configuration in the world, even a little percentage of problem can afflict many, many users).
Regards
Daniele Lupo
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 22/03/2024 18:16, Hassan Sajjad wrote:
If some feature is missing, like support for modules, we can update it if needed
I think this will need some effort. A better alternative could be to use a newer build-system maybe.
I don't think that updating a software that works for adding a feature requires more effort that reinvent another build system that must do all the things of the existing one and then also the same new ones. Obviously it depends also on the original software architecture and code, but I can bet that's simpler to update boost build.
But switching to a build system not widely used and/or tested it has no advantages and a lot of disadvantages and potential problems (cmake and b2 are tested over all kind of computer configuration in the world, even a little percentage of problem can afflict many, many users).
HMake testing is quite extensive but at the moment it is 8/10. I know of 2 bugs. These would be fixed using test test-driven approach i.e. first a failing test-case would be added and then fixed. A production build-system cannot afford any bugs. I am confident that I can deliver such a system in the timeframe quoted and that the transition will be smooth sailing. Again, the biggest advantage compared to the competition is drop-in header-units support which could result in 2.5x speed-up. If this could be added in b2 for cheaper then well and good, otherwise you can consider mine.
I'm sure that everything can be done, but I'm not sure that everything that can be done must be done. In my opinion is reinventing the wheel, even if HMake was already invented by you, because for boost it is. I don't see at the moment anything that cannot be implemented in boost build, if necessary, that requires more effort that create/introduce a new build system from scratch. There's also another thing to consider, that it's good to add new features only if they are requested from the community. At the moment we didn't see anything like that. Obviously being proactive is always a good idea, but taking the effort for updating boost build, or even changing entirely the build system, for adding features that are not requested and so only very few people will use, it's a waste of time and effort. It's true that I don't read this mailing list very often, but I didn't see anything like that. When we want to update something, usually it's necessary think about what to change, and only then how to do it. the "what" is missing at the moment in the community, so first there should be discussion like: - "Oh, I'd like to have boost in new c++ modules" - "Really? That sounds good... Who wants it?" - "Me!" -"Me!" - "Ok, X people wants this feature. They are a lot or are only a few? It's worth to develop such a feature?" - "Let's discuss and decide here" So we discuss, and only if the community decides that the feature is needed, and the "what" is clear, it's possible to start to think about the "how". So, for example, we should talk first about the possibility/necessity to have new build features, at high level, and only if we decide that they're worth the effort, we can start to say "ok, and now how can we do it?". And even if the "what" is good, the "how" can be complicated time consuming, so it's also possible that the wanted feature will be discarded or paused for future discussions. It's something that happens often in software development. So, let's talk about the new build features, let's decide if they are good ideas, and only after that we can start to talk about a new build system instead of updating the existing one. For those reason, "at this time", I'm not convinced to use a new build system. But if the "what" is good enough, I think that it's something that can be taken into account. Regards. Daniele Lupo
There is not an immediate need to switch to C++20 modules or header-units. But why miss-out if we can get some compilation speed-up. In-case of header-units, no changes need to be made in source-code. Slow compilation is not a big problem for Boost compared to other projects like UE5 and LibreOffice. However, there is still interest in using C++20 modules and header-units.
I don't think that updating a software that works for adding a feature requires more effort that reinvent another build system that must do all the things of the existing one and then also the same new ones. Obviously it depends also on the original software architecture and code, but I can bet that's simpler to update boost build.
If you accept the 'what' part described above, you are wrong in evaluating
the 'how' part. The Boost build maintainer could comment on it.
On Fri, Mar 22, 2024 at 11:03 PM Daniele Lupo
On 22/03/2024 18:16, Hassan Sajjad wrote:
If some feature is missing, like support for modules, we can update
it if needed
I think this will need some effort. A better alternative could be to use a newer build-system maybe.
I don't think that updating a software that works for adding a feature requires more effort that reinvent another build system that must do all the things of the existing one and then also the same new ones. Obviously it depends also on the original software architecture and code, but I can bet that's simpler to update boost build.
But switching to a build system not widely used and/or tested it has no
advantages and a lot of disadvantages and potential problems (cmake and b2 are tested over all kind of computer configuration in the world, even a little percentage of problem can afflict many, many users).
HMake testing is quite extensive but at the moment it is 8/10. I know of 2 bugs. These would be fixed using test test-driven approach i.e. first a failing test-case would be added and then fixed. A production build-system cannot afford any bugs. I am confident that I can deliver such a system in the timeframe quoted and that the transition will be smooth sailing. Again, the biggest advantage compared to the competition is drop-in header-units support which could result in 2.5x speed-up. If this could be added in b2 for cheaper then well and good, otherwise you can consider mine.
I'm sure that everything can be done, but I'm not sure that everything that can be done must be done. In my opinion is reinventing the wheel, even if HMake was already invented by you, because for boost it is. I don't see at the moment anything that cannot be implemented in boost build, if necessary, that requires more effort that create/introduce a new build system from scratch.
There's also another thing to consider, that it's good to add new features only if they are requested from the community. At the moment we didn't see anything like that. Obviously being proactive is always a good idea, but taking the effort for updating boost build, or even changing entirely the build system, for adding features that are not requested and so only very few people will use, it's a waste of time and effort. It's true that I don't read this mailing list very often, but I didn't see anything like that.
When we want to update something, usually it's necessary think about what to change, and only then how to do it. the "what" is missing at the moment in the community, so first there should be discussion like:
- "Oh, I'd like to have boost in new c++ modules" - "Really? That sounds good... Who wants it?" - "Me!" -"Me!" - "Ok, X people wants this feature. They are a lot or are only a few? It's worth to develop such a feature?" - "Let's discuss and decide here"
So we discuss, and only if the community decides that the feature is needed, and the "what" is clear, it's possible to start to think about the "how".
So, for example, we should talk first about the possibility/necessity to have new build features, at high level, and only if we decide that they're worth the effort, we can start to say "ok, and now how can we do it?". And even if the "what" is good, the "how" can be complicated time consuming, so it's also possible that the wanted feature will be discarded or paused for future discussions. It's something that happens often in software development.
So, let's talk about the new build features, let's decide if they are good ideas, and only after that we can start to talk about a new build system instead of updating the existing one.
For those reason, "at this time", I'm not convinced to use a new build system. But if the "what" is good enough, I think that it's something that can be taken into account.
Regards.
Daniele Lupo
Hi.
Could you please confirm if my proposal is under consideration? If so, when
can I expect an update regarding its status?
Best,
Hassan Sajjad
On Sat, Mar 23, 2024 at 3:50 AM Hassan Sajjad
There is not an immediate need to switch to C++20 modules or header-units. But why miss-out if we can get some compilation speed-up. In-case of header-units, no changes need to be made in source-code. Slow compilation is not a big problem for Boost compared to other projects like UE5 and LibreOffice. However, there is still interest in using C++20 modules and header-units.
I don't think that updating a software that works for adding a feature requires more effort that reinvent another build system that must do all the things of the existing one and then also the same new ones. Obviously it depends also on the original software architecture and code, but I can bet that's simpler to update boost build.
If you accept the 'what' part described above, you are wrong in evaluating the 'how' part. The Boost build maintainer could comment on it.
On Fri, Mar 22, 2024 at 11:03 PM Daniele Lupo
wrote: On 22/03/2024 18:16, Hassan Sajjad wrote:
If some feature is missing, like support for modules, we can update
it if needed
I think this will need some effort. A better alternative could be to use a newer build-system maybe.
I don't think that updating a software that works for adding a feature requires more effort that reinvent another build system that must do all the things of the existing one and then also the same new ones. Obviously it depends also on the original software architecture and code, but I can bet that's simpler to update boost build.
But switching to a build system not widely used and/or tested it has no
advantages and a lot of disadvantages and potential problems (cmake and b2 are tested over all kind of computer configuration in the world, even a little percentage of problem can afflict many, many users).
HMake testing is quite extensive but at the moment it is 8/10. I know of 2 bugs. These would be fixed using test test-driven approach i.e. first a failing test-case would be added and then fixed. A production build-system cannot afford any bugs. I am confident that I can deliver such a system in the timeframe quoted and that the transition will be smooth sailing. Again, the biggest advantage compared to the competition is drop-in header-units support which could result in 2.5x speed-up. If this could be added in b2 for cheaper then well and good, otherwise you can consider mine.
I'm sure that everything can be done, but I'm not sure that everything that can be done must be done. In my opinion is reinventing the wheel, even if HMake was already invented by you, because for boost it is. I don't see at the moment anything that cannot be implemented in boost build, if necessary, that requires more effort that create/introduce a new build system from scratch.
There's also another thing to consider, that it's good to add new features only if they are requested from the community. At the moment we didn't see anything like that. Obviously being proactive is always a good idea, but taking the effort for updating boost build, or even changing entirely the build system, for adding features that are not requested and so only very few people will use, it's a waste of time and effort. It's true that I don't read this mailing list very often, but I didn't see anything like that.
When we want to update something, usually it's necessary think about what to change, and only then how to do it. the "what" is missing at the moment in the community, so first there should be discussion like:
- "Oh, I'd like to have boost in new c++ modules" - "Really? That sounds good... Who wants it?" - "Me!" -"Me!" - "Ok, X people wants this feature. They are a lot or are only a few? It's worth to develop such a feature?" - "Let's discuss and decide here"
So we discuss, and only if the community decides that the feature is needed, and the "what" is clear, it's possible to start to think about the "how".
So, for example, we should talk first about the possibility/necessity to have new build features, at high level, and only if we decide that they're worth the effort, we can start to say "ok, and now how can we do it?". And even if the "what" is good, the "how" can be complicated time consuming, so it's also possible that the wanted feature will be discarded or paused for future discussions. It's something that happens often in software development.
So, let's talk about the new build features, let's decide if they are good ideas, and only after that we can start to talk about a new build system instead of updating the existing one.
For those reason, "at this time", I'm not convinced to use a new build system. But if the "what" is good enough, I think that it's something that can be taken into account.
Regards.
Daniele Lupo
чт, 21 мар. 2024 г. в 20:54, Hassan Sajjad via Boost
I developed the build-system HMake https://github.com/HassanSajjad-302/HMake.
Looking at example 2 I see void buildSpecification() { Configuration &debug = GetConfiguration("Debug"); debug.ASSIGN(ConfigType::DEBUG); Configuration &release = GetConfiguration("Release"); release.ASSIGN(LTO::ON); configurationSpecification(debug); configurationSpecification(release); } and then
Building this example will create two directories Debug and Release, based on the GetConfiguraion line in buildSpecification.
Do I understand correctly, that I would have to change the build script (since it's a cpp file, is build source file more fitting?) to build a new configuration?
It is in C++ and MIT Licenced.
Our main build system is currently delivered together with Boost sources and it bootstraps itself. Meaning, Boost doesn't need anything beyond a compiler to be built. HMake appears to require CMake to be built. In addition, if hmake is to become a part of Boost release archive, it would have to be licensed under BSL.
Do I understand correctly, that I would have to change the build script (since it's a cpp file, is build source file more fitting?) to build a new configuration?
Yes. Do you think it is wrong? Why? How do you envision it? HMake appears to require CMake to be
built. In addition, if hmake is to become a part of Boost release archive, it would have to be licensed under BSL.
Unless HMake gets very popular, this will remain a dependency for the time being. I would license it under BSL. No problem. On Fri, Mar 22, 2024 at 9:14 PM Дмитрий Архипов via Boost < boost@lists.boost.org> wrote:
чт, 21 мар. 2024 г. в 20:54, Hassan Sajjad via Boost < boost@lists.boost.org>:
I developed the build-system HMake https://github.com/HassanSajjad-302/HMake.
Looking at example 2 I see
void buildSpecification() { Configuration &debug = GetConfiguration("Debug"); debug.ASSIGN(ConfigType::DEBUG); Configuration &release = GetConfiguration("Release"); release.ASSIGN(LTO::ON);
configurationSpecification(debug); configurationSpecification(release); }
and then
Building this example will create two directories Debug and Release, based on the GetConfiguraion line in buildSpecification.
Do I understand correctly, that I would have to change the build script (since it's a cpp file, is build source file more fitting?) to build a new configuration?
It is in C++ and MIT Licenced.
Our main build system is currently delivered together with Boost sources and it bootstraps itself. Meaning, Boost doesn't need anything beyond a compiler to be built. HMake appears to require CMake to be built. In addition, if hmake is to become a part of Boost release archive, it would have to be licensed under BSL.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
пт, 22 мар. 2024 г. в 20:32, Hassan Sajjad
Do I understand correctly, that I would have to change the build script (since it's a cpp file, is build source file more fitting?) to build a new configuration?
Yes. Do you think it is wrong? Why? How do you envision it?
Consider this b2 project: # jamroot.jam exe main : main.cpp ; In order to build it in debug and release I would call b2 like this: b2 variant=debug,release With a similar CMake project: #CMakeLists.txt cmake_minimum_required(VERSION 3.5) project(proj LANGUAGES CXX) add_executable(main main.cpp) Building debug and release (using a single config generator) is done with e.g. cmake -S. -Bbuild/debug -DCMAKE_BUILD_TYPE=Debug cmake --build build/debug cmake -S. -Bbuild/release -DCMAKE_BUILD_TYPE=Release cmake --build build/release Neither b2 nor CMake requires any changes to build scripts to build different configurations.
Hi. Yes. You need to edit the hmake.cpp to define the new configuration. You can create custom configurations very easily e.g. a configuration in which half the targets are built with debug symbols while the other half are not. And you don't need to build each configuration manually. One hbuild command in build-dir will build all the configurations. But you can build only one configuration by cd build-dir/conf hbuild You can rebuild by first deleting the configuration directory contents and then running hbuild in that directory. It will not impact other configurations. On Fri, Mar 22, 2024 at 10:57 PM Дмитрий Архипов via Boost < boost@lists.boost.org> wrote:
пт, 22 мар. 2024 г. в 20:32, Hassan Sajjad
: Do I understand correctly, that I would have to change the build script (since it's a cpp file, is build source file more fitting?) to build a new configuration?
Yes. Do you think it is wrong? Why? How do you envision it?
Consider this b2 project:
# jamroot.jam exe main : main.cpp ;
In order to build it in debug and release I would call b2 like this:
b2 variant=debug,release
With a similar CMake project:
#CMakeLists.txt cmake_minimum_required(VERSION 3.5) project(proj LANGUAGES CXX) add_executable(main main.cpp)
Building debug and release (using a single config generator) is done with e.g. cmake -S. -Bbuild/debug -DCMAKE_BUILD_TYPE=Debug cmake --build build/debug cmake -S. -Bbuild/release -DCMAKE_BUILD_TYPE=Release cmake --build build/release
Neither b2 nor CMake requires any changes to build scripts to build different configurations.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
сб, 23 мар. 2024 г. в 01:24, Hassan Sajjad
Hi.
Yes. You need to edit the hmake.cpp to define the new configuration. You can create custom configurations very easily e.g. a configuration in which half the targets are built with debug symbols while the other half are not. And you don't need to build each configuration manually. One hbuild command in build-dir will build all the configurations. But you can build only one configuration by cd build-dir/conf build
If I understand you correctly then it is a serious issue. The way build systems, which I am aware of, work is that project authors define how targets are to be built given certain circumstances, and then project users decide what the circumstances are. For example (using b2): lib a : a.cpp : <link>static:<define>STATIC_BUILD ; This defines a target which has one source file, and IF the target is built with static linking, the macro STATIC_BUILD is defined. The build script does not limit the build to static or dynamic linking. The user can choose to build a static library, a dynamic library, or both. It doesn't require the user to change the build script. Same for debug or release builds, or using different compilers. The user can even define more macros or set specific compiler and linker flags. None of that requires changing the build script. At most the user would edit a special configuration file. So, do I understand you correctly that this is not how hmake works, and one would have to change the build script for any variation of build circumstances? I.e. if I want to add a static build to a library, enable a sanitizer, use a different compiler, add a specific compiler flag, define a macro, disable exceptions, change strictness of warnings --- any of that would require a change to the build script.
If I understand you correctly then it is a serious issue.
You have explained the HMake working correctly but I fail to see why it is a serious issue. You can define a new configuration by adding one more line in the configuration file. In other build-systems, you would write this line on the command-line. In HMake, you write this in the configuration file. On Sat, Mar 23, 2024 at 6:59 PM Дмитрий Архипов via Boost < boost@lists.boost.org> wrote:
сб, 23 мар. 2024 г. в 01:24, Hassan Sajjad
: Hi.
Yes. You need to edit the hmake.cpp to define the new configuration. You
can create custom configurations very easily e.g. a configuration in which half the targets are built with debug symbols while the other half are not. And you don't need to build each configuration manually. One hbuild command in build-dir will build all the configurations. But you can build only one configuration by
cd build-dir/conf build
If I understand you correctly then it is a serious issue. The way build systems, which I am aware of, work is that project authors define how targets are to be built given certain circumstances, and then project users decide what the circumstances are. For example (using b2):
lib a : a.cpp : <link>static:<define>STATIC_BUILD ;
This defines a target which has one source file, and IF the target is built with static linking, the macro STATIC_BUILD is defined. The build script does not limit the build to static or dynamic linking. The user can choose to build a static library, a dynamic library, or both. It doesn't require the user to change the build script. Same for debug or release builds, or using different compilers. The user can even define more macros or set specific compiler and linker flags. None of that requires changing the build script. At most the user would edit a special configuration file.
So, do I understand you correctly that this is not how hmake works, and one would have to change the build script for any variation of build circumstances? I.e. if I want to add a static build to a library, enable a sanitizer, use a different compiler, add a specific compiler flag, define a macro, disable exceptions, change strictness of warnings --- any of that would require a change to the build script.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
сб, 23 мар. 2024 г. в 17:15, Hassan Sajjad
You have explained the HMake working correctly but I fail to see why it is a serious issue. You can define a new configuration by adding one more line in the configuration file. In other build-systems, you would write this line on the command-line. In HMake, you write this in the configuration file.
The key difference is that users' configuration files and common line invocations are not project files. Build scripts are necessarily a part of the project, they have to be tracked by git. If in order to try a different configuration I would have to change a tracked file, I suddenly have a non-clean workspace, and the change is most likely not something I would want to add to the next commit. Further, now to build the project for their required configuration users would have to effectively patch the project sources. This is very unwieldy for any kind of automation (CI). And will also require them to know hmake. I don't see how this is reasonable from UX perspective.
You have raised 3 points in your email. 1) git tracking I think one option could be to define all configurations wrapped by cache variables, out of which just one builds. <quote> void buildSpecification() { if(CacheVariable("Debug", true).value) { Configuration &debug = GetConfiguration("Debug"); debug.ASSIGN(ConfigType::DEBUG); configurationSpecification(debug); } if(CacheVariable("Release", false).value) { Configuration &release = GetConfiguration("Release"); release.ASSIGN(LTO::ON); configurationSpecification(release); } } </quote> Now, there is a cache variable Release set to false in cache.json in build-dir. By default, there will be only one configuration Debug configured. To configure the Release configuration, the user will have to set the Release variable in cache.json to true and then run hhelper command again in the build-dir. This way user can control the configured configurations by changing the cache.json file which is not tracked by git. 2) CI automation I think the above could be configured through command-line as-well. 3) And will also require them to know hmake. The alternative is to learn the command-line. So, I feel that it is better as it is. But the alternative is possible as-well. hbuild could be configured to pass command-line options to our hmake.cpp where we can parse them and define configurations based on those variables. On Sat, Mar 23, 2024 at 7:38 PM Дмитрий Архипов via Boost < boost@lists.boost.org> wrote:
You have explained the HMake working correctly but I fail to see why it is a serious issue. You can define a new configuration by adding one more
сб, 23 мар. 2024 г. в 17:15, Hassan Sajjad
: line in the configuration file. In other build-systems, you would write this line on the command-line. In HMake, you write this in the configuration file. The key difference is that users' configuration files and common line invocations are not project files. Build scripts are necessarily a part of the project, they have to be tracked by git. If in order to try a different configuration I would have to change a tracked file, I suddenly have a non-clean workspace, and the change is most likely not something I would want to add to the next commit. Further, now to build the project for their required configuration users would have to effectively patch the project sources. This is very unwieldy for any kind of automation (CI). And will also require them to know hmake.
I don't see how this is reasonable from UX perspective.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Am 23.03.24 um 19:49 schrieb Hassan Sajjad via Boost:
3) And will also require them to know hmake. The alternative is to learn the command-line. The commandline is a closed set. I.e. there are a few allowed switches with a few allowed values. E.g. for b2 it is `b2 variant=release`, `b2 variant=debug` or even `b2 variant=release,debug`
hbuild could be configured to pass command-line options to our hmake.cpp where we can parse them and define configurations based on those variables. Now library authors need to add and maintain code for command-line
In your example C++ is used which means the possible things you can do are literally unlimitted. This makes it much harder to use. parsing just to build. Not an improvement IMO. Especially as it means that different libraries might parse the commandline differently. Especially for Boost which is a collection of libraries, not a single one., although they get built together. Another downside of your example: To build only a release build, which is likely the only thing users care about, you need to set 2 cache variables. This is one reason an established build system is better: People know things like this or they have been solved/improved over the time the build system already exists. Also the supposed benefit of faster compile times isn't convincing for me: In your example you saved 5 seconds. I don't really build often and especially I don't build from scratch often but rather do incremental builds. So the saved time compiling doesn't make up the time writing stuff for a new build system. So my vote would be NO on this proposal. PS: Please also read the mailing list rules as already indicated by others, especially https://www.boost.org/community/policy.html#quoting
E.g. for b2 it is `b2 variant=release`, `b2 variant=debug` or even `b2 variant=release,debug`
In your example C++ is used which means the possible things you can do are literally unlimitted. This makes it much harder to use.
This is not a bad thing. By using C++, users have full power available to them in case they need it. The pattern demonstrated in Example 2 is modeled on Boost build-system b2. So there is a limited set of enmu values that can be assigned to the Configuration. These enums are declared here. https://github.com/HassanSajjad-302/HMake/blob/main/hconfigure/header/Featur... Now library authors need to add and maintain code for command-line
parsing just to build. Not an improvement IMO. Especially as it means that different libraries might parse the commandline differently. Especially for Boost which is a collection of libraries, not a single one., although they get built together.
Firstly, I argued against command-line parsing in favor of editing cache.json file. But, if the user wants command-line parsing, there needs to be just one function used in hmake.cpp that parses arguments and declares configurations based on that. Library authors don't need to edit it. But again, it is not needed in the first place. Another downside of your example: To build only a release build, which
is likely the only thing users care about, you need to set 2 cache variables. This is one reason an established build system is better: People know things like this or they have been solved/improved over the time the build system already exists.
I don't see how it is worse than the alternatives. You need to edit one variable in cache.json. That's it. Also the supposed benefit of faster compile times isn't convincing for
me: In your example you saved 5 seconds. I don't really build often and especially I don't build from scratch often but rather do incremental builds. So the saved time compiling doesn't make up the time writing stuff for a new build system.
I compiled the SFML conventional build in the 20s while drop-in build was compiled in the 16s. So, it saved 4s which is 20% in clean-build. The benefit of header-units becomes bigger in bigger projects because it prevents the same file from being compiled repeatedly. It will be very noticeable in Boost compilation which would be, I estimate, 2.5x faster. I compiled Boost in 5mis (300s), so I expect drop-in configuration to build in 2min (120s). Header-units will save me 180s. Also, the incremental build is 2.5x faster. You can test it by editing a file and then measuring 'hbuild' time in conventional and drop-in directories in SFML build-dir. Again, if you are working on a project with numerous dependencies the impact of header-units for the incremental build is bigger because the compiler benefits from using the precompiled code instead of compiling it. This means that >4x improvement could be expected for some libraries. So, the compile-edit-run cycle will become instantaneous. On Mon, Mar 25, 2024 at 4:34 PM Alexander Grund via Boost < boost@lists.boost.org> wrote:
Am 23.03.24 um 19:49 schrieb Hassan Sajjad via Boost:
3) And will also require them to know hmake. The alternative is to learn the command-line. The commandline is a closed set. I.e. there are a few allowed switches with a few allowed values. E.g. for b2 it is `b2 variant=release`, `b2 variant=debug` or even `b2 variant=release,debug`
In your example C++ is used which means the possible things you can do are literally unlimitted. This makes it much harder to use.
hbuild could be configured to pass command-line options to our hmake.cpp where we can parse them and define configurations based on those variables. Now library authors need to add and maintain code for command-line parsing just to build. Not an improvement IMO. Especially as it means that different libraries might parse the commandline differently. Especially for Boost which is a collection of libraries, not a single one., although they get built together.
Another downside of your example: To build only a release build, which is likely the only thing users care about, you need to set 2 cache variables. This is one reason an established build system is better: People know things like this or they have been solved/improved over the time the build system already exists.
Also the supposed benefit of faster compile times isn't convincing for me: In your example you saved 5 seconds. I don't really build often and especially I don't build from scratch often but rather do incremental builds. So the saved time compiling doesn't make up the time writing stuff for a new build system.
So my vote would be NO on this proposal.
PS: Please also read the mailing list rules as already indicated by others, especially https://www.boost.org/community/policy.html#quoting
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
participants (9)
-
Alexander Grund
-
Andrey Semashev
-
Boris Kolpackov
-
Daniele Lupo
-
Hassan Sajjad
-
Niall Douglas
-
Rainer Deyke
-
Ruben Perez
-
Дмитрий Архипов