Could perhaps some of the more experienced developers provide sth like a pro and con list comparing cmake and the boost.build system? (maybe also including the coverage of test and documentation?) or also examples where we can see the benefit in terms of simplicity, maintainence, user experience :-), etc. I think some contributors/maintainers (including myself) are not able to appreciate (lack of experience/knowledge) the advantage of cmake over the boost.build system. For some the 'real' benefits might not be as clear as for the experienced users of build systems. Best C Am So., 16. Sep. 2018 um 13:47 Uhr schrieb James E. King III via Boost < boost@lists.boost.org>:

On Sat, Sep 15, 2018 at 11:33 AM mike via Boost wrote:

...

...

-----Original Message----- From: Boost On Behalf Of Bjorn Reese via Boost Sent: Saturday, September 15, 2018 2:11 PM To: Mike Dev via Boost Cc: Bjorn Reese Subject: Re: [boost] [cmake] Pull request announcement

On 09/14/18 13:44, Mike Dev via Boost wrote:

...

This will not address - Testing [...] In short: all the difficult stuff is left out ;) The goal is not to present a solution that can replace b2 but to have a minimal starting point upon which individual library authors or the community as a whole can make iterative improvements. If and when BCM (or an alternative) is accepted into boost, individual libraries can then easily switch to this full-fledged solution and of course any author can create a more complete CMake solution for his library at any time (as some already have).

In my experience, we want to have both an integration and standalone CMake file. This split is necessary to avoid problems with multiple instances of enable_testing() when using one library from another.

* The integration file builds only what is necessary for integration into other projects. This is similar to what you are proposing.

* The standalone file builds everything including test and documentation. This reuses the integration file to build the library itself.

If we place your proposed CMake file in the library root, and dependent libraries start using that location, then it becomes difficult to later have the standalone CMake file in the library root, which is the more natural choice. So we should be cautious about where we place these files.

Could you elaborate, why this would require separate/conflicting cmake files as opposed to having multiple different targets in a single file?

More to the point: Anything related to testing should reside in a cmake file inside the "tests" folder, which is called from the cmake file at the root, but that is mainly a matter of properly structuring the cmake code - just as you usually don't put all your code into a single header file, even if it might end up in a single translation unit in the end. The same is true for docs.

In particular: Just because a cmake file is processed doesn't mean any of the targets it contains have to be build automatically.

If you have genuine concerns however, I'm happy to further discuss them in detail. I certainly don't want to do anything that will prevent a future full-fledged cmake version.

I have used CMake for about 10 years now on pretty large projects, and I have never seen a need to have different CMakeLists files for different purposes. If you want to provide a build option where tests are not built or executed, you would add a top level option (throwing in examples and docs too):

OPTION(BOOST_DOCS "Build documentation." OFF) # most people won't have the required tooling OPTION(BOOST_EXAMPLES "Build examples." ON) OPTION(BOOST_TESTS "Build the tests." ON)

Unlike b2, tests do not typically run as part of a build when you use cmake. There are three phases of a cmake based build (well, four really...)

1. Create an out-of-tree build directory and generate the build environment with cmake. 2. Build using the generated build environment (which can be invoked with "cmake --build .") 3. Run tests using ctest. (4. cpack to build packages)

If you were to perform step 1 (generating the build) with "cmake -DBOOST_TESTS=OFF" then step 3 would either do nothing or produce an error indicating tests were disabled.

Someone asked what the naming convention for the libraries should be. You can teach CMake how to name libraries; the naming convention can be identical to how b2 does it. I don't see why it has to be any different than b2.

CMake isn't without it's oddities either. Selecting the build architecture and type (debug, x64) is done in step 1 on unix, but it is done on step 2 on windows. CMake does not natively provide windows package management. In all of the CMake build systems I have used or created so far I have always had to make an environmental harness to deal with third party deps. If anyone has tried to build libicu on windows (as an example), you know what a pain this can be. Therefore for it to be successful, tight integration with a package manager that supports Windows would be ideal - something that can provide libicu and the other boost dependencies automatically. I believe for example conan has some cmake integration.

Finally, someone suggested the project maintain both build systems long term. That would be a mistake. I've seen what this can do in other projects (like Apache Thrift, which uses autoconf for a complete linux build, and cmake for windows which is only a subset of languages). Each check-in would need to prove it builds correctly in two very different build systems. It is a lot to ask of folks submitting changes, but obviously the "complete" build environment has to work until the new one becomes "complete" and replaces it.

- Jim

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Am So., 16. Sep. 2018 um 18:08 Uhr schrieb Mateusz Loskot via Boost < boost@lists.boost.org>:

On Sun, 16 Sep 2018, 14:21 Cem Bassoy via Boost, wrote:

...

Could perhaps some of the more experienced developers provide sth like a pro and con list comparing cmake and the boost.build system?

IMHO, those have been listed here in countless posts already. One 'just' needs to dedicate one or two hours of life and learn about it from the archive.

Yes the problem is the word 'just'. There is no recap or summary of the discussion so it is quite hard in some cases to identify the pros and cons that are implicitly generated the mailing list. Don't get me wrong. I am not against reading old mails but I doubt that this process takes only one or two hours, especially in case of 'normal' users of the boost build, see e.g. Stefan's answer (active maintainer and developer). IMHO, it might be good for the transition (or inclusion of cmake) to give maintainers and contributers a chance (despite their lack of knowledge and experience) to make up their mind.

Instead of reposting pros/cons here again to make it into yet another forgotten post, those who care and need it should create a table on wiki with clear structured comparison.

Yes. Perfect. That must be enough for all to judge.

However, we should stop arguing CMake vs Build.Boost immediately, unless we have too much time to kill. Instead, we should work out how (not if!) to make room for both camps, Boost.Build and CMake, to co-exist officially and in peace within Boost.

Yes. That is what I meant. Provide technical solutions and *based on the technical solutions* there could be an extra list of pros and cons added and/or script comparison. Every maintainer can then make up their mind what the transition means and whether they 'really' need it. Could that be done? Where should the wiki or sth similar go? Best C

Best regards, Mateusz Loskot

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Instead of reposting pros/cons here again to make it into yet another forgotten post, those who care and need it should create a table on wiki with clear structured comparison.

It seems to me that the putative review manager (Robert) should canvass for solutions (there appear to be at least three now), do a quick check of them to see what tradeoffs each has, and make that table summarising their different approaches. He then could take a quick straw poll of here to see which set of tradeoffs the community dislikes least, and encourage that author to get their solution ready for review by an agreed date. Some have argued that all this is too confusing. It is in my opinion no different to any other Boost library. Somebody becomes willing to invest the time and effort to get a solution to a problem past review. Whichever successfully passes muster here is what we adopt. It's exactly the same as any other Boost library, and calling on the Steering Committee to declare a design by decree is the wrong way of doing this. Niall

On 9/16/18 11:30 AM, Rene Rivera via Boost wrote:

On Sun, Sep 16, 2018 at 1:20 PM Niall Douglas via Boost < boost@lists.boost.org> wrote:

...

...

Instead of reposting pros/cons here again to make it into yet another forgotten post, those who care and need it should create a table on wiki with clear structured comparison.

It seems to me that the putative review manager (Robert) should [...] make that table summarising their different approaches.

That's not normally the job of a review manager.

This is true.

It's usually on the authors of the solution(s) to make relevant comparisons to other solutions both in features and performance.

Also true. So the CMake "review" is going to be somewhat different than traditional reviews. Traditionally, a) we haven't reviewed tools. They were submitted and just accepted by ... I'm not sure whom. My motivation for promoting the idea of a review is to diminish the chances of me getting stuck with something that I don't like or is not useful to me. I also strongly believe in the value of the boost review process, in spite of the fact that it's often tedious, frustrating and never produces the exact result I want. b) we haven't solicited new software components in any manner official or otherwise. c) the author has an idea he wants boost to accept. The defines the scope, the interface and implementation and submits if for acceptance or rejection. If accepted, it becomes the defacto standard for that functionality and it effectively shuts down alternative future proposals. All in all this hasn't been a bad thing as it helps users with the selection of the component. But I don't see this working well in this case as I want everyone who want's to take a crack at this to take a shot. And I want them to know that if they do the work in the specified time frame, their work will be considered even though it can't be guaranteed to be accepted. d) often times libraries have been rejected because reviewers felt that the scope of the library was ill-chosen. This was OK as the library author got to propose what he wanted to. But now we're soliciting proposals, so it's unfair not to specify what we're asking for - that is the "scope". Since previous discussions have demonstrated (to me at least) that there is currently no well defined consensus as to what CMake should do in the context of boost, I've proposed that this be nailed down first. So here is what I propose: a) I will review the history of the topic, including previous proposals and get up to speed on the lastest facilities that CMake provides and post a document: "Solicitation for proposals to add CMake support to Boost" (or something like that - let the bike shedding begin!). I will try to post this by 1 October 2018. This document will contain: A list of functions that a CMake implementation is expected to perform. These might include some of the following and perhaps other stuff. 1) Help users import Boost Libraries in to their projects 2) Help users test a subset of Boost libraries on their own machine 3) Help library authors run tests of their libaries during development and post the results to a common area. 4) Help authors build IDE for their libraries to aid in development 5) Help everyone keep track of and accommodate source code and library dependencies. 6) Other stuff that hasn't occurred to me yet. A list of requirements that the submission should fulfill: License, Documentation, quality of implementation, platform coverage, commitment to implementation and maintenance for one year etc.... b) this "solicitation" will be subjected to comment and review by members of this list for up to 30 days. At the end of this period, I'll consolidate all this into a quasi official "Solicitation". If there isn't sufficient consensus reached to create such a "Solicitation", The process will end there. c) On or about 1 November 2018, this "Solicitation" will be posted on this list and the r/cpp forum on reddit. Authors will submit their proposals "unopened" to me until 1 February 2019. d) On or about 1 February 2019, Proposals will "unsealed" by me and I will inspect them to verify that they meet the stated minimum requirements such as documentation, tests etc. Any ones that fail this pre-review will be rejected with thanks and a polite note written by me. e) Any proposals still remaining will be posted along any explanatory notes that I have on them. It is quite possible that no proposals will remain at this point and process can then end without more discussion. f) After a two week review period, I'll decide whether or not the proposal is rejected or accepted and under what conditions. Note that in the above I'm used the first person I rather than review manager. This is just a convenience to make it easier to explain how the process is expected to work. If there is a consensus that someone else should do this, I'll be happy to support any other selection. Interested parties should feel free to nominate themselves for this job. So far, the Bored of Directors has been silent on the whole issue since the initial proclamation a year ago. I don't think this is necessarily a bad thing - if only to see what was going to happen. But now I think it's time for them to decide a couple of things and make a couple of announcements: a) that they endorse the process and designate the review manager (me or someone else). b) consider offering a monetary "prize" for submission and implementation of a winning proposal. I would expect such a prize to be in the neighbor of 5000-15000 US$. Funds might come from any where and the BOD would make some effort to gain sponsors for such a project. Robert Ramey

Hello everyone, I feel the need to clarify a few things, as this thread grew well beyond what I originally intended. Essentially I wrote the initial mail to inform you of pull request intentions [1], to prevent any impression that I wanted to sneak cmake into boost behind the back of the community. So, first to the WHY: The desire to have native support for cmake in boost does NOT stem from any (real or perceived) shortcoming of b2. It is about adding support for the de-facto standard build tool for cross-platform c++ library projects[2] instead of requiring users and contributors to deal with a proprietary one that is rarely used outside of boost. I'm explicitly mentioning this, because there were some voices asking for a comparison of b2 and cmake. While it would certainly be useful to have such a comparison documented in the wiki, I don't think it is particularly relevant to the discussion at hand. As to the WHAT: I applaud any effort to make Cmake a first level citizen in addition of or as a replacement for b2. If I have a vote, Robert absolutely gets my vote for his proposed process and for him as a review manager. HOWEVER, that is NOT the goal of the PRs I announced in this thread. And I WON'T submit any proposal to that end, because I think what I'm offering there is far too narrow in scope to be considered an "alternative" to BCM or the like. All I'm trying to achieve is to get a minimal support for cmake into as many boost libraries as possible, in a simple enough manner that any library maintainer feels comfortable in having the file as an official part of his library (although I'd be happy to provide support for them myself). The primary goal is to make project local compilation and consummation of boost easier, as well as lowering the bar for contributors. If one is just interested in consuming system installed boost libraries, the cmake developers have already HOW is this then related to BCM or other proposed full-fledged cmake solutions for boost? IT ISN'T! At least no more than the CMakeList.txt files that already exist in some of the libraries (e.g. hof, serialization, beast, hana, yap, gil, signals2, compute are some examples I'm aware of). The only thing I want to make sure is that any common, publicly visible practices my PRs introduce (E.g. naming convention) don't conflict with a future full-fledged solution. This is (I believe) achieved in two ways: a) There are very few such practices in the first place (naming convention is the only real one that is difficult to change after the fact) b) I'm sticking to well established practices in the Cmake world and hope that any other future solution isn't going against them (e.g. the name Boost::filesystem is the alias by which cmake user can express their dependency on boost.filesystem right now. Also, placing the root CMakeLists.txt file into the project root folder is default practice for cmake and used by almost any project I am aware of. Best Mike PS.: As a final note: Considering the shit storm Boost Steering Committee received after their initial announcement, I'm not surprised that they don't want to push the issue further. [1] Here are a few examples of my PRs (so far I've only submitted a handful anyway): https://github.com/boostorg/boost/pull/193 https://github.com/boostorg/predef/pull/86 https://github.com/boostorg/assert/pull/10 [2] Just as a related data point: Jetbrains makes a regular market survey "The State of Developer Ecosystem Survey" You can find the latest numbers related to c++ here (based on ~1000 c++ developers): https://www.jetbrains.com/research/devecosystem-2018/cpp/ CMake is used by 36% of the participants, next comes visual studio with 32%. In comparison, boost.Build ranges around 3%. Also, lookup a handful of your favorite non-boost OSS c++ libraries on GitHub. I bet most (probably even all) will support cmake at least as one alternative. For my team, boost is actually the only cross-platform dependency in any of our projects that doesn't support Cmake.

Am Mo., 17. Sep. 2018 um 13:21 Uhr schrieb Peter Dimov via Boost < boost@lists.boost.org>:

Mike Dev wrote:

...

All I'm trying to achieve is to get a minimal support for cmake into as many boost libraries as possible, in a simple enough manner that any library maintainer feels comfortable in having the file as an official part of his library (although I'd be happy to provide support for them myself). The primary goal is to make project local compilation and consummation of boost easier, as well as lowering the bar for contributors.

I'd like to see an example of how a CMake project using Boost (f.ex. System) would look like, assuming your proposed pull requests are accepted.

Yes. I agree. From your pull requests I can see only the root CMakeLists.txt for boost I guess. I would like to see examples of the CMakeLists for a project (maybe also inluding test, documentation?) In general, I really appreciate your effort to slowly integrate CMake without touching Boost's build system. And I think it might be the right way to do so. However, please note that accepting your solution (as an intermediate step) would mean that library maintainers/contributers need to support/maintain/debug two build system solutions. What about the integration tests? Do we need to deploy and test twice? Cheers C

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Mike Dev wrote:

...

I'd like to see an example of how a CMake project using Boost (f.ex. System) would look like, assuming your proposed pull requests are accepted.

Sure. You can have a look at https://github.com/Mike-Devel/DummyBoostUsage

Thanks.

file(GLOB boost_lib_cmake_files libs/*/CMakeLists.txt)

I wonder what would happen if Hana, Yap or Beast are present. :-)

On Tue, Sep 18, 2018 at 3:31 AM Mike Dev via Boost wrote:

...

-----Original Message----- From: Boost On Behalf Of Peter Dimov via Boost I wonder what would happen if Hana, Yap or Beast are present. :-)

Well, their cmake files get processed and do whatever they do. Obviously, they will not automagically benefit from the new boot targets created by me, as they don't know about them.

The main problem I encountered on linux (the offending modules refuse to build on windows anyway) is that some existing cmake files have bad hygiene.

E.g. "yap", "hana" and "callable_traits" each try to create a target "check".

What's wrong with that (not upset, just curious)? Zach

On Tue, Sep 18, 2018 at 10:39 AM Alexander Grund via Boost < boost@lists.boost.org> wrote:

...

...

E.g. "yap", "hana" and "callable_traits" each try to create a target "check".

What's wrong with that (not upset, just curious)? They conflict with each other. Targets need to have unique names.

Ah, of course, thanks. Do you have a preferred way to address this? F.ex., I only care about the "check" target in independent builds of Yap, mainly for Yap-only Travis CI. I could always turn it on conditional on a CI configuration variable I cold define. Zach

On Tue, Sep 18, 2018 at 10:52 AM Roger Leigh via Boost < boost@lists.boost.org> wrote:

On 18/09/18 16:43, Zach Laine via Boost wrote:

...

On Tue, Sep 18, 2018 at 10:39 AM Alexander Grund via Boost < boost@lists.boost.org> wrote:

...

...

...

E.g. "yap", "hana" and "callable_traits" each try to create a target "check".

What's wrong with that (not upset, just curious)? They conflict with each other. Targets need to have unique names.

Ah, of course, thanks. Do you have a preferred way to address this? F.ex., I only care about the "check" target in independent builds of Yap, mainly for Yap-only Travis CI. I could always turn it on conditional on a CI configuration variable I cold define.

One possibility:

if(NOT TARGET check) create_custom_target(check DEPENDS foo-check) else() add_dependencies(check foo-check) endif()

The top-level CMakeLists.txt would need to define an empty check target up front which the else() case will add the local check target to.

I like this. Louis and Barrett, if I were to make a PR for each of your repos, would you find this acceptable too? Zach

On Tue, Sep 18, 2018 at 12:23 PM Zach Laine wrote:

On Tue, Sep 18, 2018 at 10:52 AM Roger Leigh via Boost < boost@lists.boost.org> wrote:

...

On 18/09/18 16:43, Zach Laine via Boost wrote:

...

On Tue, Sep 18, 2018 at 10:39 AM Alexander Grund via Boost < boost@lists.boost.org> wrote:

...

...

...

E.g. "yap", "hana" and "callable_traits" each try to create a target "check".

What's wrong with that (not upset, just curious)? They conflict with each other. Targets need to have unique names.

Ah, of course, thanks. Do you have a preferred way to address this? F.ex., I only care about the "check" target in independent builds of Yap, mainly for Yap-only Travis CI. I could always turn it on conditional on a CI configuration variable I cold define.

One possibility:

if(NOT TARGET check) create_custom_target(check DEPENDS foo-check) else() add_dependencies(check foo-check) endif()

The top-level CMakeLists.txt would need to define an empty check target up front which the else() case will add the local check target to.

I like this. Louis and Barrett, if I were to make a PR for each of your repos, would you find this acceptable too?

Thanks for the PR. Merged. Louis

Zach

On 18/09/18 17:47, Peter Dimov via Boost wrote:

Roger Leigh wrote:

...

One possibility:

if(NOT TARGET check)    create_custom_target(check DEPENDS foo-check) else()    add_dependencies(check foo-check) endif()

Sounds good.

How would this interact with enable_testing() and add_test()?

Since "check" is a non-standard target added by individual components, it shouldn't interact with enable_testing() and add_test() at all as far as I can see. (If the dependencies of the check target aren't built by default (ALL) and are also used by add_test() then that might be a potential problem, but I'd then question what the "check" target was really for in the first place… since testing is usually done by running ctest and not building a target.)

On 18/09/18 20:58, Zach Laine via Boost wrote:

On Tue, Sep 18, 2018 at 2:53 PM Roger Leigh via Boost wrote:

...

On 18/09/18 17:47, Peter Dimov via Boost wrote:

...

...

How would this interact with enable_testing() and add_test()?

Since "check" is a non-standard target added by individual components, it shouldn't interact with enable_testing() and add_test() at all as far as I can see. (If the dependencies of the check target aren't built by default (ALL) and are also used by add_test() then that might be a potential problem, but I'd then question what the "check" target was really for in the first place… since testing is usually done by running ctest and not building a target.)

It exists because many, many users are used to building a library in these steps:

./configure make make check make install

And with a check target, you get something eerily similar:

cmake .. make make check make install

CMake already provides two mechanisms for running tests: 1) ctest (the recommended way) 2) "make test" (when using the "Unix Makefiles" generator) Doing it a third way which is non-standard for CMake is the least useful choice. I agree it's superficially compatible with the Automake check/check-TESTS targets and it does have developer familiarity. However, while common it is tool-specific and it's not necessarily great for getting people familiar with the best practices when using CMake. Using ctest directly is vastly more flexible (none of its options are exposed via the target). It would be nice if you could alias "check" to "test" with add_custom_target(check DEPENDS test) but the test target is only emitted in the Makefile as boilerplate; it never exists internally as a CMake target. # Special rule for the target test test: @$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --cyan "Running tests..." /usr/bin/ctest --force-new-ctest-process $(ARGS) .PHONY : test You can set ARGS to pass options, but that's it. It's still generator-specific and non-standard. A custom check target doesn't even allow the flexibility of passing different options at runtime. Regards, Roger

As far as I understand, the problem with `make test` is that it doesn't build the tests. To build them, one does `make all test`, which (a) requires tests to be built as part of the "all" target - which we don't want - and (b) doesn't work with -jN. And `make all && ctest` has the problem of test executable build errors not being part of the ctest output.

To clarify, this understanding of mine comes from https://stackoverflow.com/questions/733475/cmake-ctest-make-test-doesnt-buil...

So the custom "check" target that just invokes `ctest` and depends on all test executables is a workaround for the above.

https://gitlab.kitware.com/cmake/community/wikis/doc/tutorials/EmulateMakeCh... What I still don't get however is that, by the look of it, this `make check` emulation will fail when one of the test executables fails to compile, thereby aborting the whole test procedure, which is not what we want.

Raffi Enficiaud wrote:

On 18.09.18 23:04, Peter Dimov via Boost wrote:

...

As far as I understand, the problem with `make test` is that it doesn't build the tests. To build them, one does `make all test`, which (a) requires tests to be built as part of the "all" target - which we don't want - and (b) doesn't work with -jN. And `make all && ctest` has the problem of test executable build errors not being part of the ctest output.

The way I understand it is that this is a make behaviour and not a cmake one: as soon as an error is detected, the goal cannot be reached and make stops. You change this behaviour by passing "-k" / "--keep-going"

Turns out that the "portable" cmake way to achieve the above is to make building the test executable a test. See https://github.com/pdimov/mincmake/blob/master/test/CMakeLists.txt https://github.com/pdimov/mincmake/blob/master/cmake/boost_test.cmake (and https://github.com/boostorg/assert/blob/develop/test/CMakeLists.txt) for an example implementation of the Boost test types. With this in place, "cmake --build . --target check" builds and runs the tests as expected: https://travis-ci.org/pdimov/mincmake/builds/431060394 This still won't support parallel testing properly if some of the libraries require building, which is probably fixable by also making building the library a test on which the rest depend, but I'm not investing any effort into this specific issue at the moment. Serial testing was good enough for going to the moon.

I read this conversion and thought that it might be the proper place and time to ask this question (sorry, if it's not). Is there any chance that Boost becomes more modular in a way if I need just a few libs (or only one), I am not forced to get all of the libs in boost? --- Kostia Kyiv, Ukraine On Fri, Sep 21, 2018 at 1:48 AM Raffi Enficiaud via Boost wrote:

On 20.09.18 23:51, Peter Dimov via Boost wrote:

...

Raffi Enficiaud wrote:

...

...

As far as I understand, the problem with `make test` is that it doesn't > build the tests. To build them, one does `make all test`, which (a) > requires tests to be built as part of the "all" target - which we don't > want - and (b) doesn't work with -jN. And `make all && ctest` has the > problem of test executable build errors not being

On 18.09.18 23:04, Peter Dimov via Boost wrote: part of the ctest > output.

The way I understand it is that this is a make behaviour and not a cmake one: as soon as an error is detected, the goal cannot be reached and make stops. You change this behaviour by passing "-k" / "--keep-going"

Turns out that the "portable" cmake way to achieve the above is to make building the test executable a test. See

https://github.com/pdimov/mincmake/blob/master/test/CMakeLists.txt https://github.com/pdimov/mincmake/blob/master/cmake/boost_test.cmake

(and https://github.com/boostorg/assert/blob/develop/test/CMakeLists.txt)

Yes I know well this technique, and there are other ways to achieve it. For instance, you can call ctest as a post-build of your test binary target, and you still benefit from the semantics offered by cmake for the tests (LABELS, FAILURE, etc), as your example does.

But you (and many others in the list) are focusing on the "test execution" part pulling the "test target compilation". IMO, those are different phases of the overall project, and this is a detail that each maintainer can address. As you show, there are ways to do it, and many examples on the Internet, and my taste would go on the "do-it-yourself" rather than "use-boost-cmake-tools". Personal taste, I know cmake.

What I personally *strongly* do not want is that a failure in a very remote component of boost affects the execution of my components of boost. Say in boost.graph a test fails - not even a compilation of the library -, and it turns out that those boost.graph tests are executed before boost.test builds.

In this case, b2 continues as deep as possible even if there is a failure (without -q), while make does not (without -k) - because the "all" target depends on ... all targets.

This is why I would also like people to focus more on isolation of the different libraries build/doc/test phases.

"Isolation" relates to "interfaces". The question I am trying to address is, what would be a good "build interface" between the boost libraries, that is modular, scales well and allows for isolation.

The design I suggest and have implemented has "packages" (boost libraries) and "components" (build, test, doc, ...), with very little semantics: - a component may have several targets (eg. static/shared of boost.X, aliases, etc), - and a component may depend on another component (tests of boost.X depends on build of boost.X).

This is very simple, this works, this scales. See for instance how quickbook dependencies works:

https://github.com/raffienficiaud/boost-cmake/blob/0d4c9d5ab6d8f3e2f2f10087b...

and how it is used in the build:

https://github.com/raffienficiaud/boost-cmake/blob/0d4c9d5ab6d8f3e2f2f10087b...

and cherry on top, we can then use quickbook like this, which requires in turn a globally available tool: https://github.com/raffienficiaud/boost-cmake/blob/0d4c9d5ab6d8f3e2f2f10087b...

For this globally available tool, this is ok to have it in boost/tools. For being able to build a library, I believe this is a showstopper to depend on a boost/tool, as it will kill a lot of modularisation efforts.

...

for an example implementation of the Boost test types. With this in place, "cmake --build . --target check" builds and runs the tests as expected:

https://travis-ci.org/pdimov/mincmake/builds/431060394

Nice. I need to devote more time to Travis as well. I have ~170 tests so far, but some need tweaking like this:

https://github.com/raffienficiaud/boost-cmake/blob/0d4c9d5ab6d8f3e2f2f10087b...

...

This still won't support parallel testing properly if some of the libraries require building, which is probably fixable by also making building the library a test on which the rest depend, but I'm not investing any effort into this specific issue at the moment. Serial testing was good enough for going to the moon.

Well ... how many tests are running right now on all boost?

Raffi

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

On Wed, Sep 26, 2018 at 10:28 AM Stefan Seefeld via Boost < boost@lists.boost.org> wrote:

Personally I'd go as far as suggesting that it be a requirement for any new build tool proposal that individual projects be compilable (and installable) individually.

I should mention.. that there's nothing in B2 that prevents that modularity. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net

On Wed, Sep 26, 2018 at 11:26 AM Stefan Seefeld wrote:

Hi Rene,

On 2018-09-26 02:16 PM, Rene Rivera wrote:

On Wed, Sep 26, 2018 at 10:28 AM Stefan Seefeld via Boost < boost@lists.boost.org> wrote:

...

Personally I'd go as far as suggesting that it be a requirement for any new build tool proposal that individual projects be compilable (and installable) individually.

I should mention.. that there's nothing in B2 that prevents that modularity.

I'm not exactly sure what you mean by "prevent", but as far as I know, modular builds (at least the variant as I describe it in my other reply) aren't supported by Boost.Build right now. I'd be more than happy to be proven wrong.

(I know we have gone over exactly this question years ago, when you told me that it could be done with a bit of adjustment to the Boost.Build internal code. Has anything changed since then ?)

They are prevented by the Boost building scripts, i.e. boost-root/jamroot. If Boost was willing to be truly modular it would be a fairly simple matter to make each Boost library it's own B2 project (i.e with their own jamroot) and import dependencies with the usual B2 "using" declaration you would easily get individual builds (which like cmake would also force dealing with cycles). That would be at least equivalent to the cmake "add_subdirectory" suggestions (although it would give you a bit more). About the only "new" piece of functionality needed, like cmake, would be to Boost define a standard for what installed Boost library modules means. With such a definition it would then also be easy to also account for externally installed modular Boost library with B2. Although we do already have part of that in the B2 equivalent of FindBoost < https://github.com/boostorg/build/blob/develop/src/contrib/boost.jam> (note that the file is 10 years old). Hence, there is nothing new here. It's always been possible for Boost to be modular without cmake. And ultimately it's not a question of what build system Boost uses.. It's always solely been a question of the structure of Boost. Sorry if that sounds like a rant. But I tire of people placing blame at B2's feet. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net

On 26.09.18 18:36, Kostiantyn Ponomarenko via Boost wrote:

I read this conversion and thought that it might be the proper place and time to ask this question (sorry, if it's not). Is there any chance that Boost becomes more modular in a way if I need just a few libs (or only one), I am not forced to get all of the libs in boost?

Hi, From your user perspective, how do you want this implemented? (just a question). With the changes I am proposing (sorry for self promotion, see https://github.com/raffienficiaud/boost-cmake), I can: * create a cmake build tree only for a subset of boost. Say you want to compile only library X that depends on other libraries Y and Z, and only X,Y and Z will be added to the cmake project. This is already in place. * create a stub from cmake that automatically checks out the required dependencies. This needs to be added, but should be easy to do. Is this second use case a good scenario for you? Raffi

--- Kostia Kyiv, Ukraine

On Fri, Sep 21, 2018 at 1:48 AM Raffi Enficiaud via Boost wrote:

...

On 20.09.18 23:51, Peter Dimov via Boost wrote:

...

Raffi Enficiaud wrote:

...

On 18.09.18 23:04, Peter Dimov via Boost wrote:

...

[snip all the rest]

On 26.09.18 20:09, Stefan Seefeld via Boost wrote:

On 2018-09-26 01:59 PM, Raffi Enficiaud via Boost wrote:

...

* create a cmake build tree only for a subset of boost. Say you want to compile only library X that depends on other libraries Y and Z, and only X,Y and Z will be added to the cmake project. This is already in place. * create a stub from cmake that automatically checks out the required dependencies. This needs to be added, but should be easy to do.

Is this second use case a good scenario for you?

A use-case I'd like to see supported:

* allow an individual project to be built against prerequisite boost libraries that are pre-installed on the system (no matter whether that installation was done manually or using some system package management, as is common on Linux).

This is a precondition for considering Boost project repos as truly independent, as the two use-cases you suggest above would still imply a dependency on the source repo level, rather than the package level.

I am sure if I dig deep in the mail archive, I will find some details about what you need, right now it is not very clear to me. So, is this what you want: 1. you do eg a sudo apt-get boost-X 2. you clone boost-Y that requires boost-X 3. you develop your local clone of boost-Y that links to boost-X installed on your system Is that what you are describing? If not, you can stop reading... In that case, we need a versioning. But I believe this is a bad practice. I do not know if I should elaborate... IMO it will make the life of developers really hard for the following reasons: 1. First of all, the dependencies change over time: for the system libraries you would have for instance X<-Y<-Z and for the local clone you may have X<-Q<-Z. This ends up in weird scenarios: you checked out only Y and Z, but you need Q and not Y. 2. Imagine you have for library X<-Y<-Z again, and you work on X and Z. You may have then 2 copies of "Z" (system + clone) with different versions. We can say that one takes precedence over the other one, but still you will end-up in an inconsistent chain of dependencies, and the developer can silently source/link the wrong "Y" or "Z" ("Y" on system links to user "Z", etc) The root of the problem with such an approach is that the user and system packages are interleaved, while there should be only one arrow on the dependencies: user clone should depend on system installation, system installation should never see user clone. The benefits of having a full clone (or a consistent subset) are much higher. Raffi

Stefan Seefeld wrote:

Why are you making things so complicated ?

Because they are, by nature. If you already have Boost installed, you already have the library. Since you want to install that same library, it must be somehow different than the one already installed. Hence, it's a different version from the installed Boost, and consequently if code using the two is linked together, Things Will Be Happening.

Stefan Seefeld wrote:

I do not want to install the library (the one I'm working on), at least not while I'm developing it. What I'm asking is to be able to *build* (read: develop) it stand-alone.

OK... but in this case, who are you asking for support? What do you want done?

Stefan Seefeld wrote:

...

* allow an individual project to be built against prerequisite boost libraries that are pre-installed on the system (no matter whether that installation was done manually or using some system package management, as is common on Linux).

which as of now isn't supported by B2...

Isn't this a matter of you writing an appropriate Jamroot for standalone use?

Stefan Seefeld wrote:

On 2018-09-26 05:30 PM, Peter Dimov via Boost wrote:

...

Stefan Seefeld wrote:

...

...

* allow an individual project to be built against prerequisite boost libraries that are pre-installed on the system (no matter whether that installation was done manually or using some system package management, as is common on Linux).

which as of now isn't supported by B2...

Isn't this a matter of you writing an appropriate Jamroot for standalone use?

Believe me, if I knew how to do it, I had done it already. I did discuss with Rene a few years ago.

I'm sure you would have; I'm just trying to determine the specific obstacle we're facing. What do we need to add to Boost.Build?

On 9/26/18 7:55 PM, Rene Rivera via Boost wrote:

On Wed, Sep 26, 2018 at 2:40 PM Peter Dimov via Boost wrote:

...

Stefan Seefeld wrote:

...

On 2018-09-26 05:30 PM, Peter Dimov via Boost wrote:

...

Stefan Seefeld wrote:

...

...

* allow an individual project to be built against prerequisite boost libraries that are pre-installed on the system (no matter whether that installation was done manually or using some system package management, as is common on Linux). which as of now isn't supported by B2... Isn't this a matter of you writing an appropriate Jamroot for standalone use? Believe me, if I knew how to do it, I had done it already. I did discuss with Rene a few years ago. I'm sure you would have; I'm just trying to determine the specific obstacle we're facing. What do we need to add to Boost.Build?

All that's needed is to add a Jamroot to your lib-root that defines your dependencies. For example, for BPL, it could look like this:

=== import boost ; boost.use-project ; project : requirements <use>/boost//bind <use>/boost//config <use>/boost//conversion <use>/boost//core <use>/boost//detail <use>/boost//foreach <use>/boost//function <use>/boost//iterator <use>/boost//lexical_cast <use>/boost//mpl <use>/boost/numeric//conversion <use>/boost//preprocessor <use>/boost//smart_ptr <use>/boost//tuple <use>/boost//type_traits <use>/boost//utility ; ===

WARNING: This is entirely off the cuff code that I'm writing at CppCon while listening to a presentation at the end of the day :-(

This **only** gives you the ability to do stuff in your library with whatever the default detect system Boost is. Which means that as soon as we do that your lib is no longer able to use the sibling / integrated / monolithic Boost cloned tree. And perhaps this is the real problem that Stefan is having. That it's not possible to currently implement a b2 project file that can support both modular and monolithic *at the same time*. But this is software, and hence anything is ultimately possible. Is it something that is worth supporting.. maybe? or maybe not?

Indeed: While the above is a great starting point to support what I want, I don't want to loose the ability for BPL to be built (and tested) as part of the rest of Boost. I would thus prefer something that gives me the ability to choose the build strategy e.g. using some command-line option, including flag to be passed to the build (e.g., `b2 ... standalone=on`), rather than having to modify the (build) code. Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...

Stefan Seefeld wrote:

...

Indeed: While the above is a great starting point to support what I want, I don't want to loose the ability for BPL to be built (and tested) as part of the rest of Boost. I would thus prefer something that gives me the ability to choose the build strategy e.g. using some command-line option, including flag to be passed to the build (e.g., `b2 ... standalone=on`), rather than having to modify the (build) code.

Scanning for Jamroot to identify the project root is fundamental to Jam (the Perforce one). We could perhaps add something like

b2 --jamroot=.

that would act as if there were an empty Jamroot at . (or equivalently if the Jamfile at . were named Jamroot.)

Apart from that, since the existing installations of Boost aren't modular and are installed in the system-global directory /usr, there's not even need to enumerate the dependencies accurately or to add an include path. You could if you wanted to, I suppose. Python, specifically, only uses header-only libraries, so you don't even need to bother with library targets.

The only thing that will not work is this: import ../../config/checks/config : requires ; (https://github.com/boostorg/python/blob/develop/test/Jamfile#L7)

On Thu, Sep 27, 2018 at 1:29 AM Mateusz Loskot via Boost < boost@lists.boost.org> wrote:

...

Stefan Seefeld wrote:

...

Indeed: While the above is a great starting point to support what I want, I don't want to loose the ability for BPL to be built (and tested) as

...

...

of the rest of Boost. I would thus prefer something that gives me the ability to choose the build strategy e.g. using some command-line

On Thu, 27 Sep 2018 at 05:18, Peter Dimov via Boost wrote: part option,

...

...

including flag to be passed to the build (e.g., `b2 ... standalone=on`), rather than having to modify the (build) code.

Scanning for Jamroot to identify the project root is fundamental to Jam (the Perforce one). We could perhaps add something like

b2 --jamroot=.

that would act as if there were an empty Jamroot at . (or equivalently if the Jamfile at . were named Jamroot.)

Does it mean there could be two jamfile-s? libs/gil/Jamroot for standalone build libs/gil/Jamfile for in-Boost tree build and the latter is simply ignored in normal b2 run, ie. without --jamroot=.

No. Such a theoretical feature would use the existing Jamfile at the location as the jamroot. I think, this would be also very helpful to apply the Jamroot trick [1],

recently presented to me by Steven Watanabe, to significantly speed up b2 startup procedures:

But, yes, it would allow for that trick. In general I think the boost-root/Jamroot does way-way too much. If Boost where restructure to be modular only most of that work would unnecessary. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net

Am I the only one who thinks this has gotten completely off-topic?

-----Original Message----- From: Boost On Behalf Of Mateusz Loskot via Boost Sent: Thursday, September 27, 2018 5:01 PM To: boost@lists.boost.org Cc: Mateusz Loskot Subject: Re: [boost] [cmake] Pull request announcement

On Thu, 27 Sep 2018 at 16:57, Rene Rivera wrote:

...

On Thu, Sep 27, 2018 at 1:29 AM Mateusz Loskot via Boost wrote:

...

On Thu, 27 Sep 2018 at 05:18, Peter Dimov via Boost wrote:

...

Stefan Seefeld wrote:

...

Indeed: While the above is a great starting point to support what I want, I don't want to loose the ability for BPL to be built (and tested) as part of the rest of Boost. I would thus prefer something that gives me the ability to choose the build strategy e.g. using some command-line option, including flag to be passed to the build (e.g., `b2 ... standalone=on`), rather than having to modify the (build) code.

Scanning for Jamroot to identify the project root is fundamental to Jam (the Perforce one). We could perhaps add something like

b2 --jamroot=.

that would act as if there were an empty Jamroot at . (or equivalently if the Jamfile at . were named Jamroot.)

Does it mean there could be two jamfile-s? libs/gil/Jamroot for standalone build libs/gil/Jamfile for in-Boost tree build and the latter is simply ignored in normal b2 run, ie. without --jamroot=.

No. Such a theoretical feature would use the existing Jamfile at the location as the jamroot.

...

I think, this would be also very helpful to apply the Jamroot trick [1], recently presented to me by Steven Watanabe, to significantly speed up b2 startup procedures:

But, yes, it would allow for that trick. In general I think the boost-root/Jamroot does way-way too much. If Boost where restructure to be modular only most of that work would unnecessary.

That sounds compelling enough to me :-) Thanks for the clarification

-- Mateusz Loskot, http://mateusz.loskot.net

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

-----Original Message----- From: Boost On Behalf Of Mateusz Loskot via Boost Sent: Thursday, September 27, 2018 8:27 PM

On Thu, 27 Sep 2018 at 20:23, Mike Dev via Boost wrote:

...

Am I the only one who thinks this has gotten completely off-topic?

I confess, I've lost the track of the topic of this thread long time ago.

My apology, I'll stop trashing it further.

No problem I just thought that some of the discussions (and particular the information surfacing as part of them) deserve their own thread, which would make it easier to follow them and look them up later. Best Mike

Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

On 2018-09-26 05:02 PM, Raffi Enficiaud via Boost wrote:

Sure, but you do not see the dependency graph in its full extent, and this is a good thing: you want to see your direct dependencies, because those are the ones that you directly manage.

Right.

As soon as you (or any tool) start unrolling the chains of dependencies, it may becomes messy. By taking the previous examples -- very simple I have to say -- you may not even be aware of the 2nd copy of library "Z".

To avoid to get into grid-lock in the discussion (similar to Peter's argument) I'm assuming that each library manages its own dependencies, and "installing a library" will imply resolving them. And indeed, if you install any Boost library (apt-get, dnf, vcpkg, ...) these dependencies will have been taken care of by some upstream package manager. So let's consider this a resolved issue.

...

All I want is the ability to work on a Boost library project, compiling it against some other prerequisite (versioned) libraries, some of which may be part of Boost.

Yes, I understand.

Do you mind having the ones that are part of boost - but only the ones you need for working on your library - checked out/cloned on request?

I do. That's precisely the difference between your second use-case and the one I added, and I have explained what additional situation that would cover that aren't covered by the earlier ones. [...]

What I was saying before is the following. If I know that libX depends on libY, I can make it such that by cloning or setting up libX, I would like to get libY automatically.

Yes, I understand. I think that's a fine use-case to support, but it shouldn't be the only one.

If I need external libraries, I can specify them explicitly from the command line, or use facilities such as the cmake's find_package to get them from the OS or any other location.

I am fine with:

cmake -DLIB_GIL=ON -DLIB_JPEG_LOCATION=/some/path/on/disk

Yeah, and that's all I'm asking, only that rather than libjpeg I'd also like to be able to find Boost dependencies (e.g., filesystem, core, etc.) that way. Stefan -- ...ich hab' noch einen Koffer in Berlin...

On 9/26/2018 2:29 PM, Raffi Enficiaud via Boost wrote:

On 26.09.18 20:09, Stefan Seefeld via Boost wrote:

...

On 2018-09-26 01:59 PM, Raffi Enficiaud via Boost wrote:

...

* create a cmake build tree only for a subset of boost. Say you want to compile only library X that depends on other libraries Y and Z, and only X,Y and Z will be added to the cmake project. This is already in place. * create a stub from cmake that automatically checks out the required dependencies. This needs to be added, but should be easy to do.

Is this second use case a good scenario for you?

A use-case I'd like to see supported:

* allow an individual project to be built against prerequisite boost libraries that are pre-installed on the system (no matter whether that installation was done manually or using some system package management, as is common on Linux).

This is a precondition for considering Boost project repos as truly independent, as the two use-cases you suggest above would still imply a dependency on the source repo level, rather than the package level.

I am sure if I dig deep in the mail archive, I will find some details about what you need, right now it is not very clear to me.

So, is this what you want:

1. you do eg a sudo apt-get boost-X 2. you clone boost-Y that requires boost-X 3. you develop your local clone of boost-Y that links to boost-X installed on your system

Is that what you are describing? If not, you can stop reading...

In that case, we need a versioning. But I believe this is a bad practice. I do not know if I should elaborate...

IMO it will make the life of developers really hard for the following reasons:

1. First of all, the dependencies change over time: for the system libraries you would have for instance X<-Y<-Z and for the local clone you may have X<-Q<-Z. This ends up in weird scenarios: you checked out only Y and Z, but you need Q and not Y. 2. Imagine you have for library X<-Y<-Z again, and you work on X and Z. You may have then 2 copies of "Z" (system + clone) with different versions. We can say that one takes precedence over the other one, but still you will end-up in an inconsistent chain of dependencies, and the developer can silently source/link the wrong "Y" or "Z" ("Y" on system links to user "Z", etc)

The root of the problem with such an approach is that the user and system packages are interleaved, while there should be only one arrow on the dependencies: user clone should depend on system installation, system installation should never see user clone.

The benefits of having a full clone (or a consistent subset) are much higher.

It can work without versioning when you have library X which depends on library Y, and both libraries come from Boost 1.xx, and no other library X or library Y can be picked up at either the compilation, and/or linking, and/or run-time phase. But what happens when you have library X from Boost 1.xx and library Y from Boost 1.yy ? There is no guarantee that they will work together without some sort of versioning system. Add to that the fact that the placement of library X and library Y may be anywhere on the end-user's hard drive, and that that their may be multiple releases of library X and library Y in different places on the end-user's hard drive, and that a typical dependency graph involves much more than just library X and library Y; I do not think such a realistic scenario can possible work without some sort of versioning system.

Raffi

On Wed, Sep 26, 2018 at 5:02 PM Raffi Enficiaud via Boost < boost@lists.boost.org> wrote:

On 26.09.18 21:49, Edward Diener via Boost wrote:

...

On 9/26/2018 2:29 PM, Raffi Enficiaud via Boost wrote:

...

On 26.09.18 20:09, Stefan Seefeld via Boost wrote:

...

On 2018-09-26 01:59 PM, Raffi Enficiaud via Boost wrote:

...

* create a cmake build tree only for a subset of boost. Say you want to compile only library X that depends on other libraries Y and Z, and only X,Y and Z will be added to the cmake project. This is already in place. * create a stub from cmake that automatically checks out the required dependencies. This needs to be added, but should be easy to do.

Is this second use case a good scenario for you?

A use-case I'd like to see supported:

* allow an individual project to be built against prerequisite boost libraries that are pre-installed on the system (no matter whether that installation was done manually or using some system package management, as is common on Linux).

This is a precondition for considering Boost project repos as truly independent, as the two use-cases you suggest above would still imply a dependency on the source repo level, rather than the package level.

I am sure if I dig deep in the mail archive, I will find some details about what you need, right now it is not very clear to me.

So, is this what you want:

1. you do eg a sudo apt-get boost-X 2. you clone boost-Y that requires boost-X 3. you develop your local clone of boost-Y that links to boost-X installed on your system

Is that what you are describing? If not, you can stop reading...

In that case, we need a versioning. But I believe this is a bad practice. I do not know if I should elaborate...

IMO it will make the life of developers really hard for the following reasons:

1. First of all, the dependencies change over time: for the system libraries you would have for instance X<-Y<-Z and for the local clone you may have X<-Q<-Z. This ends up in weird scenarios: you checked out only Y and Z, but you need Q and not Y. 2. Imagine you have for library X<-Y<-Z again, and you work on X and Z. You may have then 2 copies of "Z" (system + clone) with different versions. We can say that one takes precedence over the other one, but still you will end-up in an inconsistent chain of dependencies, and the developer can silently source/link the wrong "Y" or "Z" ("Y" on system links to user "Z", etc)

The root of the problem with such an approach is that the user and system packages are interleaved, while there should be only one arrow on the dependencies: user clone should depend on system installation, system installation should never see user clone.

The benefits of having a full clone (or a consistent subset) are much higher.

It can work without versioning when you have library X which depends on library Y, and both libraries come from Boost 1.xx, and no other library X or library Y can be picked up at either the compilation, and/or linking, and/or run-time phase.

Or even at setup before build phase. I am a small step behind the cmake -DWITH_TEST=ON -DAUTO_CLONE_DEPENDENCIES=ON $boost_folder

You do not need a full clone, just a consistent subset that is being pulled from what you need. And this should work.

... but here, you have indeed an implicit versioning that is the current repo or superproject GIT version/hash. You want library X checked out from rev R of the superproject. You will clone all the dependencies of libX that were at the same rev R of the superproject (or higher, actually I do not know...).

...

But what happens when you have library X from Boost 1.xx and library Y from Boost 1.yy ? There is no guarantee that they will work together without some sort of versioning system.

This is also what I am saying, I believe we agree on this.

...

Add to that the fact that the placement of library X and library Y may be anywhere on the end-user's hard drive, and that that their may be multiple releases of library X and library Y in different places on the end-user's hard drive, and that a typical dependency graph involves much more than just library X and library Y; I do not think such a realistic scenario can possible work without some sort of versioning system.

Exactly. And even with packaging tools such as pip, brew or conan, this causes some headaches sometimes.

Hi all, I also agree that mixing a libraries from boost 1.xx with boost 1.yy should simply be discouraged. I also find promising the idea of making a step before invoking CMake to configure the modules. Supose one wants to build the libraries xxx and yyy: # step 1 # get boost root git clone -b develop https://github.com/boostorg/boost.git boost-root cd boost-root # step 2 # Execute a script that executes `git submodule init ` for # libs/xxx, libs/yyy and all submodules that they depend on: ./init_submodules_dependencies.sh libs/xxx libs/yyy # step 3 # Configure the build system to work only with this modules: # ( and their dependencies ) ./select_cmake_modules.sh libs/xxx libs/yyy # step 4: use the build system to generate, build, install, test, etc This looks so simple and effective to me. Does anyone disagree with this approach ? Note that instead of step 1 and 2 one could simple download and unpack the boost tarball.

It is very sad that cmake doesn't (AFAIK) support directory scoped target names.

It's unnecessary if you're using cmake right.

Conditional activation is not ideal either, because at some point someone probably wants to build all of boost and run all the tests together (e.g. release / integration tests).

In which case have cmake execute cmake as a custom build command to build the ctest database, and then run ctest separately.

Prefixing everything with boost_<lib>_<target> is probably the cleanest solution but annoying to type.

Not sure if there is a better solution

The cmake idiomatic solution is to export those cmake targets from each library which can be externally consumed, during which you can prefix them with a namespace. Stephen Kelly outlined all this on here many years ago. And he would certainly know. Search the boost-dev archives. Niall

mike wrote:

Anyway, this is precisely the reason, why my PRs try to stay clear of anything more advanced than simple target declaration, static builds and dependency information: Too many possibilities and opinions.

I still don't understand your insistence on building static libraries when BUILD_SHARED_LIBS is ON. I mean, you have to go out of your way to override the default behavior (and the user's wish) to do so. What is your reasoning here?

...

...

It is very sad that cmake doesn't (AFAIK) support directory scoped target names.

It's unnecessary if you're using cmake right.

I've stopped trying to discern, what is the "right way" to use cmake, but I'm Sure it doesn't hurt to be robust against a common way to use cmake.

Not using import and export of targets is bad cmake, period. It doesn't scale well, causes maintenance headaches, and most importantly of all, it's trivially easy to avoid. You have two choices, low fat and high fat. Low fat is https://cmake.org/cmake/help/latest/command/export.html. I generally write a bit of cmake which extracts the targets in the default build set, and feeds them to export(TARGETS targets... NAMESPACE libname ...). This generates an exported build targets file for exactly one build configuration. This can be implemented within ten minutes. Trivially easy. High fat involves creating an install layout i.e. staging using https://cmake.org/cmake/help/latest/command/install.html install(TARGETS). You do this for each build configuration you want. You then use install(EXPORT) to generate a combined exported build targets file for many build configurations at once. This takes quite a bit more effort, I have taken days on this when converting some large projects. This is the correct way to use cmake. If you're not doing one of the above, you're doing it wrong.

...

The cmake idiomatic solution is to export those cmake targets from each library which can be externally consumed, during which you can prefix them with a namespace.

Doing it that way, Is there an easy way to ensure the same compiler flags and definitions are used for all compilations?

INTERFACE_COMPILE_OPTIONS, INTERFACE_COMPILE_DEFINITIONS. Use generator expression $ and $ to control what gets used at which stage.

Or the reverse: Is there a way to communicate the available options from the sub-libraries to the super-project (e.g. for use in ccmake)? So far I've only used this style with bigger groups of applications/frameworks not on a per-library basis (especially with a tightly coupled group of libraries like boost).

Once a cmake build group is exported, it is written in stone. That's what makes doing this scalable - the consuming build system can treat it as a black box.

Anyway, this is precisely the reason, why my PRs try to stay clear of anything more advanced than simple target declaration, static builds and dependency information: Too many possibilities and opinions. This is something I'm happy to leave to the cmake professionals and the boost review process

What you're not getting here is that anything you submit which doesn't follow cmake idiomatic practice is highly likely to get rejected during review. So you're wasting your time, unless this is for personal use only.

...

Stephen Kelly outlined all this on here many years ago. And he would certainly know. Search the boost-dev archives.

Could you be a bit more specific? There is a lot of cmake related discussion in the archives. Was that part of this thread: https://lists.boost.org/boost-build/2016/05/28637.php?

That looks about right. I was only talking to Steve last Monday in fact. He came to my talk to the Dublin C++ users group "Deterministic Disappointment". He was telling me he refuses to touch cmake ever again unless he's getting paid for it. A nasty addiction it was for him. Niall

-----Original Message----- From: Boost On Behalf Of James E. King III via Boost Sent: Wednesday, September 19, 2018 3:09 PM To: boost@lists.boost.org Cc: James E. King III ; Peter Dimov Subject: Re: [boost] [cmake] Pull request announcement

On Mon, Sep 17, 2018 at 12:02 PM Peter Dimov via Boost < boost@lists.boost.org> wrote:

...

Mike Dev wrote:

...

...

I'd like to see an example of how a CMake project using Boost (f.ex. System) would look like, assuming your proposed pull requests are accepted.

Sure. You can have a look at https://github.com/Mike-Devel/DummyBoostUsage

Thanks.

...

file(GLOB boost_lib_cmake_files libs/*/CMakeLists.txt)

I wonder what would happen if Hana, Yap or Beast are present. :-)

Absolutely, positively do NOT use GLOB for project build chains or for header or source discovery. There are well documented reasons not to do this. You must express the dependency chain in linear order in libs/CMakeLists.txt.

I would have the boostorg/CMakeLists.txt call:

add_subdirectories ( libs ) after it has included all the things libs depends on.

Then have libs/CMakeLists.txt call add_subdirectories in dependency order. That is the only way you can reasonably ensure a linear dependency chain.

I (believe) I know about the arguments to not use glob expressions for discovering sources. I happen to still be of the opposite opinion that globbing is still preferable to manually maintaining a large lists of source files, but as you can see, I don't use globbing in the library cmake files I propose. I see even less reason to avoid globbing when searching for cmake files in subdirectories of boost/libs, which it is in the top level PR, but I've no problem with changing that too [1]. What I don't understand however, is why the subdirectories need to be fed to cmake in dependency order. CMake is perfectly capable of resolving dependencies it sees in reverse order and I have a lot of projects that add the used library subdirectories in alphabetical order (or something completely different) and haven't encountered any problems. And what exactly do you mean by "a linear dependency chain"? Mike [1] I'd prefer to have discussions about the top level cmake file at the actual PR (https://github.com/boostorg/boost/pull/193), but two notes on globbing upfront: 1) Not globbing means that every time a new library adds a cmake file and wants to hook up to the project wide mechanism, a new PR to the top level project needs to be merged (in addition to any automatic update of the submodule). I can see that this can be seen as an advantage as well as a disadvantage, but wanted to call it out regardless. 2) Even if no globbing is being used, there should be a mechanism to only add a subdirectory if the folder/CMakeFile it is actually present (often I don't want to clone all of boost but just a defined subset). Maybe that makes some of the arguments against globbing moot.

- Jim

On Sun, 16 Sep 2018 at 20:20, Niall Douglas via Boost wrote:

...

Instead of reposting pros/cons here again to make it into yet another forgotten post, those who care and need it should create a table on wiki with clear structured comparison.

It seems to me that the putative review manager (Robert) should canvass for solutions (there appear to be at least three now), do a quick check of them to see what tradeoffs each has, and make that table summarising their different approaches. He then could take a quick straw poll of here to see which set of tradeoffs the community dislikes least, and encourage that author to get their solution ready for review by an agreed date.

Sure, why not, or just decide if all proposals are ready for review, then all could be reviewed (straw poll the order which one goes first, etc.)

Some have argued that all this is too confusing. It is in my opinion no different to any other Boost library. Somebody becomes willing to invest the time and effort to get a solution to a problem past review. Whichever successfully passes muster here is what we adopt. It's exactly the same as any other Boost library, and calling on the Steering Committee to declare a design by decree is the wrong way of doing this.

I see one aspect that makes CMake for Boost different and this resembles (or respects) voices like Stefan Seefeld's. That is, one of the critical questions to CMake submitters should be: Are you ready and willing to maintain CMake across Boost and help library maintainers interested to learn CMake for some extended period of time. Although such situation would be hard to deal with or even not sensible to consider, I can see it plausible, that if maintainer of CMake solution disappears, the whole solution may stale and be voted for complete removal, or replacment with new version that has active maintainer. That also makes CMake for Boost different than a library acceptance. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

On Sun, 16 Sep 2018 at 14:47, James E. King III via Boost < boost@lists.boost.org> wrote:

Therefore for it to be successful, tight integration with a package manager that supports Windows would be ideal - something that can provide libicu and the other boost dependencies automatically.

I'm not generalising the "other boost dependencies" (I would say MPIR (a GMP-drop-in) is deficient), but vcpkg https://github.com/Microsoft/vcpkg provides ICU (cross platform, win, nix, osx). It uses CMake and actually builds Boost (with ICU support) as well. degski -- *“If something cannot go on forever, it will stop" - Herbert Stein* *“No, it isn’t truth. Truth isn’t truth" - Rudolph W. L. Giuliani*

On 17/09/2018 06:12, Niall Douglas wrote:

...

...

Could you elaborate, why this would require separate/conflicting cmake files as opposed to having multiple different targets in a single file?

If I include your project's root CMake file that contains tests added by add_test(), then I cannot avoid those when running my own tests.

I don't think it's sufficiently useful, but there is a CONFIGURATIONS parameter on add_test which allows limiting tests to specific named configurations. Probably more useful is to condition it on some project-specific property that's defined when you want to run tests.

The way I've solved this in my cmake is to detect if the project is the root project or not. So if the base CMakeLists.txt in the build is mine, I compile in tests etc. If it is not, I assume I am being included as a dependency, and leave out lots of stuff.

That sounds simple enough when you're a single project being built directly or not. It doesn't sound like it would work when someone wants to run "all Boost tests", or if someone using your library as a dependency does indeed want to run your tests.

Alexander Grund wrote:

...

If I include your project's root CMake file that contains tests added by add_test(), then I cannot avoid those when running my own tests.

Usual action for this is to only add the tests folder if CMAKE_SOURCE_DIR==PROJECT_SOURCE_DIR (you are the project being build)

This sounds sensible. Out of curiosity, I looked at Hana and Yap as two examples that already have CMakeLists files and neither bothers to do so. Both assume that they are the project being built, and do not support add_subdirectory use.

Out of curiosity, I looked at Hana and Yap as two examples that already have CMakeLists files and neither bothers to do so. Both assume that they are the project being built, and do not support add_subdirectory use. What they do is they use "EXCLUDE_FROM_ALL" on all tests (checked Hana only). This way tests are only build when requested explicitly. There is a custom target called `tests` which builds all those tests. Hence you do `make tests` to build them. Not sure how well this is handled by "make test"/ctest though.

My proposal would be a check like `if(CMAKE_SOURCE_DIR==PROJECT_SOURCE_DIR OR BOOST_BUILD_TESTS OR BOOST_<LIBRARY> _BUILD_TESTS)`

The two options I see for running the tests are

The tests should not have any dependencies above the library dependencies. And even then they will be brought in by the "top-top-level" CMakeLists (that does add_subdirectory on all libraries) They can then be "just" used. The cleaner approach is conditionally including the "tests" folder. (Same for "docs" etc) Besides that: Don't use upward-references ("../some/foo") but rather e.g. "PROJECT_SOURCE_DIR" etc.

On 18/09/2018 00:47, Peter Dimov wrote:

Alexander Grund wrote:

...

The tests should not have any dependencies above the library dependencies.

It's very common for tests to have additional dependencies beyond what the library needs.

Tests will *always* have more dependencies than the library. If nothing else, they will depend on the test framework in addition to the library under test and its dependencies. Sometimes they might use some additional things beyond that as well, for convenience or to do appropriate reporting or diagnosis of failures.

On Mon, Sep 17, 2018 at 8:49 AM Peter Dimov via Boost wrote:

Alexander Grund wrote:

...

Besides that: Don't use upward-references ("../some/foo") but rather e.g. "PROJECT_SOURCE_DIR" etc.

How would that work?

Easiest thing to do here is in each project's CMakeLists.txt file, use a standard naming convention to hoist the public include directory up to the superproject. I would macro-ize the dependency definition and building of the dependency header list, like: in libs/uuid/CMakeLists.txt: boost_depends_on ( serialization throw_exception ) Which would in turn set: set ( BOOST_UUID_DEPINCS ${BOOST_SERIALIZATION_HEADERS} ${BOOST_THROW_EXCEPTION_HEADERS} ) (and if those had link libraries, which would be identified by another variable of some sort, it would add to the link library list needed) The project itself would set the variable BOOST_UUID_HEADERS as: set ( BOOST_UUID_HEADERS ${CMAKE_CURRENT_SOURCE_DIR}/include ${BOOST_UUID_DEPINCS} PARENT_SCOPE ) This defines BOOST_UUID_HEADERS one level up - in the "libs" project. This variable now contains all of the include paths needed build anything that depends on uuid. In each dependent project, adding the following allows for access to the headers: boost_depends_on ( uuid ) which as part of the macro would run: include_directories ( ${BOOST_UUID_HEADERS} ) And obviously back in uuid where it says "boost_depends_on" that also added include directories for serialization and throw_exception for the uuid build. Now here's the really neat thing: the list of subdirectories in libs/CMakeLists.txt MUST be in dependency order which FORCES you to remove any cyclic dependencies in the overall project, which is a big plus! - Jim

On Thu, Sep 20, 2018 at 4:43 AM Alexander Grund via Boost < boost@lists.boost.org> wrote:

...

Easiest thing to do here is in each project's CMakeLists.txt file, use a standard naming convention to hoist the public include directory up to the superproject. I would macro-ize the dependency definition and building of the dependency header list, like:

in libs/uuid/CMakeLists.txt:

boost_depends_on ( serialization throw_exception )

Which would in turn set:

set ( BOOST_UUID_DEPINCS ${BOOST_SERIALIZATION_HEADERS} ${BOOST_THROW_EXCEPTION_HEADERS} ) (and if those had link libraries, which would be identified by another variable of some sort, it would add to the link library list needed)

The project itself would set the variable BOOST_UUID_HEADERS as:

set ( BOOST_UUID_HEADERS ${CMAKE_CURRENT_SOURCE_DIR}/include ${BOOST_UUID_DEPINCS} PARENT_SCOPE )

This defines BOOST_UUID_HEADERS one level up - in the "libs" project. This variable now contains all of the include paths needed build anything that depends on uuid.

In each dependent project, adding the following allows for access to the headers:

boost_depends_on ( uuid )

which as part of the macro would run:

include_directories ( ${BOOST_UUID_HEADERS} ) Definitely NOT. What you describe here is already part of modern CMake: You use targets:

add_library(boost_serialization INTERFACE) target_include_directories(boost_serialization INTERFACE include) # CMAKE_CURRENT_SOURCE_DIR is implicit

and in uuid: add_library(boost_uuid src/mysrc.cpp) target_include_directories(boost_uuid PUBLIC include) target_link_libraries(boost_uuid PUBLIC boost_serialization)

This basically does the same as your macros and stuff but in the modern CMake way where CMake resolves all dependencies. For cases where you may not have e.g. the boost_serialization target yet one may assume to be run as part of the super-boost project and do this instead:

add_library(boost_uuid src/mysrc.cpp) target_include_directories(boost_uuid PUBLIC include ${CMAKE_SOURCE_DIR}/libs/boost_serialization/include)

Or better: Define the target yourself: if(NOT TARGET boost_serialization) add_library(boost_serialization IMPORTED) ...

Excellent, thanks for that info! - Jim