[mpl] [build] [testing] Merge pull requests
Hi, There were a lot of discussions about MPL recently, and I created a few pull requests to address issues: https://github.com/boostorg/boost/pull/39 Invoke 'headers' target before building testing tools. Note that several testers are currently failing to run because of this problem. Since this affects the testing process, I'd appreciate if someone familiar with it could take a look. https://github.com/boostorg/boost/pull/38 Add MPL.Core tests to the test report. These tests are currently not run by testers. https://github.com/boostorg/mpl/pull/12 Move a misplaced header from MPL to MPL.Core to resolve circular dependency between the two. I believe the first PR is especially critical. Could someone with permissions merge these? Thanks.
On 9/17/2014 2:39 PM, Andrey Semashev wrote:
Hi,
There were a lot of discussions about MPL recently, and I created a few pull requests to address issues:
https://github.com/boostorg/boost/pull/39
Invoke 'headers' target before building testing tools. Note that several testers are currently failing to run because of this problem. Since this affects the testing process, I'd appreciate if someone familiar with it could take a look.
https://github.com/boostorg/boost/pull/38
Add MPL.Core tests to the test report. These tests are currently not run by testers.
The first two are not MPL changes so somebody else needs to merge them.
https://github.com/boostorg/mpl/pull/12
Move a misplaced header from MPL to MPL.Core to resolve circular dependency between the two.
The last is for MPL so I merged this pull request.
I believe the first PR is especially critical. Could someone with permissions merge these?
On Sep 17, 2014, at 12:39 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
There were a lot of discussions about MPL recently, and I created a few pull requests to address issues:
I’d like to revert the MPL PR that caused our testing problems, and the one Edward just applied. It was wrong of me to commit the PR without any testing. There’s been some list discussion but my overall concern is that we can’t clearly explain why some develop testers are broken and others are working okay. The commit completely broke Darwin, Debian, FreeBSD, QNX and Windows. But what’s really worrying is that some RHEL testers are still merrily cycling while other RHEL testers are broken. Same for Ubuntu, some are cycling okay while some are broken. I think it’s too risky to apply a patch when we, honestly I, don’t understand why this is happening. While we could apply Andrey’s PR (below) I’m afraid that may mask real underlying problems that we may not find out about until this hits master, or worse, users. I’ll work with Andrey to get the MPL changes applied, perhaps in smaller PRs. Please let me know if you have concerns. — Noel
https://github.com/boostorg/boost/pull/39
Invoke 'headers' target before building testing tools. Note that several testers are currently failing to run because of this problem. Since this affects the testing process, I'd appreciate if someone familiar with it could take a look.
https://github.com/boostorg/boost/pull/38
Add MPL.Core tests to the test report. These tests are currently not run by testers.
https://github.com/boostorg/mpl/pull/12
Move a misplaced header from MPL to MPL.Core to resolve circular dependency between the two.
I believe the first PR is especially critical. Could someone with permissions merge these?
Thanks.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 9/18/2014 11:54 PM, Belcourt, Kenneth wrote:
On Sep 17, 2014, at 12:39 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
There were a lot of discussions about MPL recently, and I created a few pull requests to address issues:
I’d like to revert the MPL PR that caused our testing problems, and the one Edward just applied. It was wrong of me to commit the PR without any testing. There’s been some list discussion but my overall concern is that we can’t clearly explain why some develop testers are broken and others are working okay. The commit completely broke Darwin, Debian, FreeBSD, QNX and Windows. But what’s really worrying is that some RHEL testers are still merrily cycling while other RHEL testers are broken. Same for Ubuntu, some are cycling okay while some are broken. I think it’s too risky to apply a patch when we, honestly I, don’t understand why this is happening.
While we could apply Andrey’s PR (below) I’m afraid that may mask real underlying problems that we may not find out about until this hits master, or worse, users. I’ll work with Andrey to get the MPL changes applied, perhaps in smaller PRs.
Please let me know if you have concerns.
Is not the pull request just in the developer branch ? In that case you could leave the pull request as is and try to solve it without affecting the master branch at all.
On Sep 18, 2014, at 11:25 PM, Edward Diener <eldiener@tropicsoft.com> wrote:
On 9/18/2014 11:54 PM, Belcourt, Kenneth wrote:
On Sep 17, 2014, at 12:39 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
There were a lot of discussions about MPL recently, and I created a few pull requests to address issues:
I’d like to revert the MPL PR that caused our testing problems, and the one Edward just applied. It was wrong of me to commit the PR without any testing. There’s been some list discussion but my overall concern is that we can’t clearly explain why some develop testers are broken and others are working okay. The commit completely broke Darwin, Debian, FreeBSD, QNX and Windows. But what’s really worrying is that some RHEL testers are still merrily cycling while other RHEL testers are broken. Same for Ubuntu, some are cycling okay while some are broken. I think it’s too risky to apply a patch when we, honestly I, don’t understand why this is happening.
While we could apply Andrey’s PR (below) I’m afraid that may mask real underlying problems that we may not find out about until this hits master, or worse, users. I’ll work with Andrey to get the MPL changes applied, perhaps in smaller PRs.
Please let me know if you have concerns.
Is not the pull request just in the developer branch ? In that case you could leave the pull request as is and try to solve it without affecting the master branch at all.
I’d prefer to revert and ensure all develop testers come back online. Then can we work on figuring out how to apply these changes and test them to ensure no breakage *before* we push them into develop.
On Sat, Sep 20, 2014 at 4:11 AM, Belcourt, Kenneth <kbelco@sandia.gov> wrote:
On Sep 18, 2014, at 11:25 PM, Edward Diener <eldiener@tropicsoft.com> wrote:
On 9/18/2014 11:54 PM, Belcourt, Kenneth wrote:
On Sep 17, 2014, at 12:39 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
There were a lot of discussions about MPL recently, and I created a few pull requests to address issues:
I’d like to revert the MPL PR that caused our testing problems, and the one Edward just applied. It was wrong of me to commit the PR without any testing. There’s been some list discussion but my overall concern is that we can’t clearly explain why some develop testers are broken and others are working okay. The commit completely broke Darwin, Debian, FreeBSD, QNX and Windows. But what’s really worrying is that some RHEL testers are still merrily cycling while other RHEL testers are broken. Same for Ubuntu, some are cycling okay while some are broken. I think it’s too risky to apply a patch when we, honestly I, don’t understand why this is happening.
While we could apply Andrey’s PR (below) I’m afraid that may mask real underlying problems that we may not find out about until this hits master, or worse, users. I’ll work with Andrey to get the MPL changes applied, perhaps in smaller PRs.
Please let me know if you have concerns.
Is not the pull request just in the developer branch ? In that case you could leave the pull request as is and try to solve it without affecting the master branch at all.
I’d prefer to revert and ensure all develop testers come back online. Then can we work on figuring out how to apply these changes and test them to ensure no breakage *before* we push them into develop.
Are there any news about this? Is someone looking into the problem with Boost.Build?
On 11/12/2014 2:41 AM, Andrey Semashev wrote:
On Sat, Sep 20, 2014 at 4:11 AM, Belcourt, Kenneth <kbelco@sandia.gov> wrote:
On Sep 18, 2014, at 11:25 PM, Edward Diener <eldiener@tropicsoft.com> wrote:
On 9/18/2014 11:54 PM, Belcourt, Kenneth wrote:
On Sep 17, 2014, at 12:39 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
There were a lot of discussions about MPL recently, and I created a few pull requests to address issues:
I’d like to revert the MPL PR that caused our testing problems, and the one Edward just applied. It was wrong of me to commit the PR without any testing. There’s been some list discussion but my overall concern is that we can’t clearly explain why some develop testers are broken and others are working okay. The commit completely broke Darwin, Debian, FreeBSD, QNX and Windows. But what’s really worrying is that some RHEL testers are still merrily cycling while other RHEL testers are broken. Same for Ubuntu, some are cycling okay while some are broken. I think it’s too risky to apply a patch when we, honestly I, don’t understand why this is happening.
While we could apply Andrey’s PR (below) I’m afraid that may mask real underlying problems that we may not find out about until this hits master, or worse, users. I’ll work with Andrey to get the MPL changes applied, perhaps in smaller PRs.
Please let me know if you have concerns.
Is not the pull request just in the developer branch ? In that case you could leave the pull request as is and try to solve it without affecting the master branch at all.
I’d prefer to revert and ensure all develop testers come back online. Then can we work on figuring out how to apply these changes and test them to ensure no breakage *before* we push them into develop.
Are there any news about this? Is someone looking into the problem with Boost.Build?
For mpl and 'develop' the latest commits that I see are pull requests from Eric Niebler which I merged. I do not believe these have caused any underlying problems.
On Wed, Nov 12, 2014 at 4:10 PM, Edward Diener <eldiener@tropicsoft.com> wrote:
On 11/12/2014 2:41 AM, Andrey Semashev wrote:
On Sat, Sep 20, 2014 at 4:11 AM, Belcourt, Kenneth <kbelco@sandia.gov> wrote:
On Sep 18, 2014, at 11:25 PM, Edward Diener <eldiener@tropicsoft.com> wrote:
On 9/18/2014 11:54 PM, Belcourt, Kenneth wrote:
On Sep 17, 2014, at 12:39 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
There were a lot of discussions about MPL recently, and I created a few pull requests to address issues:
I’d like to revert the MPL PR that caused our testing problems, and the one Edward just applied. It was wrong of me to commit the PR without any testing. There’s been some list discussion but my overall concern is that we can’t clearly explain why some develop testers are broken and others are working okay. The commit completely broke Darwin, Debian, FreeBSD, QNX and Windows. But what’s really worrying is that some RHEL testers are still merrily cycling while other RHEL testers are broken. Same for Ubuntu, some are cycling okay while some are broken. I think it’s too risky to apply a patch when we, honestly I, don’t understand why this is happening.
While we could apply Andrey’s PR (below) I’m afraid that may mask real underlying problems that we may not find out about until this hits master, or worse, users. I’ll work with Andrey to get the MPL changes applied, perhaps in smaller PRs.
Please let me know if you have concerns.
Is not the pull request just in the developer branch ? In that case you could leave the pull request as is and try to solve it without affecting the master branch at all.
I’d prefer to revert and ensure all develop testers come back online. Then can we work on figuring out how to apply these changes and test them to ensure no breakage *before* we push them into develop.
Are there any news about this? Is someone looking into the problem with Boost.Build?
For mpl and 'develop' the latest commits that I see are pull requests from Eric Niebler which I merged. I do not believe these have caused any underlying problems.
Sorry, the context is probably lost. When my pull request that separated MPL into MPL.Core was merged, it broke some testers. The breakage was caused by Boost.Build behavior, which did not create links to all headers in MPL and MPL.Core. The relevant threads are: http://thread.gmane.org/gmane.comp.lib.boost.devel/254291/ http://thread.gmane.org/gmane.comp.lib.boost.devel/254629/ It is not clear why that happens but I created a pull request with a workaround: https://github.com/boostorg/boost/pull/39 However, the commit to MPL was reverted and the workaround was not applied. I was asking about the state of the issue with Boost.Build.
On 11/12/2014 8:39 AM, Andrey Semashev wrote:
On Wed, Nov 12, 2014 at 4:10 PM, Edward Diener <eldiener@tropicsoft.com> wrote:
On 11/12/2014 2:41 AM, Andrey Semashev wrote:
On Sat, Sep 20, 2014 at 4:11 AM, Belcourt, Kenneth <kbelco@sandia.gov> wrote:
On Sep 18, 2014, at 11:25 PM, Edward Diener <eldiener@tropicsoft.com> wrote:
On 9/18/2014 11:54 PM, Belcourt, Kenneth wrote:
On Sep 17, 2014, at 12:39 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
> There were a lot of discussions about MPL recently, and I created a > few pull > requests to address issues:
I’d like to revert the MPL PR that caused our testing problems, and the one Edward just applied. It was wrong of me to commit the PR without any testing. There’s been some list discussion but my overall concern is that we can’t clearly explain why some develop testers are broken and others are working okay. The commit completely broke Darwin, Debian, FreeBSD, QNX and Windows. But what’s really worrying is that some RHEL testers are still merrily cycling while other RHEL testers are broken. Same for Ubuntu, some are cycling okay while some are broken. I think it’s too risky to apply a patch when we, honestly I, don’t understand why this is happening.
While we could apply Andrey’s PR (below) I’m afraid that may mask real underlying problems that we may not find out about until this hits master, or worse, users. I’ll work with Andrey to get the MPL changes applied, perhaps in smaller PRs.
Please let me know if you have concerns.
Is not the pull request just in the developer branch ? In that case you could leave the pull request as is and try to solve it without affecting the master branch at all.
I’d prefer to revert and ensure all develop testers come back online. Then can we work on figuring out how to apply these changes and test them to ensure no breakage *before* we push them into develop.
Are there any news about this? Is someone looking into the problem with Boost.Build?
For mpl and 'develop' the latest commits that I see are pull requests from Eric Niebler which I merged. I do not believe these have caused any underlying problems.
Sorry, the context is probably lost.
When my pull request that separated MPL into MPL.Core was merged, it broke some testers. The breakage was caused by Boost.Build behavior, which did not create links to all headers in MPL and MPL.Core. The relevant threads are:
http://thread.gmane.org/gmane.comp.lib.boost.devel/254291/ http://thread.gmane.org/gmane.comp.lib.boost.devel/254629/
It is not clear why that happens but I created a pull request with a workaround:
https://github.com/boostorg/boost/pull/39
However, the commit to MPL was reverted and the workaround was not applied.
I was asking about the state of the issue with Boost.Build.
Maybe posting on the Boost Build mailing list would help since your pull request with a workaround refers to Boost Build.
On 11/12/2014 04:50 PM, Edward Diener wrote:
On 11/12/2014 8:39 AM, Andrey Semashev wrote:
On Wed, Nov 12, 2014 at 4:10 PM, Edward Diener <eldiener@tropicsoft.com> wrote:
On 11/12/2014 2:41 AM, Andrey Semashev wrote:
On Sat, Sep 20, 2014 at 4:11 AM, Belcourt, Kenneth <kbelco@sandia.gov> wrote:
On Sep 18, 2014, at 11:25 PM, Edward Diener <eldiener@tropicsoft.com> wrote:
On 9/18/2014 11:54 PM, Belcourt, Kenneth wrote: > > > On Sep 17, 2014, at 12:39 PM, Andrey Semashev > <andrey.semashev@gmail.com> wrote: > >> There were a lot of discussions about MPL recently, and I created a >> few pull >> requests to address issues: > > > I’d like to revert the MPL PR that caused our testing problems, and the > one Edward just applied. It was wrong of me to commit the PR without any > testing. There’s been some list discussion but my overall concern is that > we can’t clearly explain why some develop testers are broken and others are > working okay. The commit completely broke Darwin, Debian, FreeBSD, QNX and > Windows. But what’s really worrying is that some RHEL testers are still > merrily cycling while other RHEL testers are broken. Same for Ubuntu, some > are cycling okay while some are broken. I think it’s too risky to apply a > patch when we, honestly I, don’t understand why this is happening. > > While we could apply Andrey’s PR (below) I’m afraid that may mask real > underlying problems that we may not find out about until this hits master, > or worse, users. I’ll work with Andrey to get the MPL changes applied, > perhaps in smaller PRs. > > Please let me know if you have concerns.
Is not the pull request just in the developer branch ? In that case you could leave the pull request as is and try to solve it without affecting the master branch at all.
I’d prefer to revert and ensure all develop testers come back online. Then can we work on figuring out how to apply these changes and test them to ensure no breakage *before* we push them into develop.
Are there any news about this? Is someone looking into the problem with Boost.Build?
For mpl and 'develop' the latest commits that I see are pull requests from Eric Niebler which I merged. I do not believe these have caused any underlying problems.
Sorry, the context is probably lost.
When my pull request that separated MPL into MPL.Core was merged, it broke some testers. The breakage was caused by Boost.Build behavior, which did not create links to all headers in MPL and MPL.Core. The relevant threads are:
http://thread.gmane.org/gmane.comp.lib.boost.devel/254291/ http://thread.gmane.org/gmane.comp.lib.boost.devel/254629/
It is not clear why that happens but I created a pull request with a workaround:
https://github.com/boostorg/boost/pull/39
However, the commit to MPL was reverted and the workaround was not applied.
I was asking about the state of the issue with Boost.Build.
Maybe posting on the Boost Build mailing list would help since your pull request with a workaround refers to Boost Build.
If I recall correctly, it was never determined what the breakage is, and how to reproduce it at will. Is there a reproduction recipe now? -- Vladimir Prus CodeSourcery / Mentor Embedded http://vladimirprus.com
On Wed, Nov 12, 2014 at 5:04 PM, Vladimir Prus <vladimir@codesourcery.com> wrote:
On 11/12/2014 04:50 PM, Edward Diener wrote:
On 11/12/2014 8:39 AM, Andrey Semashev wrote:
Sorry, the context is probably lost.
When my pull request that separated MPL into MPL.Core was merged, it broke some testers. The breakage was caused by Boost.Build behavior, which did not create links to all headers in MPL and MPL.Core. The relevant threads are:
http://thread.gmane.org/gmane.comp.lib.boost.devel/254291/ http://thread.gmane.org/gmane.comp.lib.boost.devel/254629/
It is not clear why that happens but I created a pull request with a workaround:
https://github.com/boostorg/boost/pull/39
However, the commit to MPL was reverted and the workaround was not applied.
I was asking about the state of the issue with Boost.Build.
Maybe posting on the Boost Build mailing list would help since your pull request with a workaround refers to Boost Build.
If I recall correctly, it was never determined what the breakage is, and how to reproduce it at will.
Is there a reproduction recipe now?
Yes, I posted the link to the reproducing steps in the pull request. Since MPL is now reverted, the steps change a little: 1. Checkout libs/mpl from my fork https://github.com/lastique/mpl/tree/modularization. This is how MPL looked before it was reverted. You can do it like this: cd libs/mpl git remote add lastique git@github.com:Lastique/mpl.git git fetch lastique git checkout modularization cd ../.. At this point you should see modularized MPL - there will be libs/mpl/core directory. 2. rm -f bin* boost 3. cd tools/regression/build 4. b2 -q 5. Notice that this directory is missing: boost/mpl/aux_/preprocessed. You will also notice that a lot of boost/ content is missing.
On 11/12/2014 05:16 PM, Andrey Semashev wrote:
1. Checkout libs/mpl from my fork https://github.com/lastique/mpl/tree/modularization. This is how MPL looked before it was reverted. You can do it like this:
cd libs/mpl git remote add lastiquegit@github.com:Lastique/mpl.git git fetch lastique git checkout modularization cd ../..
At this point you should see modularized MPL - there will be libs/mpl/core directory.
2. rm -f bin* boost 3. cd tools/regression/build 4. b2 -q 5. Notice that this directory is missing: boost/mpl/aux_/preprocessed. You will also notice that a lot of boost/ content is missing.
Andrey, the error I get is: ../../../boost/mpl/aux_/include_preprocessed.hpp:37:90: fatal error: boost/mpl/aux_/preprocessed/gcc/or.hpp: No such file or directory The immediate reason it happens is that the inclusion of or.hpp is indirect, via include_preprocessed.hpp: # define AUX778076_PREPROCESSED_HEADER \ BOOST_MPL_CFG_COMPILER_DIR/BOOST_MPL_PREPROCESSED_HEADER \ # include BOOST_PP_STRINGIZE(boost/mpl/aux_/preprocessed/AUX778076_PREPROCESSED_HEADER) Boost.Build cannot see that dependency so it does not know what gcc/or.hpp has to be linked prior to building this source. It's working in current devel tree, because MPL is handled by a single link action:building reports this: mklink-or-dir ../../../boost/mpl So boost/mpl is just a symlink to original location, and it's created very early due to other, discovered, dependencies on MPL. After your modularization, you have headers to two different components. But, you want all headers to be present in boost/mpl, so the only way to accomplish that is to link headers one-by-one, and track header dependencies one-by-one, and it fails due to preprocessor usage. I see a few possible options: - Modularize MPL all the way, so that the sub-components are disjoint in install tree, not just in source tree - Add a mechanism to explicitly describe header dependencies that cannot be found - Unconditionally link MPL headers prior to any build Comments? -- Vladimir Prus CodeSourcery / Mentor Embedded http://vladimirprus.com
On Wednesday 03 December 2014 11:33:25 Vladimir Prus wrote:
On 11/12/2014 05:16 PM, Andrey Semashev wrote:
1. Checkout libs/mpl from my fork https://github.com/lastique/mpl/tree/modularization. This is how MPL
looked before it was reverted. You can do it like this: cd libs/mpl git remote add lastiquegit@github.com:Lastique/mpl.git git fetch lastique git checkout modularization cd ../..
At this point you should see modularized MPL - there will be libs/mpl/core directory.
2. rm -f bin* boost 3. cd tools/regression/build 4. b2 -q 5. Notice that this directory is missing: boost/mpl/aux_/preprocessed. You will also notice that a lot of boost/ content is missing.
Andrey,
the error I get is:
../../../boost/mpl/aux_/include_preprocessed.hpp:37:90: fatal error: boost/mpl/aux_/preprocessed/gcc/or.hpp: No such file or directory
The immediate reason it happens is that the inclusion of or.hpp is indirect, via include_preprocessed.hpp:
# define AUX778076_PREPROCESSED_HEADER \ BOOST_MPL_CFG_COMPILER_DIR/BOOST_MPL_PREPROCESSED_HEADER \ # include BOOST_PP_STRINGIZE(boost/mpl/aux_/preprocessed/AUX778076_PREPROCESSED_HEADE R)
Boost.Build cannot see that dependency so it does not know what gcc/or.hpp has to be linked prior to building this source.
It's working in current devel tree, because MPL is handled by a single link action:building reports this:
mklink-or-dir ../../../boost/mpl
So boost/mpl is just a symlink to original location, and it's created very early due to other, discovered, dependencies on MPL. After your modularization, you have headers to two different components. But, you want all headers to be present in boost/mpl, so the only way to accomplish that is to link headers one-by-one, and track header dependencies one-by-one, and it fails due to preprocessor usage.
Yes, I suspected that was the reason.
I see a few possible options:
- Modularize MPL all the way, so that the sub-components are disjoint in install tree, not just in source tree - Add a mechanism to explicitly describe header dependencies that cannot be found - Unconditionally link MPL headers prior to any build
Comments?
Such mechanism would require modification to the MPL Jamfile, am I correct? I generally don't like when I have to manually describe dependencies, because when the code is modified these descriptions often get overlooked. However, if the description is made on the sublibs level (i.e. that "libs/mpl depends on libs/mpl/core" and that makes all MPL.Core headers linked) this might be ok. Why not just run the headers target before building anything. b2 headers does the right thing, and any project inside Boost implicitly relies on the boost/ directory tree. I was trying to accomplish that by modifying the tools/regression/build/Jamroot.jam (see my pull request). My reasoning is that since b2 cannot deduce #includes from headers in cases involving preprocessor, like in MPL, we should not rely on it and just always link all headers. By the way, do you know why implicit-dependency on the headers target in that Jamfile does not work? My understanding is that its intention was exactly what I suggested above. PS: Thank you for looking into this.
On 12/03/2014 11:53 AM, Andrey Semashev wrote:
Comments?
Such mechanism would require modification to the MPL Jamfile, am I correct? I generally don't like when I have to manually describe dependencies, because when the code is modified these descriptions often get overlooked.
I was thinking more about "Whenever <mpl/...> include is found, add dependency on every MPL header".
However, if the description is made on the sublibs level (i.e. that "libs/mpl depends on libs/mpl/core" and that makes all MPL.Core headers linked) this might be ok.
Why not just run the headers target before building anything. b2 headers does the right thing, and any project inside Boost implicitly relies on the boost/ directory tree. I was trying to accomplish that by modifying the tools/regression/build/Jamroot.jam (see my pull request). My reasoning is that since b2 cannot deduce #includes from headers in cases involving preprocessor, like in MPL, we should not rely on it and just always link all headers.
It is an option, yes. It's not entirely perfect because all headers will be linked even if you build a single library.
By the way, do you know why implicit-dependency on the headers target in that Jamfile does not work? My understanding is that its intention was exactly what I suggested above.
Because the effect of 'implicit-dependency' is: 1. When you see include of a header ... 2. ... check that other target mentioned by implicit-dependency ... 3. ... and it's generating such header ... 4. ... add dependency. In our cases, preprocessor magic hides the fact that gcc/or.hpp is included, so the process breaks at step 1, and whether we have implicit-dependency or not no longer matters. -- Vladimir Prus CodeSourcery / Mentor Embedded http://vladimirprus.com
On Wednesday 03 December 2014 12:48:18 Vladimir Prus wrote:
On 12/03/2014 11:53 AM, Andrey Semashev wrote:
Such mechanism would require modification to the MPL Jamfile, am I correct? I generally don't like when I have to manually describe dependencies, because when the code is modified these descriptions often get overlooked.
I was thinking more about "Whenever <mpl/...> include is found, add dependency on every MPL header".
Hmm, that means to hardcode MPL somewhere, and this won't scale for other libraries, if we go on modularizing. I'd prefer a more general solution - ideally, totally automagic and transparent, otherwise, if not possible, at the library developers' control.
Why not just run the headers target before building anything. b2 headers does the right thing, and any project inside Boost implicitly relies on the boost/ directory tree. I was trying to accomplish that by modifying the tools/regression/build/Jamroot.jam (see my pull request). My reasoning is that since b2 cannot deduce #includes from headers in cases involving preprocessor, like in MPL, we should not rely on it and just always link all headers. It is an option, yes. It's not entirely perfect because all headers will be linked even if you build a single library.
Well, yes. In the long term, when we support modularized builds this can probably matter, although not necessarily so. I imagine, when Boost gets truly modular, one would download sources of the needed libraries separately, and to build them one would have to link all headers from these libraries anyway. With the current state of affair, I don't think there's a problem to link all Boost headers regardless of the built libraries, at least for testing. I mean, we have to do this anyway in order to test and distribute Boost, and I bet no- one is trying to build Boost themselves from git without running 'b2 headers'. AFAIR, no-one complained about it so far.
By the way, do you know why implicit-dependency on the headers target in that Jamfile does not work? My understanding is that its intention was exactly what I suggested above.
Because the effect of 'implicit-dependency' is:
1. When you see include of a header ... 2. ... check that other target mentioned by implicit-dependency ... 3. ... and it's generating such header ... 4. ... add dependency.
In our cases, preprocessor magic hides the fact that gcc/or.hpp is included, so the process breaks at step 1, and whether we have implicit-dependency or not no longer matters.
I see, thanks. So, in order to resolve the problem with testing, do you think my pull request[1] is valid? It basically changes 'implicit-dependency' to 'dependency', which, I assume, is not conditional on the included headers. [1] https://github.com/boostorg/boost/pull/39
Andrey Semashev wrote:
I imagine, when Boost gets truly modular, one would download sources of the needed libraries separately,
That is explicitly a non-goal for Boost. http://thread.gmane.org/gmane.comp.lib.boost.devel/254635 If you want your imagined scenario to result in a new reality, start by making that a goal for boost. If you don't do it, no one will :). Thanks, Steve.
On 12/3/2014 7:56 PM, Stephen Kelly wrote:
Andrey Semashev wrote:
I imagine, when Boost gets truly modular, one would download sources of the needed libraries separately,
That is explicitly a non-goal for Boost.
http://thread.gmane.org/gmane.comp.lib.boost.devel/254635
If you want your imagined scenario to result in a new reality, start by making that a goal for boost. If you don't do it, no one will :).
I believe that Boost should pursue the goal of allowing individual libraries to be downloaded and used without having to download the complete Boost structure. A couple of things however I will mention in my usual list-like way: 1) The people who are interested in this goal have to be willing to "get together" and discuss this in a calm and rational way in order to come up with a solution. This is not going to be easy and those who want to be involved need to take their time with it and not work in some sort of panic mode. This means everyone needs to back off the rhetoric and carefully consider possible solutions. I have my own ideas and I am sure a number of people also have their own ideas, so people need to be tolerant of individual solutions. If enough people agree on a particular solution it can be done by those people. There should not be any pressure on anybody to work on the solution since it is almost ceetainly going to be much effort. 2) Allowing individual libraries to be used, rather than current monolithic Boost, is being done for the end-user community. We need to make the goal that the way of doing this is as simple for the end-user as possible, no matter what has to be done underneath to get everything to work properly. If we make things arcane at all for the end-user the efforts will be a failure, no matter how clever we think we are. The end-users are not Linux gurus or Windows gurus or Mac gurus. They are simply programmers wanting to use individual Boost libraries for their programming efforts. Assuming guru status and telling them to follow a complicated menu of items just to get an individual library and its dependencies working correctly, which Boost developers themselves might accept, is not going to work with with individual end-user programmers or programmers in corporate America. 3) Modular Boost with Git has already been a success over the previous SVN system. As someone who previous to the move knew next to nothing about Git and has had the usual beginner's problems with it I think I can say unequivocally that it is a much better system than previously. So let's not spend time carping over that choice. 4) We should probably just start with proposals, preferable published on Github. Whoever has a proposal specifies what it is in whatever general terms or detail terms he chooses. We can say that proposals should be made within some time period, let's say within the next three months. After which people who are interested can discuss the proposals and try to see if there is one that meets a fairly large consensus. If that is the case, then those who are part of that consensus can work together to make it happen. Endless discussions of "let's try this, no let's do that, no that is no good but maybe this will work or that will work, but if we could only do this or that, and if we used my tool everything is fine, or if we used X tool on the Internet it might fly, or Y development does this and Z development does that so why don't we follow their example etc." is not going to get us anywhere. The problem is complicated and we really need serious proposals and plans. 5) Even with a consensus plan it's very important that there is no rush or panic to getting i done and getting everything right. There is no time by which this must be done. Monolithic Boost is not a failure, it's just the single alternative we have now and it works. But if we can provide an alternative, of individual libraries and their dependencies being used without the necessity of all of Boost, I believe it would be greatly appreciated in programming shops where monolithic Boost is seen as an impediment to program development ( because of the large size of the directory structure, because many businesses have tight rules above 3rd party software, because of the fear of too many dependencies or too fragile dependencies etc. etc. ). 6) I see the ability to use only individual subsets of Boost libraries as a worthwhile goal. This is not because I view monolithic Boost as a failure of any kind, or that I feel that the libraries that are part of Boost are not of the highest quality, but because I understand the impetus of programmer end-users who only want to deal physically with the libraries they actually use without the responsibility of even downloading libraries to which they are indifferent.
Edward Diener wrote:
I believe that Boost should pursue the goal of allowing individual libraries to be downloaded and used without having to download the complete Boost structure.
That would be nice :).
If enough people agree on a particular solution it can be done by those people.
That's not how Boost works. Boost libraries and decisions related to them are owned by the maintainer of the library. There is no 'enough people' who could decide to split the parts of Boost.Serialization which depend on Boost.Spirit away from the rest of the library, for example. Robert doesn't want that, so it won't happen. Decisions are not made as a community. That's how Boost works. 'Enough people' is not a factor. Am I wrong?
2) Allowing individual libraries to be used, rather than current monolithic Boost, is being done for the end-user community. We need to make the goal that the way of doing this is as simple for the end-user as possible
That would be nice :). I don't see how it is possible without changing the Boost model of 'each individual maintainer decides, always'. If such rules were adopted you'd have to make it clear what the implications of having such a goal are.
They are simply programmers wanting to use individual Boost libraries for their programming efforts.
Are you sure users want to use individual libraries? Do you see calls for that from users?
After which people who are interested can discuss the proposals and try to see if there is one that meets a fairly large consensus. If that is the case, then those who are part of that consensus can work together to make it happen.
Are you suggesting the result of such consensus would be forced on Boost maintainers independent of what they may want? That would seem to be a break from the current model of how Boost works. Thanks, Steve.
Stephen Kelly wrote:
Are you sure users want to use individual libraries? Do you see calls for that from users?
This was certainly true in the past. It is why bcp exists. shared_ptr.hpp for years even supported compilation with a completely empty config.hpp on a reasonably conforming compiler, for people who didn't even want Boost.Config. It may be less true today though, as it's gotten way easier to "use Boost" in a project. But there are still places where Boost as a whole is banned, yet a module may be able to slip through.
On 12/6/2014 8:46 AM, Stephen Kelly wrote:
Edward Diener wrote:
I believe that Boost should pursue the goal of allowing individual libraries to be downloaded and used without having to download the complete Boost structure.
That would be nice :).
If enough people agree on a particular solution it can be done by those people.
That's not how Boost works. Boost libraries and decisions related to them are owned by the maintainer of the library. There is no 'enough people' who could decide to split the parts of Boost.Serialization which depend on Boost.Spirit away from the rest of the library, for example. Robert doesn't want that, so it won't happen. Decisions are not made as a community. That's how Boost works. 'Enough people' is not a factor. Am I wrong?
You are right that you can't force a particular library's maintenance team to do something. That does not necessarily mean that people would not come together to make changes benefitting Boost in general.
2) Allowing individual libraries to be used, rather than current monolithic Boost, is being done for the end-user community. We need to make the goal that the way of doing this is as simple for the end-user as possible
That would be nice :).
I don't see how it is possible without changing the Boost model of 'each individual maintainer decides, always'.
If such rules were adopted you'd have to make it clear what the implications of having such a goal are.
They are simply programmers wanting to use individual Boost libraries for their programming efforts.
Are you sure users want to use individual libraries? Do you see calls for that from users?
I have seen the calls for that from users. The general objection is that downloading/installing monolithic Boost when only a few libraries are needed "seems" wrong.
After which people who are interested can discuss the proposals and try to see if there is one that meets a fairly large consensus. If that is the case, then those who are part of that consensus can work together to make it happen.
Are you suggesting the result of such consensus would be forced on Boost maintainers independent of what they may want? That would seem to be a break from the current model of how Boost works.
No. I still believe that things can be done by consensus. Recently a 'meta' directory with some information was added to all Boost libraries. I did not see any uprising against this. So clearly it is possible.
Edward Diener wrote:
Are you sure users want to use individual libraries? Do you see calls for that from users?
I have seen the calls for that from users. The general objection is that downloading/installing monolithic Boost when only a few libraries are needed "seems" wrong.
I haven't seen calls for modular downloads from users on the mailing list http://news.gmane.org/gmane.comp.lib.boost.user Maybe because they have other solutions to the issue, or use bcp? I know we do have a home-made solution in my workplace. Some time ago, it was not possible to move some files to another repository expressly because 'I maintain those files and he maintains that repo - It's not a reasonable situation for me to maintain those files in his repo'. I found that interesting. Anyway: If modular releases/downloads is a goal (I remain convinced it is not a goal for Boost thus far), are there blockers to making it happen?
Are you suggesting the result of such consensus would be forced on Boost maintainers independent of what they may want? That would seem to be a break from the current model of how Boost works.
No. I still believe that things can be done by consensus. Recently a 'meta' directory with some information was added to all Boost libraries. I did not see any uprising against this. So clearly it is possible.
The suggestion that adding meta information is comparable to anything which could be more-intrusive, controversial or bike-sheddable is interesting. I would be surprised if it follows logically. Maybe we'll get a chance to find out. Thanks, Steve.
Stephen Kelly-2 wrote
Andrey Semashev wrote:
I imagine, when Boost gets truly modular, one would download sources of the needed libraries separately,
That is explicitly a non-goal for Boost.
I don't think one can conclude from that thread that the ability to download and install one library is a non-goal for boost. In fact I think that anyone reading that thread would have to conclude the opposite. The problem is that this is a lot harder than it looks. It's much more than cleaning up dependencies. Proposals to achieve this - to the extent that they can be characterized as proposals - have been incomplete, impractible and/or unconvincing. I firmly believe that we want to do this and are on the path to doing so. -- View this message in context: http://boost.2283326.n4.nabble.com/mpl-build-testing-Merge-pull-requests-tp4... Sent from the Boost - Dev mailing list archive at Nabble.com.
Robert Ramey wrote:
Stephen Kelly-2 wrote
Andrey Semashev wrote:
I imagine, when Boost gets truly modular, one would download sources of the needed libraries separately,
That is explicitly a non-goal for Boost.
I don't think one can conclude from that thread that the ability to download and install one library is a non-goal for boost.
That's certainly my conclusion reading the thread. There is only opposition and stop-energy any time a topic related to it comes up.
In fact I think that anyone reading that thread would have to conclude the opposite.
I know you're not joking, but I find this claim hilarious :).
I firmly believe that we want to do this and are on the path to doing so.
How so? I've not seen anything to suggest that. Thanks, Steve.
Stephen Kelly-2 wrote
Robert Ramey wrote:
I firmly believe that we want to do this and are on the path to doing so.
How so? I've not seen anything to suggest that.
Hmmm - the movement to git and git submodules doesn't qualify? This was quite a bit of effort for everyone involved. Robert Ramey -- View this message in context: http://boost.2283326.n4.nabble.com/mpl-build-testing-Merge-pull-requests-tp4... Sent from the Boost - Dev mailing list archive at Nabble.com.
Robert Ramey wrote:
Stephen Kelly-2 wrote
Robert Ramey wrote:
I firmly believe that we want to do this and are on the path to doing so.
How so? I've not seen anything to suggest that.
Hmmm - the movement to git and git submodules doesn't qualify? This was quite a bit of effort for everyone involved.
git affects the developer workflow, but that thread was about creating modular releases and downloads, which is a different goal and has a different set of problems to solve. That's what that thread was about. I didn't see any desire to have that as a goal in the thread. Git doesn't affect users. Before and after transition to git, they download all-of-boost.zip from the boost website. Thanks, Steve.
On 5 Dec 2014 at 11:24, Robert Ramey wrote:
I imagine, when Boost gets truly modular, one would download sources of the needed libraries separately,
That is explicitly a non-goal for Boost.
The problem is that this is a lot harder than it looks. It's much more than cleaning up dependencies. Proposals to achieve this - to the extent that they can be characterized as proposals - have been incomplete, impractible and/or unconvincing. I firmly believe that we want to do this and are on the path to doing so.
I think that statement poorly phrased when we have a fully functional and proven working modular Boost proposal in BindLib. It's just a few libraries supporting it yet, but I claim you cannot call this "unconvincing" when it works right here and now. Want to test my proposed solution for modular Boost right now? Do this from a VS2015 CTP [1] command box: git clone https://github.com/BoostGSoC13/boost.afio.git cd boost.afio git checkout develop git submodule update --init --recursive git clone https://github.com/chriskohlhoff/asio.git cl /Zi /EHs /Oy- /Ob0 /MD /RTC1 /GF /GR /Gy /bigobj /wd4503 /Zc:forScope /Zc:wchar_t /W3 test\test_all.cpp detail\SpookyV2.cpp /DUNICODE=1 /DWIN32=1 /D_UNICODE=1 /D_WIN32=1 /Iinclude /Itest /DAFIO_STANDALONE=1 /DASIO_STANDALONE=1 /Iasio/asio/include /DSPINLOCK_STANDALONE=1 /DBOOST_AFIO_RUNNING_IN_CI=1 ... and it should hopefully spit out a test_all.exe which has not a single Boost dependency in it, despite being a 100% Boost library and still written to use Boost.Test, Boost.Config, Boost.Thread and so on. I stitched that together from my test config here so it may be slightly off, but it definitely works. I am going to set up Jenkins CI testing for standalone builds later today. [1]: Only VS2015 CTP currently has a working Filesystem TS. If you use VS2013 or GCC or clang it'll need to link in Boost.Filesystem. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
Vladimir Prus wrote:
- Add a mechanism to explicitly describe header dependencies that cannot be found
This mechanism kind of already exists, #if 0 #include "hidden-dependency" #endif but not all files have been annotated.
On Wednesday 03 December 2014 12:55:35 Peter Dimov wrote:
Vladimir Prus wrote:
- Add a mechanism to explicitly describe header dependencies that cannot be found
This mechanism kind of already exists,
#if 0 #include "hidden-dependency" #endif
but not all files have been annotated.
Yes, but that's quite ugly and error prone. In case of MPL this would mean to add all headers under boost/mpl/aux_/preprocessed (517 files) to boost/mpl/aux_/include_preprocessed.hpp or other headers that use include_preprocessed.hpp.
Andrey Semashev wrote:
On Wednesday 03 December 2014 12:55:35 Peter Dimov wrote:
This mechanism kind of already exists,
#if 0 #include "hidden-dependency" #endif
but not all files have been annotated.
Yes, but that's quite ugly and error prone. In case of MPL this would mean to add all headers under boost/mpl/aux_/preprocessed (517 files) to boost/mpl/aux_/include_preprocessed.hpp or other headers that use include_preprocessed.hpp.
It's not quite that bad. You need a section with 11 includes: #if 0 // inform boost.build of hidden dependencies # include "boost/mpl/aux_/preprocessed/bcc/_all_.hpp" # include "boost/mpl/aux_/preprocessed/bcc_pre90/_all_.hpp" ... #endif and 11 copies of the same _all_.hpp header that consists of 47 includes and will never change. Ugly and error prone? No more than what's already there, I'd say.
On Wed, Dec 3, 2014 at 2:36 PM, Peter Dimov <lists@pdimov.com> wrote:
Andrey Semashev wrote:
On Wednesday 03 December 2014 12:55:35 Peter Dimov wrote:
This mechanism kind of already exists,
#if 0 #include "hidden-dependency" #endif
but not all files have been annotated.
Yes, but that's quite ugly and error prone. In case of MPL this would mean to add all headers under boost/mpl/aux_/preprocessed (517 files) to boost/mpl/aux_/include_preprocessed.hpp or other headers that use include_preprocessed.hpp.
It's not quite that bad. You need a section with 11 includes:
#if 0 // inform boost.build of hidden dependencies # include "boost/mpl/aux_/preprocessed/bcc/_all_.hpp" # include "boost/mpl/aux_/preprocessed/bcc_pre90/_all_.hpp" ... #endif
and 11 copies of the same _all_.hpp header that consists of 47 includes and will never change.
I still don't like the idea that we have to mangle the code to guide the build system. And I don't want to set a precedent in MPL as I know there are many more places where preprocessor is used to include headers.
Ugly and error prone? No more than what's already there, I'd say.
It may not be as error prone for MPL, given that it is basically not developed nowdays. But the approach is still flawed - you have to remember to modify unrelated headers in disabled sections when you add a feature or specialized code branch. IMHO, either Boost.Build has to parse the code properly, or it should be guided by its own config files, i.e. Jamfiles. Or it could just always link all headers.
On Wed, Dec 3, 2014 at 2:07 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
On Wed, Dec 3, 2014 at 2:36 PM, Peter Dimov <lists@pdimov.com> wrote:
Andrey Semashev wrote:
On Wednesday 03 December 2014 12:55:35 Peter Dimov wrote:
#if 0 // inform boost.build of hidden dependencies # include "boost/mpl/aux_/preprocessed/bcc/_all_.hpp" # include "boost/mpl/aux_/preprocessed/bcc_pre90/_all_.hpp" ... #endif
and 11 copies of the same _all_.hpp header that consists of 47 includes and will never change.
I still don't like the idea that we have to mangle the code to guide the build system. And I don't want to set a precedent in MPL as I know there are many more places where preprocessor is used to include headers.
Ugly and error prone? No more than what's already there, I'd say.
IMHO, either Boost.Build has to parse the code properly,
Is there a build system out there that correctly handles the extent of MPLs preprocessor driven header inclusion? I'm asking so that we can ascertain how it could be done.
or it should be guided by its own config files, i.e. Jamfiles.
Don't know what you mean by that. Can you elaborate?
Or it could just always link all headers.
It's possible and easy to change the MPL build files to enumerate header dependencies (it can even be done automatically with globs) with the "dependency" feature. Sorry if I haven't followed this discussion.. But can you elaborate what the use cases are? -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
On Wed, Dec 3, 2014 at 11:30 PM, Rene Rivera <grafikrobot@gmail.com> wrote:
On Wed, Dec 3, 2014 at 2:07 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
IMHO, either Boost.Build has to parse the code properly,
Is there a build system out there that correctly handles the extent of MPLs preprocessor driven header inclusion? I'm asking so that we can ascertain how it could be done.
Frankly, I'm not sure any build system even attempts to parse C/C++, let alone do some preprocessing. I must say, I don't know many build systems, though. The way I see it, it would take a little more than a C++ preprocessor to do this. The preprocessor would have to support macro unfolding to obtain the file name. Note that the unfolding can yield different results depending on compiler/platform/etc., and the build system would have to collect all such combinations to get the full list of the possible includes.
or it should be guided by its own config files, i.e. Jamfiles.
Don't know what you mean by that. Can you elaborate?
I had something like the following in mind: 1. Boost.Build finds an #include in a source file. 2. It determines which library this header belongs to. 3. It parses the header to obtain the "obvious" dependencies from #includes not obscured by preprocessor. 4. It also peeks into the library Jamfiles to obtain dependencies defined by the library developer. 1-3 is basically what happens now. 4 is needed to explicitly describe dependencies which Boost.Build is unable to infer from headers. If Jamfiles don't fit for this role, I'm ok if it's a different config file.
Or it could just always link all headers.
It's possible and easy to change the MPL build files to enumerate header dependencies (it can even be done automatically with globs) with the "dependency" feature. Sorry if I haven't followed this discussion.. But can you elaborate what the use cases are?
AFAIU, MPL build files are not involved into the header linking process. Actually, there aren't any build files for MPL, as it's a header-only library. The use case is that when you build tools/regression/build, the build system only links headers that it can infer from #includes. It fails with modularized MPL because some headers in it are included using preprocessor. See the beginning of this thread.
On December 3, 2014 4:07:22 PM EST, Andrey Semashev <andrey.semashev@gmail.com> wrote:
On Wed, Dec 3, 2014 at 11:30 PM, Rene Rivera <grafikrobot@gmail.com> wrote:
On Wed, Dec 3, 2014 at 2:07 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
Frankly, I'm not sure any build system even attempts to parse C/C++, let alone do some preprocessing. I must say, I don't know many build systems, though.
None do, to my knowledge.
The way I see it, it would take a little more than a C++ preprocessor to do this. The preprocessor would have to support macro unfolding to obtain the file name. Note that the unfolding can yield different results depending on compiler/platform/etc., and the build system would have to collect all such combinations to get the full list of the possible includes.
Right.
I had something like the following in mind:
1. Boost.Build finds an #include in a source file. 2. It determines which library this header belongs to. 3. It parses the header to obtain the "obvious" dependencies from #includes not obscured by preprocessor. 4. It also peeks into the library Jamfiles to obtain dependencies defined by the library developer.
1-3 is basically what happens now. 4 is needed to explicitly describe dependencies which Boost.Build is unable to infer from headers. If Jamfiles don't fit for this role, I'm ok if it's a different config file.
Why not a dedicated header that exists to guide Boost.Build? ___ Rob (Sent from my portable computation engine)
On Thursday 04 December 2014 05:42:13 Rob Stewart wrote:
On December 3, 2014 4:07:22 PM EST, Andrey Semashev
I had something like the following in mind:
1. Boost.Build finds an #include in a source file. 2. It determines which library this header belongs to. 3. It parses the header to obtain the "obvious" dependencies from #includes not obscured by preprocessor. 4. It also peeks into the library Jamfiles to obtain dependencies defined by the library developer.
1-3 is basically what happens now. 4 is needed to explicitly describe dependencies which Boost.Build is unable to infer from headers. If Jamfiles don't fit for this role, I'm ok if it's a different config file.
Why not a dedicated header that exists to guide Boost.Build?
With the current state of things this header would have to be included where preprocessor magic happens, possibly #if-ed out. It isn't really different from just placing the #if-ed out #includes instead of the dedicated header. If Boost.Build is supposed to "know" about this header and parse it automatically without having to #include it anywhere then why is it a header and not a config file?
On December 4, 2014 6:07:42 AM EST, Andrey Semashev <andrey.semashev@gmail.com> wrote:
On Thursday 04 December 2014 05:42:13 Rob Stewart wrote:
On December 3, 2014 4:07:22 PM EST, Andrey Semashev
[snip]
explicitly describe dependencies which Boost.Build is unable to infer from headers. If Jamfiles don't fit for this role, I'm ok if it's a different config file.
Why not a dedicated header that exists to guide Boost.Build?
With the current state of things this header would have to be included where preprocessor magic happens, possibly #if-ed out. It isn't really different from just placing the #if-ed out #includes instead of the dedicated header.
One header to maintain seems better than many. ___ Rob (Sent from my portable computation engine)
On 4/12/2014 10:07, Andrey Semashev wrote:
On Wed, Dec 3, 2014 at 11:30 PM, Rene Rivera <grafikrobot@gmail.com> wrote:
On Wed, Dec 3, 2014 at 2:07 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
IMHO, either Boost.Build has to parse the code properly,
Is there a build system out there that correctly handles the extent of MPLs preprocessor driven header inclusion? I'm asking so that we can ascertain how it could be done.
Frankly, I'm not sure any build system even attempts to parse C/C++, let alone do some preprocessing. I must say, I don't know many build systems, though.
Bog-standard GCC makefiles usually do this by using special compiler options that generate dependency rules either instead of or as well as the normal compilation output. Of course, this isn't portable, and does require the build to be run in its entirety first -- it merely provides that dependency information to the second and subsequent builds so that it's known which source files need recompiling if a header file is changed, to avoid a full rebuild (or worse, an incomplete one). Something else still needs to know what order to build the libraries at the higher level though, especially for that first run.
Andrey Semashev wrote:
I still don't like the idea that we have to mangle the code to guide the build system. And I don't want to set a precedent in MPL as I know there are many more places where preprocessor is used to include headers.
Not very many. Boost.Config has some and it already uses the #if 0 technique to mark them. There were a few more occurrences, I forget where. Either way, if you #include via a macro, you mark with #if 0, if you don't (which is most people), you don't, and that's that.
It may not be as error prone for MPL, given that it is basically not developed nowdays.
It's not (just) that MPL is not developed; it's that the per-compiler preprocessed headers are (I suppose) generated by a script, and that same script can take care of the dependency _all_.hpp header and even the #if 0 include, if one is so inclined. (That last one would require one more level of indirection - #if 0 #include "boost/mpl/aux/preprocessed/_all_.hpp" #endif where _all_.hpp is #include "bcc/_all_.hpp" #include "bcc_pre90/_all_.hpp ... )
But the approach is still flawed - you have to remember to modify unrelated headers in disabled sections when you add a feature or specialized code branch.
IMHO, either Boost.Build has to parse the code properly, or it should be guided by its own config files, i.e. Jamfiles.
That also means that you modify unrelated files, even more unrelated than the header. It's much better if the dependency information is kept in the header rather than being a sidecar that you need to remember to keep in sync.
On Thursday 04 December 2014 00:26:32 Peter Dimov wrote:
Andrey Semashev wrote:
IMHO, either Boost.Build has to parse the code properly, or it should be guided by its own config files, i.e. Jamfiles.
That also means that you modify unrelated files, even more unrelated than the header. It's much better if the dependency information is kept in the header rather than being a sidecar that you need to remember to keep in sync.
The idea is to mark dependencies on other (sub)libraries in the Jamfiles. These dependencies don't change often. Also, Jamfiles is where I expect to find stuff related to the build system, which links the headers, not the headers themselves.
Peter Dimov wrote:
It's not quite that bad. You need a section with 11 includes:
#if 0 // inform boost.build of hidden dependencies # include "boost/mpl/aux_/preprocessed/bcc/_all_.hpp" # include "boost/mpl/aux_/preprocessed/bcc_pre90/_all_.hpp" ... #endif
Is Borland supported? Maybe those files should be removed. Then at least you need < 11. Thanks, Steve.
Stephen Kelly wrote:
Peter Dimov wrote:
It's not quite that bad. You need a section with 11 includes:
#if 0 // inform boost.build of hidden dependencies # include "boost/mpl/aux_/preprocessed/bcc/_all_.hpp" # include "boost/mpl/aux_/preprocessed/bcc_pre90/_all_.hpp" ... #endif
Is Borland supported?
Maybe those files should be removed. Then at least you need < 11.
The directory contains bcc bcc551 bcc_pre590 dmc gcc msvc60 msvc70 mwcw no_ctps no_ttp plain so I'm pretty sure _I_ only need one of those nowadays. :-) Maybe two, but I doubt it.
Peter Dimov wrote:
Stephen Kelly wrote:
Peter Dimov wrote:
It's not quite that bad. You need a section with 11 includes:
#if 0 // inform boost.build of hidden dependencies # include "boost/mpl/aux_/preprocessed/bcc/_all_.hpp" # include "boost/mpl/aux_/preprocessed/bcc_pre90/_all_.hpp" ... #endif
Is Borland supported?
Maybe those files should be removed. Then at least you need < 11.
The directory contains
bcc bcc551 bcc_pre590 dmc gcc msvc60 msvc70 mwcw no_ctps no_ttp plain
so I'm pretty sure _I_ only need one of those nowadays. :-) Maybe two, but I doubt it.
I removed some of those some time ago: http://lists.boost.org/boost-commit/2013/09/47831.php http://lists.boost.org/boost-commit/2013/10/48130.php I don't know the details of why they're there now. Thanks, Steve.
On Friday 19 September 2014 03:54:47 Belcourt, Kenneth wrote:
On Sep 17, 2014, at 12:39 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
There were a lot of discussions about MPL recently, and I created a few pull requests to address issues:
https://github.com/boostorg/boost/pull/39
Invoke 'headers' target before building testing tools. Note that several testers are currently failing to run because of this problem. Since this affects the testing process, I'd appreciate if someone familiar with it could take a look.
I’d like to revert the MPL PR that caused our testing problems, and the one Edward just applied. It was wrong of me to commit the PR without any testing. There’s been some list discussion but my overall concern is that we can’t clearly explain why some develop testers are broken and others are working okay. The commit completely broke Darwin, Debian, FreeBSD, QNX and Windows. But what’s really worrying is that some RHEL testers are still merrily cycling while other RHEL testers are broken. Same for Ubuntu, some are cycling okay while some are broken. I think it’s too risky to apply a patch when we, honestly I, don’t understand why this is happening.
While we could apply Andrey’s PR (below) I’m afraid that may mask real underlying problems that we may not find out about until this hits master, or worse, users. I’ll work with Andrey to get the MPL changes applied, perhaps in smaller PRs.
Please let me know if you have concerns.
Kenneth, I've already explained why the problem has likely appeared, and moving back to the monolithic MPL is what covers it. It's not the fix. I admit I don't know why implicit-dependency doesn't work in the tools Jamfile, but it is clear that the intention was to invoke 'headers' prior to building. That's what my second PR achieves, and I think this is the correct fix. Did you try it? The problem with implicit-dependency should be investigated too, and Vladimir indicated in the PR that he wants to do that. You can revert MPL if you still want to, but I honestly don't know how else (on the MPL side) we can do this.
Invoke 'headers' target before building testing tools. Note that
testers are currently failing to run because of this problem. Since
several this
affects the testing process, I'd appreciate if someone familiar with it could take a look.
But what¹s really worrying is that some RHEL testers are still merrily cycling while other RHEL testers are broken. Same for Ubuntu, some are cycling okay while some are broken. I think it¹s too risky to apply a patch when we, honestly I, don¹t understand why this is happening.
Would a bjam.log from a failing RHEL machine help? There is everything from: ../libs/multiprecision/test/test_rat_float_interconv.cpp:195: error: '>>' should be '> >' within a nested template argument list To: /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/s hared_ptr.h:146: error: cannot use typeid with -fno-rtti It takes 12min for my build to fail, I could up the build once an hour if that might help. I could even auto upload the bjam.log to the regression ftp.
Kenneth, I've already explained why the problem has likely appeared, and moving back to the monolithic MPL is what covers it. It's not the fix.
If we want to continue down this route, I would rather fix it than revert it. It is the develop branch, things break, they get fixed.
I admit I don't know why implicit-dependency doesn't work in the tools Jamfile, but it is clear that the intention was to invoke 'headers' prior to building. That's what my second PR achieves, and I think this is the correct fix. Did you try it?
The problem with implicit-dependency should be investigated too, and Vladimir indicated in the PR that he wants to do that.
I had brought up the issue with the headers target in the past on the boost-build list. For it to work for me I use: <use>/boost//headers <implicit-dependency>/boost//headers This was a breaking change in 1.56 for our codebase as before we simply included the /boost//headers target. If this is the intended behavior of use and implicit-dependancy I think it is possible to make the headers target an alias that has a usage-requirement of the use and implicit-dependancy of the real headers target. This would make the headers target behave like it did before 1.56. There seemed to be little interest when I brought it up before, I think I was the only one affected at the time. - Thomas
On Friday 19 September 2014 15:56:26 Suckow, Thomas J wrote:
Would a bjam.log from a failing RHEL machine help?
There is everything from: ../libs/multiprecision/test/test_rat_float_interconv.cpp:195: error: '>>' should be '> >' within a nested template argument list
To: /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/s hared_ptr.h:146: error: cannot use typeid with -fno-rtti
It takes 12min for my build to fail, I could up the build once an hour if that might help. I could even auto upload the bjam.log to the regression ftp.
This sounds like the build actually starts (i.e. the testing tools are built successfully and library tests began running, some of them failing). This is not the problem we're seeing, but it would be interesting to know why it doesn't manifest in your case. Did you somehow modify your build scripts? Do you manually invoke 'b2 headers' before starting tests? Do you delete the boost directory before running tests? Is there anything special in your user-config.jam? How do you start tests? There's clearly some difference between testers, which makes some of them fail and others succeed.
I had brought up the issue with the headers target in the past on the boost-build list.
For it to work for me I use: <use>/boost//headers
<implicit-dependency>/boost//headers
This was a breaking change in 1.56 for our codebase as before we simply included the /boost//headers target. If this is the intended behavior of use and implicit-dependancy I think it is possible to make the headers target an alias that has a usage-requirement of the use and implicit-dependancy of the real headers target. This would make the headers target behave like it did before 1.56. There seemed to be little interest when I brought it up before, I think I was the only one affected at the time.
I'm not a Boost.Build expert so I didn't quite understand your way of fixing it. Could you specify how and what code you changed (i.e. a patch or modified code sample)?
On 9/19/14, 9:22 AM, "Andrey Semashev" <andrey.semashev@gmail.com> wrote:
On Friday 19 September 2014 15:56:26 Suckow, Thomas J wrote:
Would a bjam.log from a failing RHEL machine help?
There is everything from: ../libs/multiprecision/test/test_rat_float_interconv.cpp:195: error: '>>' should be '> >' within a nested template argument list
To:
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits /s hared_ptr.h:146: error: cannot use typeid with -fno-rtti
It takes 12min for my build to fail, I could up the build once an hour if that might help. I could even auto upload the bjam.log to the regression ftp.
This sounds like the build actually starts (i.e. the testing tools are built successfully and library tests began running, some of them failing). This is not the problem we're seeing, but it would be interesting to know why it doesn't manifest in your case.
Did you somehow modify your build scripts? Do you manually invoke 'b2 headers' before starting tests? Do you delete the boost directory before running tests? Is there anything special in your user-config.jam? How do you start tests?
There's clearly some difference between testers, which makes some of them fail and others succeed.
Opps. In my haste I pulled the bjam.log from the last SUCCESSFUL build. Bjam.log is not built. I tee¹d the run.py and have uploaded it below https://gist.github.com/thomassuckow/51f3dc348f9654abfb52
I had brought up the issue with the headers target in the past on the boost-build list.
For it to work for me I use: <use>/boost//headers
<implicit-dependency>/boost//headers
This was a breaking change in 1.56 for our codebase as before we simply included the /boost//headers target. If this is the intended behavior of use and implicit-dependancy I think it is possible to make the headers target an alias that has a usage-requirement of the use and implicit-dependancy of the real headers target. This would make the headers target behave like it did before 1.56. There seemed to be little interest when I brought it up before, I think I was the only one affected at the time.
I'm not a Boost.Build expert so I didn't quite understand your way of fixing it. Could you specify how and what code you changed (i.e. a patch or modified code sample)?
When I build our project with boost-build we have to specify headers like this to make sure the symlinks are made: exe myExe : myExe.cpp /boost//program_options /boost//system : <use>/boost//headers <implicit-dependency>/boost//headers ; We used to, before 1.56, be able to do: exe myExe : myExe.cpp /boost//program_options /boost//system /boost//headers ; I¹ll go try the alias trick I thought of and see if that would work. -Thomas
There¹s been some list discussion but my overall concern is that we can¹t clearly explain why some develop testers are broken and others are working okay. The commit completely broke Darwin, Debian, FreeBSD, QNX and Windows. But what¹s really worrying is that some RHEL testers are still merrily cycling while other RHEL testers are broken. Same for Ubuntu, some are cycling okay while some are broken.
Based on my limited testing, it would appear to me that it works on systems with headers from another version of boost in /usr/include or /usr/local/include. I am going to continue to dig into this. - Thomas
On 9/19/14, 3:21 PM, "Suckow, Thomas J" <Thomas.Suckow@pnnl.gov> wrote:
I am going to continue to dig into this.
So what is occurring is that the preproccessed directory is not being brought in because the only way it can be #included is through some special #define magic. Before the splitting of core, the whole mpl include directory was symlinked. Now because files become intermixed, each file is symlinked by itself. Since the headers target is not invoked those files are not linked. I really am not sure how to fix this. I was not successful in my attempt to make the headers alias work like it did before. Boost-build complained that I had somehow made the default build different than all other builds. - Thomas
On Sep 19, 2014, at 5:20 PM, Suckow, Thomas J <Thomas.Suckow@pnnl.gov> wrote:
On 9/19/14, 3:21 PM, "Suckow, Thomas J" <Thomas.Suckow@pnnl.gov> wrote:
I am going to continue to dig into this.
So what is occurring is that the preproccessed directory is not being brought in because the only way it can be #included is through some special #define magic.
Thanks Thomas. That’s the same failure we’ve been seeing.
On Friday 19 September 2014 23:20:30 Suckow, Thomas J wrote:
On 9/19/14, 3:21 PM, "Suckow, Thomas J" <Thomas.Suckow@pnnl.gov> wrote:
I am going to continue to dig into this.
So what is occurring is that the preproccessed directory is not being brought in because the only way it can be #included is through some special #define magic.
Before the splitting of core, the whole mpl include directory was symlinked. Now because files become intermixed, each file is symlinked by itself. Since the headers target is not invoked those files are not linked.
Yes, that was my thought as well.
I really am not sure how to fix this. I was not successful in my attempt to make the headers alias work like it did before. Boost-build complained that I had somehow made the default build different than all other builds.
You can try the change from my PR. https://github.com/boostorg/boost/pull/39
On Friday 19 September 2014 22:21:27 Suckow, Thomas J wrote:
There¹s been some list discussion but my overall concern is that
we can¹t clearly explain why some develop testers are broken and others
are
working okay. The commit completely broke Darwin, Debian, FreeBSD, QNX
and
Windows. But what¹s really worrying is that some RHEL testers are still merrily cycling while other RHEL testers are broken. Same for Ubuntu,
some
are cycling okay while some are broken.
Based on my limited testing, it would appear to me that it works on systems with headers from another version of boost in /usr/include or /usr/local/include.
That's an interesting idea. MPL didn't change for quite a while, so it could work. Noel, if you have access to a running tester, could you verify this theory?
On Sep 19, 2014, at 12:27 AM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
On Friday 19 September 2014 03:54:47 Belcourt, Kenneth wrote:
On Sep 17, 2014, at 12:39 PM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
There were a lot of discussions about MPL recently, and I created a few pull requests to address issues:
https://github.com/boostorg/boost/pull/39
Invoke 'headers' target before building testing tools. Note that several testers are currently failing to run because of this problem. Since this affects the testing process, I'd appreciate if someone familiar with it could take a look.
I’d like to revert the MPL PR that caused our testing problems, and the one Edward just applied. It was wrong of me to commit the PR without any testing. There’s been some list discussion but my overall concern is that we can’t clearly explain why some develop testers are broken and others are working okay. The commit completely broke Darwin, Debian, FreeBSD, QNX and Windows. But what’s really worrying is that some RHEL testers are still merrily cycling while other RHEL testers are broken. Same for Ubuntu, some are cycling okay while some are broken. I think it’s too risky to apply a patch when we, honestly I, don’t understand why this is happening.
While we could apply Andrey’s PR (below) I’m afraid that may mask real underlying problems that we may not find out about until this hits master, or worse, users. I’ll work with Andrey to get the MPL changes applied, perhaps in smaller PRs.
Please let me know if you have concerns.
Kenneth,
dba Noel
I've already explained why the problem has likely appeared, and moving back to the monolithic MPL is what covers it. It's not the fix.
We’re in agreement that implicit-dependency is broken for some reason. But note that several of our internal projects utilize BB implicit-dependency and it works fine for us. My concern is that these projects will break when they pull Boost with this change, and this may well happen to other users.
I admit I don't know why implicit-dependency doesn't work in the tools Jamfile, but it is clear that the intention was to invoke 'headers' prior to building.
Yes, but be aware that it’s not just tools, every invocation of b2 is broken, fails to install full set of headers.
That's what my second PR achieves, and I think this is the correct fix. Did you try it?
Yes, it works on Linux, but I’m still reluctant to apply it without a firm understanding why the random tester behavior is occurring. I’m sort of okay with wholesale failure but I’m not okay with some RHEL and Ubuntu testers working okay, others failing.
The problem with implicit-dependency should be investigated too, and Vladimir indicated in the PR that he wants to do that.
Yes, saw that.
You can revert MPL if you still want to, but I honestly don't know how else (on the MPL side) we can do this.
I guess that’s really the question I plan to investigate (while the develop testers are all working correctly).
On Saturday 20 September 2014 00:26:26 Belcourt, Kenneth wrote:
On Sep 19, 2014, at 12:27 AM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
Kenneth,
dba Noel
Sorry, I took the name from your email address. I've never heard of "dba" before, I suppose its your alias for Boost activity?
I've already explained why the problem has likely appeared, and moving back to the monolithic MPL is what covers it. It's not the fix.
We’re in agreement that implicit-dependency is broken for some reason. But note that several of our internal projects utilize BB implicit-dependency and it works fine for us. My concern is that these projects will break when they pull Boost with this change, and this may well happen to other users.
Well, now that you're aware of the problem, you can prevent pulling from Boost until implicit-dependency is fixed. And the problem was already released with 1.56, so users who updated are affected. Since there were not many reports I'd assume Boost.Build (or specifically implicit-dependency) is not widely used in users' projects.
That's what my second PR achieves, and I think this is the correct fix. Did you try it?
Yes, it works on Linux, but I’m still reluctant to apply it without a firm understanding why the random tester behavior is occurring. I’m sort of okay with wholesale failure but I’m not okay with some RHEL and Ubuntu testers working okay, others failing.
Ok, lets's find the difference between the failing and running testers. Do you have access to a running one?
You can revert MPL if you still want to, but I honestly don't know how else (on the MPL side) we can do this.
I guess that’s really the question I plan to investigate (while the develop testers are all working correctly).
I restored my modularization branch and incorporated the commit from https://github.com/boostorg/mpl/pull/12 just in case so that the work is not lost. If you revert MPL, you can pull from that branch instead of develop to reproduce the failing behavior. Let me know if I should do anything else.
On Sat, Sep 20, 2014 at 12:03:07PM +0400, Andrey Semashev wrote:
On Saturday 20 September 2014 00:26:26 Belcourt, Kenneth wrote:
On Sep 19, 2014, at 12:27 AM, Andrey Semashev <andrey.semashev@gmail.com> wrote:
Kenneth,
dba Noel
Sorry, I took the name from your email address. I've never heard of "dba" before, I suppose its your alias for Boost activity?
It appears to be an initialism in some English-speaking countries for "doing business as", being the name they prefer their correspondence to be addressed to, but may be used to have to disclose for reasons that it's not their "true" moniker. http://en.wikipedia.org/wiki/Doing_business_as -- Lars Viklund | zao@acc.umu.se
participants (13)
-
Andrey Semashev -
Belcourt, Kenneth -
Edward Diener -
Gavin Lambert -
Lars Viklund -
Niall Douglas -
Peter Dimov -
Rene Rivera -
Rob Stewart -
Robert Ramey -
Stephen Kelly -
Suckow, Thomas J -
Vladimir Prus