[modular-boost] Questions about development procedures and more
Hi, Trying to start using modular boost (and git for that matter). I have several questions about this. These are in no particular order: 1. Why does ./b2 headers on windows creates hardlink for each file instead of directories? The process takes forever. 2. What the story about cmake based build? What state is it in? Can I try it now? How? Can I help with anything to push this further? I got a bit of experience with cmake since last time I see n this discussed. 5. What the status of testing of the new tree? How is it setup? Where can I read it? Specifically I am interested in what being tested against what. 4. First command to switch to develop branch produces warning(?) ...\modular-boost\libs\test>git checkout develop Previous HEAD position was 954d3ab... Thread: merge [81791]-tools dir missing Branch develop set up to track remote branch develop from origin. Switched to a new branch 'develop' Should I worry? 5. After I switched to development branch headers directory still "points to" master branch files. Actually I am not sure what it points to now since master files are nowhere anymore. How should I switch headers? 6. What the corresponding action with git, which does what svn up from root of tree did before? What I am after is comamnd(s) which updates other libs, but keep my lib as is (maybe merge changes from other developers). If "git submodule update" is one of these commands, why does it takes forever on fresh just cloned boost tree? 7. How do I run the test which will check how my changes affect other libraries (their master branch at least). Should I just run the test from root in my local copy? How do I test against develop branch of other libs and develop branch of other libs against me? 8. Dependency specification. This actually a big question, which can be split into several: 8.1 Is there place where I can see "released versions" for particular library/submodule? 8.2. Is there way to tell which other boost submodules I depend on? 8.3 Can I specify specific release version I want to depend on (per submodule)? 8.4 Can I somehow only checkout/pull the submodules I depend on? 8.5 Can I check which submodules depend on me and checkout only those? 8.6 Is there place where I can see which submodule version constitute specific boost release? 9. One of the wiki pages refers to "git submodule summary" command. I do not understand what this command supposed to do. 10. How do I create private branch? IOW branch which is not publicized on github, but one I can share with the team? 11. This page is not clear to me: https://svn.boost.org/trac/boost/wiki/PostCvtMergePoint What do I need to do to merge my development branch into master, given that it is significantly differ now (at the time of svn->git conversion)? Thank you. Gennadiy
On 2 February 2014 09:30, Gennadiy Rozental
Hi,
Trying to start using modular boost (and git for that matter). I have several questions about this. These are in no particular order:
It's best to ask unrelated questions in separate mails, and not to combine so many questions in a single mail. It makes it easier to track what has and hasn't been answered, and makes it less work for someone to answer. It will also be easier to find answers in the archives.
1. Why does ./b2 headers on windows creates hardlink for each file instead of directories? The process takes forever.
That's a question for the build list.
2. What the story about cmake based build? What state is it in? Can I try it now? How? Can I help with anything to push this further? I got a bit of experience with cmake since last time I see n this discussed.
If you're lucky a cmake person might read this, but there would be more chance if you sent an email with 'cmake' in the title.
5. What the status of testing of the new tree? How is it setup? Where can I read it? Specifically I am interested in what being tested against what.
It's pretty much as it was before. But with 'develop' and 'master' instead of 'trunk' and 'branches/release'.
4. First command to switch to develop branch produces warning(?)
...\modular-boost\libs\test>git checkout develop Previous HEAD position was 954d3ab... Thread: merge [81791]-tools dir missing
"tools dir missing" is the latest commit on the master branch, not a warning.
5. After I switched to development branch headers directory still "points to" master branch files. Actually I am not sure what it points to now since master files are nowhere anymore. How should I switch headers?
It should happen automatically when you build, but won't if the inclusion of headers is obscured by macros (i.e. #include MACRO). You can run "b2 headers" again. If we were using soft links we wouldn't have this problem, I thought there was consensus to do that, but no one seems to be doing it.
6. What the corresponding action with git, which does what svn up from root of tree did before? What I am after is comamnd(s) which updates other libs, but keep my lib as is (maybe merge changes from other developers). If "git submodule update" is one of these commands, why does it takes forever on fresh just cloned boost tree?
You can configure git to leave your module alone when updating, see the 'gitmodules' man page. Something like: git config submodule.test.update none I haven't tested that, so it might be wrong. I'm finding it easy enough to check out the branch I want every time. 'git submodule update' takes a long time as a fresh tree because it has to clone over a 100 modules.
7. How do I run the test which will check how my changes affect other libraries (their master branch at least). Should I just run the test from root in my local copy? How do I test against develop branch of other libs and develop branch of other libs against me?
Let the automated testing do it for you? If I wanted to do this locally, I'd just use a separate clone of master and develop. I think you can use 'git clone --recursive -b develop url' to get a develop copy.
8. Dependency specification. This actually a big question, which can be split into several: 8.1 Is there place where I can see "released versions" for particular library/submodule?
Tags.
8.2. Is there way to tell which other boost submodules I depend on?
Bcp.
8.3 Can I specify specific release version I want to depend on (per submodule)?
You can do this manually, or with a bit of scripting.
8.4 Can I somehow only checkout/pull the submodules I depend on?
By combing the last two answers.
8.5 Can I check which submodules depend on me and checkout only those?
I think you'll need to work that out for yourself.
8.6 Is there place where I can see which submodule version constitute specific boost release?
Tags again.
9. One of the wiki pages refers to "git submodule summary" command. I do not understand what this command supposed to do.
I can't really answer this without a link to the page.
10. How do I create private branch? IOW branch which is not publicized on github, but one I can share with the team?
You need to have a private repo somewhere. This costs money with github, but you can create one for free with bitbucket. I think there are other providers which offer them as well (I saw an advert somewhere, but I forget the details).
11. This page is not clear to me: https://svn.boost.org/trac/boost/wiki/PostCvtMergePoint
What do I need to do to merge my development branch into master, given that it is significantly differ now (at the time of svn->git conversion)?
Creating a merge base for test isn't really practical as the two branches are completely different. You'll just need to manage the branches manually until you're reading to release the new version. Either that, or you could create a new branch for the new version, and make develop identical to master.
Daniel James
2. What the story about cmake based build? What state is it in? Can I try it now? How? Can I help with anything to push this further? I got a bit of experience with cmake since last time I see n this discussed.
If you're lucky a cmake person might read this, but there would be more chance if you sent an email with 'cmake' in the title.
Ok. I am sending separate post ;) Essentially I am looking for procedure to use cmake to build modular boost locally on Windows. Do we have cmake rules to generate quickbook docs? Gennadiy
-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Gennadiy Rozental Sent: Sunday, February 02, 2014 10:30 PM To: boost@lists.boost.org Subject: [boost] [modular-boost] cmake
Daniel James
writes:
Do we have cmake rules to generate quickbook docs? This isn't the answer you want ;-) No - we use bjam. And it *must* be possible to use bjam for others to rebuild the docs. Paul --- Paul A. Bristow, Prizet Farmhouse, Kendal LA8 8AB UK +44 1539 561830 07714330204 pbristow@hetp.u-net.com
Paul A. Bristow
writes:
Do we have cmake rules to generate quickbook docs?
This isn't the answer you want
No - we use bjam.
I understand we use bjam now,` but is the support for quickbook in cmake based build?
And it *must* be possible to use bjam for others to rebuild the docs.
Only until we switch to cmake, right? Gennadiy
AMDG On 02/03/2014 04:38 PM, Gennadiy Rozental wrote:
Paul A. Bristow
writes: And it *must* be possible to use bjam for others to rebuild the docs. Only until we switch to cmake, right?
Even if this were to happen someday, it's certainly not in the near future. In Christ, Steven Watanabe
Daniel James
3. What the status of testing of the new tree? How is it setup? Where can I read it? Specifically I am interested in what being tested against what.
It's pretty much as it was before. But with 'develop' and 'master' instead of 'trunk' and 'branches/release'.
a) Is the document explaining our testing setup? b) What development branch testing tests? I think this is the major question. In my opinion we should test development branch of each library against master branch of all other libs. Is this the case now? c) Is the plan for custom/"on demand" testing? So that one can request testing of develop branch against develop branch of other libs or some feature/hotfix branch against master branch of other libs? Gennadiy
On 3 February 2014 00:51, Gennadiy Rozental
Daniel James
writes: 3. What the status of testing of the new tree? How is it setup? Where can I read it? Specifically I am interested in what being tested against what.
It's pretty much as it was before. But with 'develop' and 'master' instead of 'trunk' and 'branches/release'.
a) Is the document explaining our testing setup?
http://www.boost.org/development/testing.html http://www.boost.org/development/running_regression_tests.html
b) What development branch testing tests?
I think this is the major question. In my opinion we should test development branch of each library against master branch of all other libs. Is this the case now?
No. It's the same as it was before. It's not because no one can conceive of something better, it's because we don't have the resources to do something better.
c) Is the plan for custom/"on demand" testing?
So that one can request testing of develop branch against develop branch of other libs or some feature/hotfix branch against master branch of other libs?
Not really. It has been mentioned before, but as far as I know no one is working on it.
Daniel James
5. After I switched to development branch headers directory still "points to" master branch files. Actually I am not sure what it points to now since master files are nowhere anymore. How should I switch headers?
It should happen automatically when you build, but won't if the inclusion of headers is obscured by macros (i.e. #include MACRO). You
I do not follow this. Are you saying something is going to parse the code and setup the links? During which build?
can run "b2 headers" again. If we were using soft links we wouldn't have this problem, I thought there was consensus to do that, but no one seems to be doing it.
There is no soft links on Windows. Actually I think if would use hardlinks to directories the issue would be resolved automatically.
6. What the corresponding action with git, which does what svn up from root of tree did before? What I am after is comamnd(s) which updates other libs, but keep my lib as is (maybe merge changes from other developers). If "git submodule update" is one of these commands, why does it takes forever on fresh just cloned boost tree?
You can configure git to leave your module alone when updating, see the 'gitmodules' man page. Something like:
git config submodule.test.update none
Where can I read about this?
I haven't tested that, so it might be wrong. I'm finding it easy enough to check out the branch I want every time.
What if I have local changes, but want to update to HEAD on master branch of other libs? I think this is very common need and we should document steps one needs to take for this.
'git submodule update' takes a long time as a fresh tree because it has to clone over a 100 modules.
I was running on freshly cloned tree. What else does it needs to clone?
7. How do I run the test which will check how my changes affect other libraries (their master branch at least). Should I just run the test from root in my local copy? How do I test against develop branch of other libs and develop branch of other libs against me?
Let the automated testing do it for you? If I wanted to do this
Do we have a test on demand? In any case I might want to do a local test
locally, I'd just use a separate clone of master and develop. I think you can use 'git clone --recursive -b develop url' to get a develop copy.
Can we setup some kind of scripts to help developers with routine tasks like this? I probably do not want to keep 2 trees around, but rather switch back and forth when necessary. Gennadiy
Hello Gennadiy,
can run "b2 headers" again. If we were using soft links we wouldn't have this problem, I thought there was consensus to do that, but no one seems to be doing it.
GR> There is no soft links on Windows.
1. There are soft links in Windows. See documentation for mklink /D. 2. This is exactly what b2 tries to do: create soft links. Only it fails, since in Windows you need elevated privileges by default to do this. Then b2 falls back to creating hard links to files, since it cannot create soft links to directories. -- Vyacheslav Andrejev
On 2/3/2014 11:00 AM, Vyacheslav Andrejev wrote:
Hello Gennadiy,
can run "b2 headers" again. If we were using soft links we wouldn't have this problem, I thought there was consensus to do that, but no one seems to be doing it.
GR> There is no soft links on Windows.
1. There are soft links in Windows. See documentation for mklink /D.
mklink without any option creates a file symbolic link.
2. This is exactly what b2 tries to do: create soft links. Only it fails, since in Windows you need elevated privileges by default to do this. Then b2 falls back to creating hard links to files, since it cannot create soft links to directories.
It does not test whether the user has administrator priveleges or not on Windows in order to create file symbolic links. I have brought this issue many times before in various posts. The 'b2 headers' command should always create symbolic links if possible, not hard links which are problematical in our modular-boos setup. Each time I have brought up this issue I have been told we are waiting for modular-boost issues itself to be fully resolved. We are evidently still waiting.
Hello Edward, ED> mklink without any option creates a file symbolic link. I know. The issue is in the links to directories. ED> It does not test whether the user has administrator priveleges or ED> not on Windows in order to create file symbolic links. Yes, it doesn't. I didn't say it does. Moreover if it did, it would be a big mistake, since the rights to create symbolic links are defined in the security policies. These policies only by default are narrowed to Administrators. It would quite silly to assume that nobody ever changes the default. ED> I have brought this issue many times before in various posts. The ED> 'b2 headers' command should always create symbolic links if ED> possible, not hard links which are problematical in our modular-boos ED> setup. Each time I have brought up this issue I have been told we ED> are waiting for modular-boost issues itself to be fully resolved. We ED> are evidently still waiting. if [ os.name ] = NT { MKLINK_OR_DIR on $(target) = mklink /D \"$(target)\" \"$(relative)\" ; } I hope you recognize /D. -- Vyacheslav Andrejev
On 02/03/2014 05:29 PM, Edward Diener wrote:
On 2/3/2014 11:00 AM, Vyacheslav Andrejev wrote:
Hello Gennadiy,
can run "b2 headers" again. If we were using soft links we wouldn't have this problem, I thought there was consensus to do that, but no one seems to be doing it.
GR> There is no soft links on Windows.
1. There are soft links in Windows. See documentation for mklink /D.
mklink without any option creates a file symbolic link.
2. This is exactly what b2 tries to do: create soft links. Only it fails, since in Windows you need elevated privileges by default to do this. Then b2 falls back to creating hard links to files, since it cannot create soft links to directories.
It does not test whether the user has administrator priveleges or not on Windows in order to create file symbolic links.
What test are you suggesting and what should b2 do given each result of that test?
I have brought this issue many times before in various posts. The 'b2 headers' command should always create symbolic links if possible, not hard links which are problematical in our modular-boos setup. Each time I have brought up this issue I have been told we are waiting for modular-boost issues itself to be fully resolved. We are evidently still waiting.
This thread is more about credentials on windows than the more general questions we have debated on the preference on use of hardlinks over symlinks to files in current "b2 headers". I think there are more or less consensus that it will be better to use symlinks also for files, but Beman asked for requirements and time to let the modularization dust settle before we made changes. I have not got any feedback on the requirements I posted, and there have as far as I have noted been a few, but not many reported real problems with hardlinks. So rushing this may not be warranted. Nevertheless, that should not prevent you from trying preference for symlinks in "b2 headers" if you need to, and report results. End-note: this thread started with a problem related to symlinks not working as expected on Windows, so the changes you say we are waiting on would likely not resolve anything here. -- Bjørn
On 2/3/2014 3:49 PM, Bjørn Roald wrote:
On 02/03/2014 05:29 PM, Edward Diener wrote:
On 2/3/2014 11:00 AM, Vyacheslav Andrejev wrote:
Hello Gennadiy,
can run "b2 headers" again. If we were using soft links we wouldn't have this problem, I thought there was consensus to do that, but no one seems to be doing it.
GR> There is no soft links on Windows.
1. There are soft links in Windows. See documentation for mklink /D.
mklink without any option creates a file symbolic link.
2. This is exactly what b2 tries to do: create soft links. Only it fails, since in Windows you need elevated privileges by default to do this. Then b2 falls back to creating hard links to files, since it cannot create soft links to directories.
It does not test whether the user has administrator priveleges or not on Windows in order to create file symbolic links.
What test are you suggesting and what should b2 do given each result of that test?
I do not know. I only know that as a user with administrator rights on Windows 7 that I can create symbolic links. Yet when using 'b2 headers' the links to files in the modular-boost boost subdirectory are hard links.
I have brought this issue many times before in various posts. The 'b2 headers' command should always create symbolic links if possible, not hard links which are problematical in our modular-boos setup. Each time I have brought up this issue I have been told we are waiting for modular-boost issues itself to be fully resolved. We are evidently still waiting.
This thread is more about credentials on windows than the more general questions we have debated on the preference on use of hardlinks over symlinks to files in current "b2 headers". I think there are more or less consensus that it will be better to use symlinks also for files, but Beman asked for requirements and time to let the modularization dust settle before we made changes. I have not got any feedback on the requirements I posted, and there have as far as I have noted been a few, but not many reported real problems with hardlinks.
The problem with hardlinks has already been reported and commented on. Do I or anyone else really need to explain this again vis-a-vis modular-boost ?
So rushing this may not be warranted. Nevertheless, that should not prevent you from trying preference for symlinks in "b2 headers" if you need to, and report results.
Why not have someone knowledgable with Boost Build actually make the changes in the 'develop' branch to 'b2 headers' so that symbolic links get created on Windows rather than hard links when the end-user's administrator priveleges allow it ? I would be glad to test this out. I assume that you may be knowledgeable enough with Boost build to do this, but I am not.
End-note: this thread started with a problem related to symlinks not working as expected on Windows, so the changes you say we are waiting on would likely not resolve anything here.
It started with Gennadiy Rozental asking about a number of issues with Modular Boost, one of which is: "1. Why does ./b2 headers on windows creates hardlink for each file instead of directories? The process takes forever."
On 02/03/2014 10:37 PM, Edward Diener wrote:
On 2/3/2014 3:49 PM, Bjørn Roald wrote:
On 02/03/2014 05:29 PM, Edward Diener wrote:
On 2/3/2014 11:00 AM, Vyacheslav Andrejev wrote:
Hello Gennadiy,
can run "b2 headers" again. If we were using soft links we wouldn't have this problem, I thought there was consensus to do that, but no one seems to be doing it.
GR> There is no soft links on Windows.
1. There are soft links in Windows. See documentation for mklink /D.
mklink without any option creates a file symbolic link.
2. This is exactly what b2 tries to do: create soft links. Only it fails, since in Windows you need elevated privileges by default to do this. Then b2 falls back to creating hard links to files, since it cannot create soft links to directories.
It does not test whether the user has administrator priveleges or not on Windows in order to create file symbolic links.
What test are you suggesting and what should b2 do given each result of that test?
I do not know. I only know that as a user with administrator rights on Windows 7 that I can create symbolic links.
OK, but that does not help anything. Given that b2 checked your privileges, it would not provide any information it does not already have by simply testing if it succeeds making symbolic and hard links.
Yet when using 'b2 headers' the links to files in the modular-boost boost subdirectory are hard links.
Yes, and b2 does not need to do any new test to gain the needed information. It already knows if symbolic links works or not. T
I have brought this issue many times before in various posts. The 'b2 headers' command should always create symbolic links if possible, not hard links which are problematical in our modular-boos setup. Each time I have brought up this issue I have been told we are waiting for modular-boost issues itself to be fully resolved. We are evidently still waiting.
This thread is more about credentials on windows than the more general questions we have debated on the preference on use of hardlinks over symlinks to files in current "b2 headers". I think there are more or less consensus that it will be better to use symlinks also for files, but Beman asked for requirements and time to let the modularization dust settle before we made changes. I have not got any feedback on the requirements I posted, and there have as far as I have noted been a few, but not many reported real problems with hardlinks.
The problem with hardlinks has already been reported and commented on.
I think I said pretty much the same.
Do I or anyone else really need to explain this again vis-a-vis modular-boost ?
Not to me or anybody else as far as I am aware. There are still some caution as there is a lot of changes for a lot of people, so risking breaking something the build which sort of works right to fix this is not a priority.
So rushing this may not be warranted. Nevertheless, that should not prevent you from trying preference for symlinks in "b2 headers" if you need to, and report results.
Why not have someone knowledgable with Boost Build actually make the changes in the 'develop' branch to 'b2 headers' so that symbolic links get created on Windows rather than hard links when the end-user's administrator priveleges allow it ?
Yes expect I don't think anybody can just have somebody else doing what they like around here. That is not how it works.
I would be glad to test this out. I assume that you may be knowledgeable enough with Boost build to do this, but I am not.
Well, I am an ignorant bjam hacker, but if you are brave and edit tools/build/src/tools/link.jam, you will find: rule do-file-link { local target = [ path.native [ path.relative-to [ path.pwd ] $(<) ] ] ; local source = [ path.native [ path.relative-to [ path.pwd ] $(>) ] ] ; LOCATE on $(target) = . ; DEPENDS $(.current-target) : $(target) ; if $(.can-hardlink) = true { DEPENDS $(target) : $(source) ; link.hardlink $(target) : $(source) ; } else if $(.can-symlink) = true { link.mklink $(target) : $(source) ; } else { DEPENDS $(target) : $(source) ; common.copy $(target) : $(source) ; } } so simply swap the order of the: if $(.can-hardlink) = true { DEPENDS $(target) : $(source) ; link.hardlink $(target) : $(source) ; } and the: if $(.can-symlink) = true { link.mklink $(target) : $(source) ; } blocks should do it -- I think.
End-note: this thread started with a problem related to symlinks not working as expected on Windows, so the changes you say we are waiting on would likely not resolve anything here.
It started with Gennadiy Rozental asking about a number of issues with Modular Boost, one of which is:
"1. Why does ./b2 headers on windows creates hardlink for each file instead of directories? The process takes forever."
The change above, which I think is what you have asked for all along will not change Gennadiy's slow "b2 headers" build. To speed that up he needs to change credentials to get symlinks to work. And the key to improved performence is symlinks to directories, not to files. I would be surprised if the change above will improve performance at all, I sort of expect it to be a tad slower, but I may be wrong. -- Bjørn
Edward Diener
"1. Why does ./b2 headers on windows creates hardlink for each file instead of directories? The process takes forever."
And answer I was after is not "Let's create symlinks instead". It was "let's create any kind of link to *directories*" ;) I am sure there must be a reason why we are not doing it now. I just would like to know it, since I missed it in our original discussions. Gennadiy
AMDG On 02/03/2014 04:53 PM, Gennadiy Rozental wrote:
Edward Diener
writes: "1. Why does ./b2 headers on windows creates hardlink for each file instead of directories? The process takes forever."
And answer I was after is not "Let's create symlinks instead". It was "let's create any kind of link to *directories*" ;)
Boost.Build will use directory symlinks if it can. If they're not supported, we can fall back on using Junctions on Windows. I worked on this a bit today. It's not too hard, but the b2 builtin, READLINK, needs to be updated to handle junctions.
I am sure there must be a reason why we are not doing it now.
The only reason is that it's yet another variation and I haven't gotten around to it yet.
I just would like to know it, since I missed it in our original discussions.
In Christ, Steven Watanabe
AMDG On 02/03/2014 08:29 AM, Edward Diener wrote:
I have brought this issue many times before in various posts. The 'b2 headers' command should always create symbolic links if possible, not hard links which are problematical in our modular-boos setup. Each time I have brought up this issue I have been told we are waiting for modular-boost issues itself to be fully resolved. We are evidently still waiting.
The regression tests are running now. I really didn't want to touch anything (especially related to Boost.Build) before that. I'll take care of this tonight. In Christ, Steven Watanabe
On 3 February 2014 01:26, Gennadiy Rozental
Daniel James
writes: 5. After I switched to development branch headers directory still "points to" master branch files. Actually I am not sure what it points to now since master files are nowhere anymore. How should I switch headers?
It should happen automatically when you build, but won't if the inclusion of headers is obscured by macros (i.e. #include MACRO). You
I do not follow this. Are you saying something is going to parse the code and setup the links? During which build?
Boost build searches for includes when compiling C++ files in order to check if any have been updated, I believe it checks if the original file has been updated when doing that in order to update the link. [snipped: question being discussed by others]
6. What the corresponding action with git, which does what svn up from root of tree did before? What I am after is comamnd(s) which updates other libs, but keep my lib as is (maybe merge changes from other developers). If "git submodule update" is one of these commands, why does it takes forever on fresh just cloned boost tree?
You can configure git to leave your module alone when updating, see the 'gitmodules' man page. Something like:
git config submodule.test.update none
Where can I read about this?
The gitmodules man page. https://www.kernel.org/pub/software/scm/git/docs/gitmodules.html
I haven't tested that, so it might be wrong. I'm finding it easy enough to check out the branch I want every time.
What if I have local changes, but want to update to HEAD on master branch of other libs? I think this is very common need and we should document steps one needs to take for this.
You could create a wiki page.
'git submodule update' takes a long time as a fresh tree because it has to clone over a 100 modules.
I was running on freshly cloned tree. What else does it needs to clone?
All the libraries and tools. If you have a clone of every module and it's still slow, then I have no idea. You haven't really explained what you've done and it's not that slow for me.
7. How do I run the test which will check how my changes affect other libraries (their master branch at least). Should I just run the test from root in my local copy? How do I test against develop branch of other libs and develop branch of other libs against me?
Let the automated testing do it for you? If I wanted to do this
Do we have a test on demand?
No.
locally, I'd just use a separate clone of master and develop. I think you can use 'git clone --recursive -b develop url' to get a develop copy.
Can we setup some kind of scripts to help developers with routine tasks like this?
Yes you can.
Daniel James
Boost build searches for includes when compiling C++ files in order to check if any have been updated, I believe it checks if the original file has been updated when doing that in order to update the link.
Hmmm.. I do not think I like this approach. What if I am not using Boost.Build (I have VS projects for Boost.Test for example)? What if I just want to see/grep the content of the file. Any chance we can reconsider? This look like a very bad setup to me. In my opinion this should be separate manual step or something which is done when one is updating boost tree. Interestingly, if we switch to links to directories, we do not need to do anything in majority of the cases. Only when new library is added or old is removed we need to chansge boost/ content.
'git submodule update' takes a long time as a fresh tree because it has to clone over a 100 modules.
I was running on freshly cloned tree. What else does it needs to clone?
All the libraries and tools. If you have a clone of every module and it's still slow, then I have no idea. You haven't really explained what you've done and it's not that slow for me.
I followed the procedures on a wiki. I cloned *full* boost.org tree and run "git submodule update" command on it.
What if I have local changes, but want to update to HEAD on master branch of other libs? I think this is very common need and we should document steps one needs to take for this.
You could create a wiki page. ...
Can we setup some kind of scripts to help developers with routine tasks like this?
Yes you can.
Daniel, I was not suggesting that you should be doing this. I was merely suggesting that we *need* to do this. Do we have anyone working on modular boost migration actively? Unfortunately I am not in a position to volontier at the moment, but I think this is important area and we need someone to be responcible for it. Regards, Gennadiy
On 4 February 2014 01:08, Gennadiy Rozental
Daniel James
writes: Boost build searches for includes when compiling C++ files in order to check if any have been updated, I believe it checks if the original file has been updated when doing that in order to update the link.
Hmmm.. I do not think I like this approach. What if I am not using Boost.Build (I have VS projects for Boost.Test for example)? What if I just want to see/grep the content of the file.
Any chance we can reconsider? This look like a very bad setup to me. In my opinion this should be separate manual step or something which is done when one is updating boost tree.
You can do it manually, just run 'b2 headers'. Hopefully, you can configure Visual Studio to do that before builds.
Interestingly, if we switch to links to directories, we do not need to do anything in majority of the cases. Only when new library is added or old is removed we need to chansge boost/ content.
Others have responded about windows, which I know nothing about. On linux directories are now soft links if they contain headers from only one module, but if they contain headers from multiple modules then they can't be a soft link.
'git submodule update' takes a long time as a fresh tree because it has to clone over a 100 modules.
I was running on freshly cloned tree. What else does it needs to clone?
All the libraries and tools. If you have a clone of every module and it's still slow, then I have no idea. You haven't really explained what you've done and it's not that slow for me.
I followed the procedures on a wiki. I cloned *full* boost.org tree and run "git submodule update" command on it.
I guess it's just slow on windows then. Btw. when referring to a wiki page, please link to it.
What if I have local changes, but want to update to HEAD on master branch of other libs? I think this is very common need and we should document steps one needs to take for this.
You could create a wiki page. ...
Can we setup some kind of scripts to help developers with routine tasks like this?
Yes you can.
Daniel,
I was not suggesting that you should be doing this. I was merely suggesting that we *need* to do this.
Do we have anyone working on modular boost migration actively? Unfortunately I am not in a position to volontier at the moment, but I think this is important area and we need someone to be responcible for it.
Beman sent an email asking for volunteers a little while ago, but no one responded. The build people are doing the build things, and the test people are doing the test things, but no one is really doing the more general things at the moment.
Daniel James
8. Dependency specification. This actually a big question, which can be split into several: 8.1 Is there place where I can see "released versions" for particular library/submodule?
Tags.
Can you please provide a bit more details. For example, if I want to know what are released versions of boost.chrono what should I do?
8.2. Is there way to tell which other boost submodules I depend on?
Bcp.
What is the specific command?
8.3 Can I specify specific release version I want to depend on (per submodule)?
You can do this manually, or with a bit of scripting.
This is not really adequate answer IMO. What I was after is to specify my dependency in my Jamfile/CMakefile and make testing framework to figure out the rest automatically. Is this realistic/in our plans? Even if we are talking only about local testing, I think we need to develop set of scripts to help boost developers with these mundane tasks.
8.4 Can I somehow only checkout/pull the submodules I depend on?
By combing the last two answers.
The same here: I think we need a scripts to do this.
8.5 Can I check which submodules depend on me and checkout only those?
I think you'll need to work that out for yourself.
Why is this difficult? Do we have an information about dependency graph somewhere? And again I think we need a scripts to do this.
8.6 Is there place where I can see which submodule version constitute specific boost release?
Tags again.
Script would be helpful. Gennadiy
On 3 February 2014 01:33, Gennadiy Rozental
Daniel James
writes: 8. Dependency specification. This actually a big question, which can be split into several: 8.1 Is there place where I can see "released versions" for particular library/submodule?
Tags.
Can you please provide a bit more details. For example, if I want to know what are released versions of boost.chrono what should I do?
cd libs/chrono git tag | grep '^boost-[0-9.]*$'
8.2. Is there way to tell which other boost submodules I depend on?
Bcp.
What is the specific command?
I don't know. The bcp documentation can help.
8.3 Can I specify specific release version I want to depend on (per submodule)?
You can do this manually, or with a bit of scripting.
This is not really adequate answer IMO.
Well, it's the best one I have.
What I was after is to specify my dependency in my Jamfile/CMakefile and make testing framework to figure out the rest automatically. Is this realistic/in our plans?
I don't think so. There was ryppl, but that doesn't seem to be making much progress.
Even if we are talking only about local testing, I think we need to develop set of scripts to help boost developers with these mundane tasks.
8.4 Can I somehow only checkout/pull the submodules I depend on?
By combing the last two answers.
The same here: I think we need a scripts to do this.
8.5 Can I check which submodules depend on me and checkout only those?
I think you'll need to work that out for yourself.
Why is this difficult? Do we have an information about dependency graph somewhere? And again I think we need a scripts to do this.
A few people have done this in different ways, you'll need to search the archives as I forget the details.
8.6 Is there place where I can see which submodule version constitute specific boost release?
Tags again.
Script would be helpful.
You can get the hashes using: git ls-tree -r boost-1.53.0 | grep '^160000 commit'
Hello Gennadiy, GR> 1. Why does ./b2 headers on windows creates hardlink for each file GR> instead of directories? The process takes forever. tools/build/src/tools/link.jam is responsible for this behavior. In particular there is a rule that creates links to directories: rule do-link { local target = [ path.native [ path.relative-to [ path.pwd ] $(<) ] ] ; local source = [ path.native [ path.relative-to [ path.pwd ] $(>) ] ] ; local relative = [ path.native [ path.relative-to [ path.parent $(<) ] $(>) ] ] ; if ! [ on $(target) return $(MKLINK_OR_DIR) ] { LOCATE on $(target) = . ; DEPENDS $(.current-target) : $(target) ; mklink-or-dir $(target) : $(source) ; } if [ os.name ] = NT { MKLINK_OR_DIR on $(target) = mklink /D \"$(target)\" \"$(relative)\" ; } else { MKLINK_OR_DIR on $(target) = ln -s $(relative) $(target) ; } } As you can see this rule uses mklink /D to create a symbolic link. However by default in Windows users have no rights to create symbolic links. You can try to change /D to /J. I personally solved the problem by using an alternative command mklnk that exists in Take Command shell I use. -- Vyacheslav Andrejev
Vyacheslav Andrejev
Hello Gennadiy,
GR> 1. Why does ./b2 headers on windows creates hardlink for each file GR> instead of directories? The process takes forever.
tools/build/src/tools/link.jam is responsible for this behavior. In particular there is a rule that creates links to directories:
I do not really want to delve into details of our makesystem. Is there someone who supports this now?
As you can see this rule uses mklink /D to create a symbolic link. However by default in Windows users have no rights to create symbolic links. You
Really? I did not know that. I used junction.exe and Far to create links both on XP and Win7 and it worked fine for both files and directories.
can try to change /D to /J. I personally solved the problem by using an alternative command mklnk that exists in Take Command shell I use.
1. I need this to work in Far's shell, which is I believe regular cmd for the most part. 2. There is indeed an issues with directory links on XP (during deletion it deletes files), but I do not believe we need to care about this. Gennadiy
Hello Gennadiy, GR> I do not really want to delve into details of our makesystem. But you can try to change a couple of symbols in the file I pointed you to. It doesn't hurt to try, right? GR> Really? I did not know that. I used junction.exe and Far to create GR> links both on XP and Win7 and it worked fine for both files and GR> directories. Really. The name “junction.exe” is taletelling: it creates NTFS junction points, not symbolic links for directories. In contrast to junction points, by default only Administrators can create symbolic links. You can easily check it in the local security policies. b2 tries to create symbolic links by running mklink /D, that is why I suggested to change /D to /J, which would tell mklink to create junction points, which in turn doesn't require special privileges. GR> 1. I need this to work in Far's shell, which is I believe regular GR> cmd for the GR> most part. Far is a file manager, not a command processor. Yes, it is cmd for ALL parts. GR> I do not believe we need to care about this. You're absolutely right. -- Vyacheslav Andrejev
Vyacheslav Andrejev
Hello Gennadiy,
GR> I do not really want to delve into details of our makesystem.
But you can try to change a couple of symbols in the file I pointed you
to.
It doesn't hurt to try, right?
GR> Really? I did not know that. I used junction.exe and Far to create GR> links both on XP and Win7 and it worked fine for both files and GR> directories.
Really. The name “junction.exe” is taletelling: it creates NTFS junction points, not symbolic links for directories. In contrast to junction
It does. I am not familiar enough with Boost.Build syntax and in no position to learn it. Regardless I was looking for someone who IS familiar with it to make the change everybody can use. points,
by default only Administrators can create symbolic links. You can easily check it in the local security policies.
b2 tries to create symbolic links by running mklink /D, that is why I suggested to change /D to /J, which would tell mklink to create junction points, which in turn doesn't require special privileges.
Ok. What prevents us from changing bjam to always use hardlinks for directories? Gennadiy
Hello Gennadiy, GR> Ok. What prevents us from changing bjam to always use hardlinks for GR> directories? My guess would be what prevent us is their limitations. They won't work on network drives, for example. Only pure NTFS on local HDD will work. -- Vyacheslav Andrejev
On 2/3/2014 7:44 PM, Gennadiy Rozental wrote:
Vyacheslav Andrejev
writes: Hello Gennadiy,
GR> I do not really want to delve into details of our makesystem.
But you can try to change a couple of symbols in the file I pointed you
to.
It doesn't hurt to try, right?
It does. I am not familiar enough with Boost.Build syntax and in no position to learn it. Regardless I was looking for someone who IS familiar with it to make the change everybody can use.
GR> Really? I did not know that. I used junction.exe and Far to create GR> links both on XP and Win7 and it worked fine for both files and GR> directories.
Really. The name “junction.exe” is taletelling: it creates NTFS junction points, not symbolic links for directories. In contrast to junction points, by default only Administrators can create symbolic links. You can easily check it in the local security policies.
b2 tries to create symbolic links by running mklink /D, that is why I suggested to change /D to /J, which would tell mklink to create junction points, which in turn doesn't require special privileges.
Ok. What prevents us from changing bjam to always use hardlinks for directories?
The issue is not the links to directories. The issue is that there are hardlinks to individual files in the boost subtree using the logic of the current 'b2 headers' when run under Windows. This is a problem because if the original file in the libs directory has been changed, as for example via some 'pull' from Git, the corresponding hardlink still points to the old file. So we have a mismatch which causes errors in testing and Boost development. With a smybolic link to the file rather than a hardlink that particular problem goes away. As for directories we already correctly have symbolic links under Windows. There is no reason to change that. It is the hardlinked individual files which cause problems.
Hello Edward, ED> The issue is not the links to directories. The issue is that there ED> are hardlinks to individual files in the boost subtree using the ED> logic of the current 'b2 headers' when run under Windows. This is a ED> problem because if the original file in the libs directory has been ED> changed, as for example via some 'pull' from Git, the corresponding ED> hardlink still points to the old file. So we have a mismatch which ED> causes errors in testing and Boost development. The issue *being discussed* is in the links to directories. Here is a quote from the top post: GR> 1. Why does ./b2 headers on windows creates hardlink for each file GR> instead of directories? The process takes forever. I.e. in Gennadiy case b2 didn't create any links to directories at all and this is the problem he wanted to solve. You are talking about another, quite valid problem. Hoewever the existance of the problem you want to discuss doesn't mean that the one that *we* discuss doesn't exist. ED> As for directories we already correctly have symbolic links under ED> Windows. We don't. Symbolic links on directories in Windows since Vista are created only with elevated privileges or with turned off UAC and in both cases only if the user has the correspondent privelege in her security token. -- Vyacheslav Andrejev
Hello Edward, ED> The issue is not the links to directories. The issue is that there ED> are hardlinks to individual files in the boost subtree using the ED> logic of the current 'b2 headers' when run under Windows. This is a ED> problem because if the original file in the libs directory has been ED> changed, as for example via some 'pull' from Git, the corresponding ED> hardlink still points to the old file. So we have a mismatch which ED> causes errors in testing and Boost development. If you want to create symbolic links to files, open link.jam, locate there rule do-file-link and change the order of “if $(.can-symlink) = true” and “if $(.can-hardlink) = true”. It should look like this after the change: rule do-file-link { local target = [ path.native [ path.relative-to [ path.pwd ] $(<) ] ] ; local source = [ path.native [ path.relative-to [ path.pwd ] $(>) ] ] ; LOCATE on $(target) = . ; DEPENDS $(.current-target) : $(target) ; if $(.can-symlink) = true { link.mklink $(target) : $(source) ; } else if $(.can-hardlink) = true { DEPENDS $(target) : $(source) ; link.hardlink $(target) : $(source) ; } else { DEPENDS $(target) : $(source) ; common.copy $(target) : $(source) ; } } -- Vyacheslav Andrejev
On 03.02.2014 01:41, Gennadiy Rozental wrote:
Vyacheslav Andrejev
writes: Hello Gennadiy,
GR> 1. Why does ./b2 headers on windows creates hardlink for each file GR> instead of directories? The process takes forever.
tools/build/src/tools/link.jam is responsible for this behavior. In particular there is a rule that creates links to directories:
I do not really want to delve into details of our makesystem. Is there someone who supports this now?
Gennadiy, I don't think the problem is changing Boost.Build to do this or that. The problem at specifying, and having everybody agree, what is desired. Ideally, that would be part of modularization design, but it's not, so now somebody should figure this out. And this is a task for modulization designers/users, not for Boost.Build maintainers. Hope this clarifies the context. - Volodya
Vladimir Prus
I don't think the problem is changing Boost.Build to do this or that.
The problem at specifying, and having everybody agree, what is desired. Ideally, that would be part of modularization design, but it's not, so now somebody should figure this out. And this is a task for modulization designers/users, not for Boost.Build maintainers.
I understand. So, can someone who is is a position to make this decision explain why we do what we do? And why can't we use hardlinks to directories? Thank you. Gennadiy
Hello Steven, SW> There is no such thing. Most systems only SW> allow hard links to ordinary files. Hardlinks to directories (aka junctions) are supported in NTFS since Windows 2000. -- Vyacheslav Andrejev SW> AMDG SW> SW> On 02/03/2014 04:48 PM, Gennadiy Rozental wrote: SW>
And why can't we use hardlinks to directories?
SW> There is no such thing. Most systems only SW> allow hard links to ordinary files. SW> In Christ, SW> Steven Watanabe SW> _______________________________________________ SW> Unsubscribe & other changes: SW> http://lists.boost.org/mailman/listinfo.cgi/boost
Hello Gennadiy, GR> I do not really want to delve into details of our makesystem. You can run b2 with elevated privileges (aka as Administrator). It will make b2 detect symbolic links creation is possible. I just checked it. -- Vyacheslav Andrejev
On 02/03/2014 06:45 PM, Vyacheslav Andrejev wrote:
Hello Gennadiy,
GR> I do not really want to delve into details of our makesystem.
You can run b2 with elevated privileges (aka as Administrator). It will make b2 detect symbolic links creation is possible. I just checked it.
I tried with limited luck to check the web for information on credential requirements for creating symbolic links and junctions on various windows versions. Does anybody know where to look for that or some clever way to Google for this? I suspected there where other and more convenient ways to be allowed to create symbolic links than having Administrators membership or elevation the command. This is the most promising I found so far: Provide SeCreateSymbolicLinkPrivilege privilege to the user who is running the script. If the user is Administrator on the machine you may need to turn off UAC. But users with the right OS versions need to test so we can provide better info to all. I only have Linux at home and I can not do this at work. refs: http://stackoverflow.com/questions/15320550/secreatesymboliclinkprivilege-ig... http://msdn.microsoft.com/en-us/library/bb530410.aspx http://superuser.com/questions/148937/windows-server-2008-create-symbolic-li... http://superuser.com/questions/124679/how-do-i-create-a-link-in-windows-7-ho... -- Bjørn
participants (8)
-
Bjørn Roald -
Daniel James -
Edward Diener -
Gennadiy Rozental -
Paul A. Bristow -
Steven Watanabe -
Vladimir Prus -
Vyacheslav Andrejev