Monotone phasing out use of bits of boost due to having to build external libraries
There has recently been some discussion on the monotone development list about phasing out their use of boost libraries that require building external lib files (filesystem and regex). I thought people might find it interesting, so I'm posting some of the comments here: "Zack Weinberg" <zackw@panix.com> writes:
Anthony Williams <anthony_w.geo@yahoo.com> wrote:
"Zack Weinberg" <zackw@panix.com> writes:
Anthony Williams <anthony_w.geo@yahoo.com> wrote:
"Zack Weinberg" <zackw@panix.com> writes:
As of revision 9546b6ea3c29b0a8f63542f6d495efd33bec9add, Monotone no longer depends on boost::filesystem.
I don't understand. Monotone uses boost all over the place. Why remove uses of boost::filesystem and replace with hand-rolled stuff that will probably have problems, especially on Windows?
For logistic reasons, we want to stop using all Boost libraries that aren't header-only; I'll get into that below.
However, besides that, IMO boost::filesystem was always a poor fit with Monotone. We had another path abstraction on top of it to provide a type-system distinction between three different kinds of paths (file_path, bookkeeping_path, system_path); there were long-standing bugs in our handling of filesystem character sets that we could not fix because fs::path got in the way; the last straw was the interface changes between 1.33 and 1.34, which took away interfaces we needed (normalize_path, especially).
boost::regex is now the only component of Boost that we use that requires an external library for part of its code.
Is that the problem --- the external libraries? If so, then people at boost would like to know. There's always discussion about whether libraries should be header-only or not.
External libraries are definitely a problem for us. Due to boost's bizarro build system, it is not practical to bundle them in the monotone build, and "I can't get boost built" is among the top five problems people have trying to build monotone from source. If we use system-provided libraries, we have to match compilers and compile settings with them (see Debian bugs #404616 and #384565, for instance). We can't even bundle the header-only libraries we use, because version skew between them and the external boost libraries would be lethal.
If you're not going to be removing regex all together, and you're not going to write your own regex engine, so monotone will still depend on an external library (e.g. pcre), why try and remove the dependency on the boost regex library?
boost::regex in particular has the additional problem of depending on libicu (if built that way) whether or not the Unicode interfaces are used. (We don't.) Libicu is huge, and has a bug that (on at least Linux and Solaris) renders ineffective all our efforts to keep thread overhead out of the C++ runtime.
If you don't need ICU, don't use it --- I never use it, so I don't compile regex that way.
Yes, and I'd be a lot happier with boost::regex if the shared library were split into ASCII and Unicode modules. (Not enough happier with it to keep using it in Monotone, though, just to be clear.)
It is also, itself, five times larger than libpcre. Is that really a problem?
Yes. It means the statically-linked-against-boost binary we offer for download is larger than it could be. Not by a large proportion, but every bit helps.
Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL
Anthony Williams wrote:
There has recently been some discussion on the monotone development list about phasing out their use of boost libraries that require building external lib files (filesystem and regex). I thought people might find it interesting, so I'm posting some of the comments here:
"Zack Weinberg" <zackw@panix.com> writes:
Anthony Williams <anthony_w.geo@yahoo.com> wrote:
"Zack Weinberg" <zackw@panix.com> writes:
Anthony Williams <anthony_w.geo@yahoo.com> wrote:
As of revision 9546b6ea3c29b0a8f63542f6d495efd33bec9add, Monotone no longer depends on boost::filesystem. I don't understand. Monotone uses boost all over the place. Why remove uses of boost::filesystem and replace with hand-rolled stuff that will probably have
"Zack Weinberg" <zackw@panix.com> writes: problems, especially on Windows? For logistic reasons, we want to stop using all Boost libraries that aren't header-only; I'll get into that below.
However, besides that, IMO boost::filesystem was always a poor fit with Monotone. We had another path abstraction on top of it to provide a type-system distinction between three different kinds of paths (file_path, bookkeeping_path, system_path); there were long-standing bugs in our handling of filesystem character sets that we could not fix because fs::path got in the way; the last straw was the interface changes between 1.33 and 1.34, which took away interfaces we needed (normalize_path, especially).
I wonder if they know normalize_path is still there as long as BOOST_FILESYSTEM_NO_DEPRECATED is not defined. I would state that I think this needs to be re-added (probably as a standalone function) for 1.35. This was *the* reason I started using Boost.Filesystem in the first place. It's a shame but it highlights the issue that was discussed here a while ago about header only vs libraries. The complexity and compiling and distributing libraries is a real issue that has consequences. Thanks, Michael Marcin
Anthony Williams wrote:
There has recently been some discussion on the monotone development list about phasing out their use of boost libraries that require building external lib files (filesystem and regex). I thought people might find it interesting, so I'm posting some of the comments here:
"Zack Weinberg" <zackw@panix.com> writes:
< huge snip>
Anthony
I'm sure I should just sit on my hands and let this one go by. I don't know much about the Monotone version control project and I don't know who Zack is; however, I do know crazy excuses when I hear them. Perhaps boost::filesystem is not a good fit and therefore shouldn't be used. But people don't just stop using a library because the new version is a pain... they continue working with the old version and see if they can get the functionality put back in that they are missing. If the build system doesn't fit your needs, you package it so it works with your stuff. Any library set as large as boost is going to have portions not needed by a project. It seems perfectly reasonable (and almost expected) to extract the required bits and package those with Monotone. Surely writing the make file is trivial compared to rewriting Regex. Sounds like a typical NIH series of excuses. With the growth of boost libraries and the compilation size/time of the whole shebang it just seems reasonable that projects package what they need instead of having users download boost release X and build it. I know this isn't how OpenSource is typically done; however, most libraries are singular in domain and don't include solutions ranging from linear algebra to python/C++ interoperability and a full blown C++ preprocessor. I knew I should have sat on my hands (o; michael -- ---------------------------------- Michael Caisse Object Modeling Designs www.objectmodelingdesigns.com
Anthony Williams wrote:
There has recently been some discussion on the monotone development list about phasing out their use of boost libraries that require building external lib files (filesystem and regex). I thought people might find it interesting, so I'm posting some of the comments here:
Anthony: can you pass on to these folks that I'm willing to work with them to refactor regex as a header only library if that's the only showstopper? I have some reservations about such a refactoring (mostly compile times get a lot longer, I suspect), but I guess it's ultimately a "suck it and see" issue. John.
John, if you are going to refactor, probably you can follow the proposals that circulated on the list, of supporting both approaches. Basically, you can give an header that has only declarations, one with (inline) definitions (inline can be controlled with preprocessor macro), and a source file that will include the definitions with disabled inlines. In this way, everyone can choose whether to include the inline definitions, or compile in its project the source file with out-of-line versions of the functions. I think that when one has the option of compiling the library using his preferred build system, this need for header only library will just disappear, and other considerations as compile times will weight more. I like very much the boost build system, but I have to admit that to understand how it worked and to appreciate its power took a lot of time. Maybe relieving just this entry barrier (easing the porting to the user-chosen build system) will be enough for improve boost acceptance. Corrado On 7/14/07, John Maddock <john@johnmaddock.co.uk> wrote:
Anthony Williams wrote:
There has recently been some discussion on the monotone development list about phasing out their use of boost libraries that require building external lib files (filesystem and regex). I thought people might find it interesting, so I'm posting some of the comments here:
Anthony: can you pass on to these folks that I'm willing to work with them to refactor regex as a header only library if that's the only showstopper?
I have some reservations about such a refactoring (mostly compile times get a lot longer, I suspect), but I guess it's ultimately a "suck it and see" issue.
John.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
-- __________________________________________________________________________ dott. Corrado Zoccolo mailto:zoccolo@di.unipi.it PhD - Department of Computer Science - University of Pisa, Italy -------------------------------------------------------------------------- The self-confidence of a warrior is not the self-confidence of the average man. The average man seeks certainty in the eyes of the onlooker and calls that self-confidence. The warrior seeks impeccability in his own eyes and calls that humbleness. Tales of Power - C. Castaneda
Corrado Zoccolo wrote:
John, if you are going to refactor, probably you can follow the proposals that circulated on the list, of supporting both approaches. Basically, you can give an header that has only declarations, one with (inline) definitions (inline can be controlled with preprocessor macro), and a source file that will include the definitions with disabled inlines.
In this way, everyone can choose whether to include the inline definitions, or compile in its project the source file with out-of-line versions of the functions.
I think that when one has the option of compiling the library using his preferred build system, this need for header only library will just disappear, and other considerations as compile times will weight more.
I'm un-dropping off the grid to mention that the Boost mandatory source files should be able to be directly incorporated into a user's build system. It's been possible in the past. Maybe we should suggest to Monotone to try that first.
I like very much the boost build system, but I have to admit that to understand how it worked and to appreciate its power took a lot of time. Maybe relieving just this entry barrier (easing the porting to the user-chosen build system) will be enough for improve boost acceptance.
I've mentioned thoughts like this before, but since our primary purpose is creating C++ libraries, I don't want our secondary projects (like Boost.Build) to create a NIH barrier for our users. I've finally posted an official bug about this (#1093) at <http://svn.boost.org/trac/boost/ticket/1093>.
On 7/14/07, John Maddock <john@johnmaddock.co.uk> wrote:
Anthony Williams wrote:
There has recently been some discussion on the monotone development list about phasing out their use of boost libraries that require building external lib files (filesystem and regex). I thought people might find it interesting, so I'm posting some of the comments here: Anthony: can you pass on to these folks that I'm willing to work with them to refactor regex as a header only library if that's the only showstopper?
I have some reservations about such a refactoring (mostly compile times get a lot longer, I suspect), but I guess it's ultimately a "suck it and see" issue.
-- Daryle Walker Mac, Internet, and Video Game Junkie darylew AT hotmail DOT com
-----Original Message----- From: boost-bounces@lists.boost.org on behalf of Daryle Walker Sent: Sat 7/14/2007 1:27 PM To: boost@lists.boost.org Subject: Re: [boost] Monotone phasing out use of bits of boost due to having to build external libraries I'm un-dropping off the grid to mention that the Boost mandatory source files should be able to be directly incorporated into a user's build system. It's been possible in the past. Maybe we should suggest to Monotone to try that first. -----End Original Message----- I second that. I do the same thing. Its just a matter of looking at the jamfiles for each library and setting defines/flags appropriately.
That's they way I do it now. I do use bjam from running the whole test suite overnight. But for making library tweaks and running particular tests and building the serialization library for experiments etc, I do every thing from Visual Studio. If its too hard to just import the sources into your project then there is a problem. If there is a lot of "hidden" information in bjam and/or bjam files that is needed to do this then that could be a problem. I should not that it WAS a pain getting all the IDE settings just right - but like my recent toothache - its all better now. I would feel better about bjam if it more like the rest of boost as far as being subjected to the normal boost gauntlet. That is I would like to see look like a libray, with self-contained complete documentaion, tutorial, samples, tests, formal review, etc., etc. I would like to see it promoted on its own rather than just a "boost tool". I think this would result in a higher level of usability and stability. Of course this would be a large effort that perhaps is too much to undertake. BTW - The same goes for quickbooks. It looks very nice. But I'm a little bit concerned that its being developped at the same time time its being used to produce boost documentation. I'm concerned that if I make a commitment to it, I'll be left standing on an iceberg. Seems solid but is really constantly shifting requiring constant attention. Just one man's 2 cents. Robert Ramey Sohail Somani wrote:
-----Original Message----- From: boost-bounces@lists.boost.org on behalf of Daryle Walker Sent: Sat 7/14/2007 1:27 PM To: boost@lists.boost.org Subject: Re: [boost] Monotone phasing out use of bits of boost due to having to build external libraries
I'm un-dropping off the grid to mention that the Boost mandatory source files should be able to be directly incorporated into a user's build system. It's been possible in the past. Maybe we should suggest to Monotone to try that first.
-----End Original Message-----
I second that. I do the same thing. Its just a matter of looking at the jamfiles for each library and setting defines/flags appropriately. _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
There is a small, yet active community of individuals working on the Improving Boost Docs project: http://svn.boost.org/trac/boost/wiki/ImprovingBoostDocs I'm forwarding your concern to the boost docs mailing list so that it is brought to the group's collective attention, as I feel that from the user's standpoint it is a valid concern. If you have any input on the state of Boost Docs, feel free to send a message to the boost-docs mailing list with your suggestions :) Jake On 7/14/07, Robert Ramey <ramey@rrsd.com> wrote:
That's they way I do it now.
I do use bjam from running the whole test suite overnight.
But for making library tweaks and running particular tests and building the serialization library for experiments etc, I do every thing from Visual Studio.
If its too hard to just import the sources into your project then there is a problem. If there is a lot of "hidden" information in bjam and/or bjam files that is needed to do this then that could be a problem.
I should not that it WAS a pain getting all the IDE settings just right - but like my recent toothache - its all better now.
I would feel better about bjam if it more like the rest of boost as far as being subjected to the normal boost gauntlet. That is I would like to see look like a libray, with self-contained complete documentaion, tutorial, samples, tests, formal review, etc., etc. I would like to see it promoted on its own rather than just a "boost tool". I think this would result in a higher level of usability and stability. Of course this would be a large effort that perhaps is too much to undertake.
BTW - The same goes for quickbooks. It looks very nice. But I'm a little bit concerned that its being developped at the same time time its being used to produce boost documentation. I'm concerned that if I make a commitment to it, I'll be left standing on an iceberg. Seems solid but is really constantly shifting requiring constant attention.
Just one man's 2 cents.
Robert Ramey
Sohail Somani wrote:
-----Original Message----- From: boost-bounces@lists.boost.org on behalf of Daryle Walker Sent: Sat 7/14/2007 1:27 PM To: boost@lists.boost.org Subject: Re: [boost] Monotone phasing out use of bits of boost due to having to build external libraries
I'm un-dropping off the grid to mention that the Boost mandatory source files should be able to be directly incorporated into a user's build system. It's been possible in the past. Maybe we should suggest to Monotone to try that first.
-----End Original Message-----
I second that. I do the same thing. Its just a matter of looking at the jamfiles for each library and setting defines/flags appropriately. _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey wrote:
BTW - The same goes for quickbooks. It looks very nice. But I'm a little bit concerned that its being developped at the same time time its being used to produce boost documentation. I'm concerned that if I make a commitment to it, I'll be left standing on an iceberg. Seems solid but is really constantly shifting requiring constant attention.
As has been said over and over again, there is nothing to fear. You can never be left standing on an iceberg because you can always escape from it whenever you want to by simply working directly on the generated BoostBook. And, you should only be concerned about the shifting if it affects you; it won't. One strict policy is to maintain 100% backward compatibility from version to version. We may chose to revamp the whole application, but your qbk files will always remain usable. Regards, -- Joel de Guzman http://www.boost-consulting.com http://spirit.sf.net
Sohail Somani wrote:
-----Original Message----- From: boost-bounces@lists.boost.org on behalf of Daryle Walker Sent: Sat 7/14/2007 1:27 PM To: boost@lists.boost.org Subject: Re: [boost] Monotone phasing out use of bits of boost due to having to build external libraries
I'm un-dropping off the grid to mention that the Boost mandatory source files should be able to be directly incorporated into a user's build system. It's been possible in the past. Maybe we should suggest to Monotone to try that first.
-----End Original Message-----
I second that. I do the same thing. Its just a matter of looking at the jamfiles for each library and setting defines/flags appropriately.
AFAIK, it's still possible to just add the source files and build as you like. And AFAIK all libraries mention any special define setup since such building has been common in the past. The Trend I see is that developers using Boost have become reticent to this for some reason. And lastly, an FYI, even though I'm the maintainer of bjam and somewhat of Boost.Build and the developer of the Boost build+install code, I don't build Boost using that other than to test it. I build the Boost libraries I use as part of my own builds. It so happens that I use Boost.Build for my own builds, but I still don't use the Jamfiles in the Boost sources. And this is why no matter what happens to the build system Boost uses, Boost.Build will still continue to be developed and maintained ;-) So, yes, please include and build Boost sources as is most convenient for you. And if there are problem in doing so I'd consider them bugs that library authors should fix. -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim - grafikrobot/yahoo
-----Original Message----- From: boost-bounces@lists.boost.org on behalf of Rene Rivera Subject: Re: [boost] Monotone phasing out use of bits of boost due to having to build external libraries Sohail Somani wrote:
-----Original Message----- From: boost-bounces@lists.boost.org on behalf of Daryle Walker Sent: Sat 7/14/2007 1:27 PM To: boost@lists.boost.org Subject: Re: [boost] Monotone phasing out use of bits of boost due to having to build external libraries
I'm un-dropping off the grid to mention that the Boost mandatory source files should be able to be directly incorporated into a user's build system. It's been possible in the past. Maybe we should suggest to Monotone to try that first.
-----End Original Message-----
I second that. I do the same thing. Its just a matter of looking at the jamfiles for each library and setting defines/flags appropriately.
[snip] So, yes, please include and build Boost sources as is most convenient for you. And if there are problem in doing so I'd consider them bugs that library authors should fix. -----End Original Message----- So given this, and John's assertion that regex is just a bunch of sources, whoever started the thread should mention that monotone should build boost using their own makefiles. They should probably also have the relevant boost sources as part of the official tree to make it easier for users and developers. Of course, if part of the requirements for building monotone were "apt-get install libboost-dev" there probably wouldn't be any issue about building boost? 2c. Sohail
on Sat Jul 14 2007, Rene Rivera <grafikrobot-AT-gmail.com> wrote:
I build the Boost libraries I use as part of my own builds. It so happens that I use Boost.Build for my own builds, but I still don't use the Jamfiles in the Boost sources.
I can't fathom why anyone would do that. At least for some libraries, there is important knowledge encapsulated in those files. Why do the work all over again yourself? -- Dave Abrahams Boost Consulting http://www.boost-consulting.com The Astoria Seminar ==> http://www.astoriaseminar.com
David Abrahams wrote:
on Sat Jul 14 2007, Rene Rivera <grafikrobot-AT-gmail.com> wrote:
I build the Boost libraries I use as part of my own builds. It so happens that I use Boost.Build for my own builds, but I still don't use the Jamfiles in the Boost sources.
I can't fathom why anyone would do that. At least for some libraries, there is important knowledge encapsulated in those files. Why do the work all over again yourself?
Simple reason... Since I started using BBv2 before a Boost release using it appeared I had to create BBv2 files for the 1.33.1 release. Also I happen to need a combination of both release and HEAD libraries, so mixing them is not possible without some special care. -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim - grafikrobot/yahoo
Daryle Walker wrote:
Corrado Zoccolo wrote:
John, if you are going to refactor, probably you can follow the proposals that circulated on the list, of supporting both approaches. Basically, you can give an header that has only declarations, one with (inline) definitions (inline can be controlled with preprocessor macro), and a source file that will include the definitions with disabled inlines.
In this way, everyone can choose whether to include the inline definitions, or compile in its project the source file with out-of-line versions of the functions.
I think that when one has the option of compiling the library using his preferred build system, this need for header only library will just disappear, and other considerations as compile times will weight more.
I'm un-dropping off the grid to mention that the Boost mandatory source files should be able to be directly incorporated into a user's build system. It's been possible in the past. Maybe we should suggest to Monotone to try that first.
That is, and will always be true for regex: as I'm tied of saying over and over, it's "just a bunch of sources", no magic involved :-) John.
"John Maddock" <john@johnmaddock.co.uk> writes:
Anthony Williams wrote:
There has recently been some discussion on the monotone development list about phasing out their use of boost libraries that require building external lib files (filesystem and regex). I thought people might find it interesting, so I'm posting some of the comments here:
Anthony: can you pass on to these folks that I'm willing to work with them to refactor regex as a header only library if that's the only showstopper?
Will do. Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL
participants (12)
-
Anthony Williams -
Corrado Zoccolo -
Daryle Walker -
David Abrahams -
Jake Voytko -
Joel de Guzman -
John Maddock -
Michael Caisse -
Michael Marcin -
Rene Rivera -
Robert Ramey -
Sohail Somani