Interface for configuring third-party dependencies
Boosters, it seems that increasing number of upcoming libraries want to depend on third- party dependencies (Locale and GIL.IO being examples) and it would be nice to agree beforehand on an interface that is acceptable to everybody. This email attempts to suggest such interface. = Configuring location of dependencies = In general, for each external dependencies Boost.Build should try to check if it's available without configuration -- that is, in locations normally searched by the compiler/linker. If that fails, user has two way to help the search process. First, it will be possible to specify environment variables (either in OS environment, or using Boost.Build -sVAR=VALUE command-line option). For example, given a name of external dependency of 'zlib', the used variables are: - ZLIB_INCLUDE_PATH - ZLIB_LIBRARY_PATH - ZLIB_LIBRARY_NAME - ZLIB_ROOT The first two specify where the headers ahd libraries are located. The third specifies how the library file itself is named (and has sensible defaults most of the times). ZLIB_ROOT is a shortcut to set both include path and library path to $root/include and $roo/lib respectively. This approach is fairly easy to use for one-off builds of Boost by users. Second, it will possible to specify external dependencies in user-config.jam, or any other configuration file. This approach is slightly more complicated, as one has to edit a config file, but retains configuration and permits to specify multiple configuration for an external dependency -- which is not possible using environment variables. Roughly, the syntax will be: using zlib : condition <target-os>windows : include-path whatever : library-path something-else ; The 'condition' parameter specifies when this declaration will be picked up. Other parameters mirror the environment variables above. = Controlling use of external dependencies = Sometimes, it might be desirable to disable use of external dependency even when it's available. To than end, for each external dependency a new feature should be provided, for example: use-zlib When a value of "off" is specified, zlib won't be used by any boost component. Also, for every specific library that uses an external dependency, a more specific feature will be defined, for example: iostream.use-zlib to control use of that dependency in specific library. Does anybody have comments on this? Thanks, -- Vladimir Prus http://vladimir_prus.blogspot.com Boost.Build: http://boost.org/boost-build2
On Thu, Oct 7, 2010 at 1:20 PM, Vladimir Prus <ghost@cs.msu.su> wrote:
Boosters,
[snip]
Does anybody have comments on this?
How would per-project dependency configuration work? If I want to configure zlib to be something when compiling one project, and something else when compiling another project with boost.build. I wouldn't want to modify user-config.jam in my HOME directory.
Thanks,
-- Vladimir Prus http://vladimir_prus.blogspot.com Boost.Build: http://boost.org/boost-build2
-- Felipe Magno de Almeida
Felipe Magno de Almeida wrote:
On Thu, Oct 7, 2010 at 1:20 PM, Vladimir Prus <ghost@cs.msu.su> wrote:
Boosters,
[snip]
Does anybody have comments on this?
How would per-project dependency configuration work? If I want to configure zlib to be something when compiling one project, and something else when compiling another project with boost.build. I wouldn't want to modify user-config.jam in my HOME directory.
You would put project-config.jam in the root of your project. For extra convenience, you can even create project-config.jam.in with some sensible defaults, put it in version control, and then ask all developers to create project-config.jam based on the .in version. - Volodya
On Thu, Oct 7, 2010 at 2:02 PM, Vladimir Prus <vladimir@codesourcery.com> wrote:
Felipe Magno de Almeida wrote:
On Thu, Oct 7, 2010 at 1:20 PM, Vladimir Prus <ghost@cs.msu.su> wrote:
Boosters,
[snip]
Does anybody have comments on this?
How would per-project dependency configuration work? If I want to configure zlib to be something when compiling one project, and something else when compiling another project with boost.build. I wouldn't want to modify user-config.jam in my HOME directory.
You would put project-config.jam in the root of your project. For extra convenience, you can even create project-config.jam.in with some sensible defaults, put it in version control, and then ask all developers to create project-config.jam based on the .in version.
Thanks, that's cool. Is the current version already this way?
- Volodya
Regards, -- Felipe Magno de Almeida
Felipe Magno de Almeida wrote:
On Thu, Oct 7, 2010 at 2:02 PM, Vladimir Prus <vladimir@codesourcery.com> wrote:
Felipe Magno de Almeida wrote:
On Thu, Oct 7, 2010 at 1:20 PM, Vladimir Prus <ghost@cs.msu.su> wrote:
Boosters,
[snip]
Does anybody have comments on this?
How would per-project dependency configuration work? If I want to configure zlib to be something when compiling one project, and something else when compiling another project with boost.build. I wouldn't want to modify user-config.jam in my HOME directory.
You would put project-config.jam in the root of your project. For extra convenience, you can even create project-config.jam.in with some sensible defaults, put it in version control, and then ask all developers to create project-config.jam based on the .in version.
Thanks, that's cool. Is the current version already this way?
project-config.jam is supported in the current version. The rest of this proposal is not available in the described form -- yet. - Volodya
On 10/7/2010 11:20 AM, Vladimir Prus wrote:
Second, it will possible to specify external dependencies in user-config.jam, or any other configuration file. This approach is slightly more complicated, as one has to edit a config file, but retains configuration and permits to specify multiple configuration for an external dependency -- which is not possible using environment variables. Roughly, the syntax will be:
using zlib : condition<target-os>windows : include-path whatever : library-path something-else ;
The 'condition' parameter specifies when this declaration will be picked up. Other parameters mirror the environment variables above.
Since this is the replacement for the external library support extension I have.. I have one question.. Where would support for configuring the "variation" (i.e. what settings to use when the library/ies are used) fit in the above? Or is the above only for pre-built libraries? -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org (msn) - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail
Rene Rivera wrote:
On 10/7/2010 11:20 AM, Vladimir Prus wrote:
Second, it will possible to specify external dependencies in user-config.jam, or any other configuration file. This approach is slightly more complicated, as one has to edit a config file, but retains configuration and permits to specify multiple configuration for an external dependency -- which is not possible using environment variables. Roughly, the syntax will be:
using zlib : condition<target-os>windows : include-path whatever : library-path something-else ;
The 'condition' parameter specifies when this declaration will be picked up. Other parameters mirror the environment variables above.
Since this is the replacement for the external library support extension I have..
Actually, as I've mentioned on IRC, it's not a replacement. Your extensions allow to build external project -- this is mostly an interface how to specify the location of pre-built things.
I have one question.. Where would support for configuring the "variation" (i.e. what settings to use when the library/ies are used) fit in the above? Or is the above only for pre-built libraries?
I assume you mean usage requirements of library targets? At this level, they don't exist -- this interface is meant to specify where to find the library, and the actual implementation (e.g. zlib.jam) is responsible to define appropriate targets (e.g. /zlib//zlib) so that their usage requirements are right. In particular, in about all cases usage requirements should contain <include>whatever property, as otherwise complication again the library will fail. Does this make sense? - Volodya
On 10/7/2010 2:39 PM, Vladimir Prus wrote:
Rene Rivera wrote:
On 10/7/2010 11:20 AM, Vladimir Prus wrote:
Second, it will possible to specify external dependencies in user-config.jam, or any other configuration file. This approach is slightly more complicated, as one has to edit a config file, but retains configuration and permits to specify multiple configuration for an external dependency -- which is not possible using environment variables. Roughly, the syntax will be:
using zlib : condition<target-os>windows : include-path whatever : library-path something-else ;
The 'condition' parameter specifies when this declaration will be picked up. Other parameters mirror the environment variables above.
Since this is the replacement for the external library support extension I have..
Actually, as I've mentioned on IRC, it's not a replacement. Your extensions allow to build external project -- this is mostly an interface how to specify the location of pre-built things.
The problem I see with that is that those will collide with my extensions for building from source. And IIRC there are some Boost libs with external dependencies that allow sources already (which builds the library). More concretely.. I can't have both this prebuilt declaration, and mine for zlib: using zlib : 1.2.3 : $(ROOT)/sdk/zlib ; I'm assuming you will have some utilities for specifying things like the version number of libraries since it's important some times to build code against specific versions. Right?
I have one question.. Where would support for configuring the "variation" (i.e. what settings to use when the library/ies are used) fit in the above? Or is the above only for pre-built libraries?
I assume you mean usage requirements of library targets?
No.
At this level, they don't exist -- this interface is meant to specify where to find the library, and the actual implementation (e.g. zlib.jam) is responsible to define appropriate targets (e.g. /zlib//zlib) so that their usage requirements are right. In particular, in about all cases usage requirements should contain<include>whatever property, as otherwise complication again the library will fail.
Right, which I already do in my extensions utility. Hopefully you will also define library specific features to be used as conditions.
Does this make sense?
Yes, but it doesn't answer my question entirely ;-) So I would say... Please come up with a syntax that allows for both pre-built and source-to-build libraries to be supported. In case it's useful my extension utilities take the form: using /name/ : /version/ : /location/ + : /options/ * ; I.e. similar to how we configure toolsets. Where /options/ are in the form of arg options (--enable-xyz, --disable-xyz, --with-xyz=val). -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org (msn) - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail
Rene Rivera wrote:
On 10/7/2010 2:39 PM, Vladimir Prus wrote:
Rene Rivera wrote:
On 10/7/2010 11:20 AM, Vladimir Prus wrote:
Second, it will possible to specify external dependencies in user-config.jam, or any other configuration file. This approach is slightly more complicated, as one has to edit a config file, but retains configuration and permits to specify multiple configuration for an external dependency -- which is not possible using environment variables. Roughly, the syntax will be:
using zlib : condition<target-os>windows : include-path whatever : library-path something-else ;
The 'condition' parameter specifies when this declaration will be picked up. Other parameters mirror the environment variables above.
Since this is the replacement for the external library support extension I have..
Actually, as I've mentioned on IRC, it's not a replacement. Your extensions allow to build external project -- this is mostly an interface how to specify the location of pre-built things.
The problem I see with that is that those will collide with my extensions for building from source. And IIRC there are some Boost libs with external dependencies that allow sources already (which builds the library). More concretely.. I can't have both this prebuilt declaration, and mine for zlib:
using zlib : 1.2.3 : $(ROOT)/sdk/zlib ;
Well, you can't have exact same declaration, but it's possible to have this: using zlib : source $(ROOT)/sdk/zlib ; Actually, it's probably reasonable to ask whether we want to support building from source for all external libraries, or for select ones, and if the latter, how do we decide? Supporting build from source for all libraries might easily become troublesome -- especially, if those libraries are being actively developed.
I'm assuming you will have some utilities for specifying things like the version number of libraries since it's important some times to build code against specific versions. Right?
That's a good question. I did not had it in my original plan. The primary problem I see is that a project might specify a certain version, while user-config.jam might not specify any, and: - requiring that user always specify version, especially for system libraries, is annoying - detecting the version automatically might be tricky. What do you suggest?
I have one question.. Where would support for configuring the "variation" (i.e. what settings to use when the library/ies are used) fit in the above? Or is the above only for pre-built libraries?
I assume you mean usage requirements of library targets?
No.
At this level, they don't exist -- this interface is meant to specify where to find the library, and the actual implementation (e.g. zlib.jam) is responsible to define appropriate targets (e.g. /zlib//zlib) so that their usage requirements are right. In particular, in about all cases usage requirements should contain<include>whatever property, as otherwise complication again the library will fail.
Right, which I already do in my extensions utility. Hopefully you will also define library specific features to be used as conditions.
Can you clarify this "used as conditions" part?
Does this make sense?
Yes, but it doesn't answer my question entirely ;-) So I would say... Please come up with a syntax that allows for both pre-built and source-to-build libraries to be supported. In case it's useful my extension utilities take the form:
using /name/ : /version/ : /location/ + : /options/ * ;
I.e. similar to how we configure toolsets. Where /options/ are in the form of arg options (--enable-xyz, --disable-xyz, --with-xyz=val).
Well, my proposed syntax differs from above in that 'version' and 'location' are not positional parameters, but rather specified by name -- must like your 'options'. - Volodya
On 20/10/10 09:13, Vladimir Prus wrote:
Rene Rivera wrote:
The problem I see with that is that those will collide with my extensions for building from source. And IIRC there are some Boost libs with external dependencies that allow sources already (which builds the library). More concretely.. I can't have both this prebuilt declaration, and mine for zlib:
using zlib : 1.2.3 : $(ROOT)/sdk/zlib ;
Well, you can't have exact same declaration, but it's possible to have this:
using zlib : source $(ROOT)/sdk/zlib ;
Would it be possible to be able to determine version and compare against min version required?
Actually, it's probably reasonable to ask whether we want to support building from source for all external libraries, or for select ones, and if the latter, how do we decide? Supporting build from source for all libraries might easily become troublesome -- especially, if those libraries are being actively developed.
IMHO, building from source is not required. There is usually no problem with building binaries if source available, So, once installed, they could be specified to use by Boost.Build.
I'm assuming you will have some utilities for specifying things like the version number of libraries since it's important some times to build code against specific versions. Right?
Ah, so I have repeated this question above.
That's a good question. I did not had it in my original plan. The primary problem I see is that a project might specify a certain version, while user-config.jam might not specify any, and: - requiring that user always specify version, especially for system libraries, is annoying - detecting the version automatically might be tricky.
What do you suggest?
Mmy experiences suggest there are usually two ways to determine version: 1. Scan public header for preprocessor definition specifying version 2. Call utility config program, like pg_config --version If it would be possible to have these two schemes provided, I think it should be well enough. For users who find it not sufficient, they always can manually assure minimum versions required are installed in their systems. My 5 groschen. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net Charter Member of OSGeo, http://osgeo.org Member of ACCU, http://accu.org
Mateusz Loskot wrote:
On 20/10/10 09:13, Vladimir Prus wrote:
Rene Rivera wrote:
The problem I see with that is that those will collide with my extensions for building from source. And IIRC there are some Boost libs with external dependencies that allow sources already (which builds the library). More concretely.. I can't have both this prebuilt declaration, and mine for zlib:
using zlib : 1.2.3 : $(ROOT)/sdk/zlib ;
Well, you can't have exact same declaration, but it's possible to have this:
using zlib : source $(ROOT)/sdk/zlib ;
Would it be possible to be able to determine version and compare against min version required?
Actually, it's probably reasonable to ask whether we want to support building from source for all external libraries, or for select ones, and if the latter, how do we decide? Supporting build from source for all libraries might easily become troublesome -- especially, if those libraries are being actively developed.
IMHO, building from source is not required. There is usually no problem with building binaries if source available, So, once installed, they could be specified to use by Boost.Build.
I'm assuming you will have some utilities for specifying things like the version number of libraries since it's important some times to build code against specific versions. Right?
Ah, so I have repeated this question above.
That's a good question. I did not had it in my original plan. The primary problem I see is that a project might specify a certain version, while user-config.jam might not specify any, and: - requiring that user always specify version, especially for system libraries, is annoying - detecting the version automatically might be tricky.
What do you suggest?
Mmy experiences suggest there are usually two ways to determine version: 1. Scan public header for preprocessor definition specifying version 2. Call utility config program, like pg_config --version
If it would be possible to have these two schemes provided, I think it should be well enough.
I suppose this can work; however the first method requires that you actually find and open the header file. Which means that either: - Build system has to replicate path search logic of all compilers, - It's possible to ask the compiler where the header was found - Build system ignores any path search compiler might do, locates the headers/libraries on its own, and the forces the compiler to use them. Neither option is particularly attractive :-( - Volodya
On 23/10/10 13:22, Vladimir Prus wrote:
Mateusz Loskot wrote:
On 20/10/10 09:13, Vladimir Prus wrote:
Rene Rivera wrote:
The problem I see with that is that those will collide with my extensions for building from source. And IIRC there are some Boost libs with external dependencies that allow sources already (which builds the library). More concretely.. I can't have both this prebuilt declaration, and mine for zlib:
using zlib : 1.2.3 : $(ROOT)/sdk/zlib ;
Well, you can't have exact same declaration, but it's possible to have this:
using zlib : source $(ROOT)/sdk/zlib ;
Would it be possible to be able to determine version and compare against min version required?
Actually, it's probably reasonable to ask whether we want to support building from source for all external libraries, or for select ones, and if the latter, how do we decide? Supporting build from source for all libraries might easily become troublesome -- especially, if those libraries are being actively developed.
IMHO, building from source is not required. There is usually no problem with building binaries if source available, So, once installed, they could be specified to use by Boost.Build.
I'm assuming you will have some utilities for specifying things like the version number of libraries since it's important some times to build code against specific versions. Right?
Ah, so I have repeated this question above.
That's a good question. I did not had it in my original plan. The primary problem I see is that a project might specify a certain version, while user-config.jam might not specify any, and: - requiring that user always specify version, especially for system libraries, is annoying - detecting the version automatically might be tricky.
What do you suggest?
Mmy experiences suggest there are usually two ways to determine version: 1. Scan public header for preprocessor definition specifying version 2. Call utility config program, like pg_config --version
If it would be possible to have these two schemes provided, I think it should be well enough.
I suppose this can work; however the first method requires that you actually find and open the header file. Which means that either:
- Build system has to replicate path search logic of all compilers, - It's possible to ask the compiler where the header was found - Build system ignores any path search compiler might do, locates the headers/libraries on its own, and the forces the compiler to use them.
Neither option is particularly attractive :-(
Volodya, I think you're right. I'm out of idea what could be best solution. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net Charter Member of OSGeo, http://osgeo.org Member of ACCU, http://accu.org
- Build system has to replicate path search logic of all compilers, - It's possible to ask the compiler where the header was found - Build system ignores any path search compiler might do, locates the headers/libraries on its own, and the forces the compiler to use them. You can also compile a test program that uses whatever version macros
On Sat, 23 Oct 2010 16:22:37 +0400 Vladimir Prus <vladimir@codesourcery.com> wrote: the library offers(e.g. boost's BOOST_VERSION) to test the version and fail to compile if version is too old. That'll avoid the need for the build system to find the header.
From: "Rene Rivera" <grafikrobot@gmail.com> Sent: Thursday, October 07, 2010 8:42 PM
Where would support for configuring the "variation" (i.e. what settings to use when the library/ies are used) fit in the above? Or is the above only for pre-built libraries?
Besides specifying the obvious things like location, version and build options of a 3rd party library there is also the issue of choosing a linking (and sometimes even initialisation) 'policy' (see the 'pps' at the very end of this post http://permalink.gmane.org/gmane.comp.lib.boost.devel/209408 for an example)... I wonder how much, if any, of this process could be reasonably automated through a build tool (bjam or otherwise)...The most difficult scenario would of course be delay loaded dynamic linking (where code for performing the equivalent of GetProcAdress() has to be generated for every used 3rd-party library function)...? -- "What Huxley teaches is that in the age of advanced technology, spiritual devastation is more likely to come from an enemy with a smiling face than from one whose countenance exudes suspicion and hate." Neil Postman
On Oct 7, 2010, at 12:20 PM, Vladimir Prus wrote:
= Controlling use of external dependencies =
Sometimes, it might be desirable to disable use of external dependency even when it's available. To than end, for each external dependency a new feature should be provided, for example:
use-zlib
When a value of "off" is specified, zlib won't be used by any boost component. Also, for every specific library that uses an external dependency, a more specific feature will be defined, for example:
iostream.use-zlib
to control use of that dependency in specific library.
Is there (or could there be) some not too difficult way to find out what the set of such external dependencies is? Otherwise it is all too easy to unintentionally build against such a dependency that happens to be present on the build machine but might not be present on the delivery machine.
Kim Barrett wrote:
On Oct 7, 2010, at 12:20 PM, Vladimir Prus wrote:
= Controlling use of external dependencies =
Sometimes, it might be desirable to disable use of external dependency even when it's available. To than end, for each external dependency a new feature should be provided, for example:
use-zlib
When a value of "off" is specified, zlib won't be used by any boost component. Also, for every specific library that uses an external dependency, a more specific feature will be defined, for example:
iostream.use-zlib
to control use of that dependency in specific library.
Is there (or could there be) some not too difficult way to find out what the set of such external dependencies is? Otherwise it is all too easy to unintentionally build against such a dependency that happens to be present on the build machine but might not be present on the delivery machine.
All configuration checks are recorded, e.g: Performing configuration checks - iconv (libc) : yes - icu : yes Of course, one needs some judgement to see which of those tests are actually about external libraries -- for example the first one checks the standard library. - Volodya
On Oct 7, 2010, at 4:41 PM, Vladimir Prus wrote:
All configuration checks are recorded, e.g:
Performing configuration checks
- iconv (libc) : yes - icu : yes
Of course, one needs some judgement to see which of those tests are actually about external libraries -- for example the first one checks the standard library.
That might be sufficient. Hm. Is there some reporting of where each was found?
On 07/10/10 17:20, Vladimir Prus wrote:
Boosters,
it seems that increasing number of upcoming libraries want to depend on third- party dependencies (Locale and GIL.IO being examples) and it would be nice to agree beforehand on an interface that is acceptable to everybody. This email attempts to suggest such interface.
Volodya, First and foremost, thank you very much for working on this. Comments? I have no critique to the interface proposal :-) Also, I really like the project-config.jam solution you've just discussed with Felipe. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net Charter Member of OSGeo, http://osgeo.org Member of ACCU, http://accu.org
participants (8)
-
Domagoj Saric -
Felipe Magno de Almeida -
Kim Barrett -
Mateusz Loskot -
Rene Rivera -
Sergey Popov -
Vladimir Prus -
Vladimir Prus