Boost.Python: Build and Install with Python 2.4 and 2.5?

Hi, I'm part of the Debian Boost packaging team, seeking some guidance on how to build and install Boost.Python so that it is usable with all Python versions shipped in Debian. Debian currently ships Python 2.4 and 2.5. When reading the following, keep in mind that Boost.Python is not a Python extension but, rather, a library that eases writing an extension in C++. So it is necessary that the compiled libraries be visible to be linked with user-written C++ code. One idea is to use a user-config.jam file containing using python : 2.4 : /usr ; using python : 2.5 : /usr ; Then run jam twice bjam variant= ... bjam variant= ... python python=2.5 This produces pairs of library files such as bin.v2/.../link-static/libboost_python-gcc42-1_34_1.a bin.v2/.../link-static/python-2.5/libboost_python-gcc42-1_34_1.a (and similar pairs for shared libraries and all other variants). The key item to note is that the two files have the same name. The python version, unlike the GCC version, is not embedded in the library name. So these files cannot both be installed into /usr/lib. The question, then, is how to distinguish them once installed? Should we: 1. Rename the resulting libraries to decorate them with the python version in addition to the gcc version? This could generate libboost_python-py24-gcc42-1_34_1.a libboost_python-py25-gcc42-1_34_1.a 2. Similar to above, but use bjam's --buildid option? This has the drawback that the build ID shows up differently than the GCC version decoration, e.g. libboost_python-gcc42-1_34_1-py24.a libboost_python-gcc42-1_34_1-py25.a 3. Put the libraries in different subdirectories, e.g. /usr/lib/python2.4/libboost_python-gcc42-1_34_1.a /usr/lib/python2.5/libboost_python-gcc42-1_34_1.a 4. Put the default version directly in /usr/lib and the others in subdirectories, e.g. /usr/lib/libboost_python-gcc42-1_34_1.a /usr/lib/python2.5/libboost_python-gcc42-1_34_1.a The drawback to all these approaches is that client code has to be adjusted to build on Debian. Linking against Boost (for non-bjam projects) is already hard enough with the decorated library names that I fear making the situation worse. The attraction of #4 is that at least the variants for the default python version are found in a natural manner. This has some appeal to me, personally, but I worry that it is too easy to build an extension to python 2.5 and link in the wrong Boost.Python library. In contrast, the other options have the advantage that it forces the user to declare which version of Python is being used. I'd appreciate your thoughts. Have I overlooked a better solution? What are the other packagers of Boost.Python doing? In principle, it would be nice to have the Boost libraries named the same across distributions. Thanks, -Steve

on Sat Feb 23 2008, "Steve M. Robbins" <steve-AT-sumost.ca> wrote:
Hi,
I'm part of the Debian Boost packaging team, seeking some guidance on how to build and install Boost.Python so that it is usable with all Python versions shipped in Debian. Debian currently ships Python 2.4 and 2.5.
When reading the following, keep in mind that Boost.Python is not a Python extension but, rather, a library that eases writing an extension in C++. So it is necessary that the compiled libraries be visible to be linked with user-written C++ code.
One idea is to use a user-config.jam file containing
using python : 2.4 : /usr ; using python : 2.5 : /usr ;
Then run jam twice
bjam variant= ... bjam variant= ... python python=2.5
Well, you can run it once: bjam python=2.4,2.5 IIRC.
This produces pairs of library files such as
bin.v2/.../link-static/libboost_python-gcc42-1_34_1.a bin.v2/.../link-static/python-2.5/libboost_python-gcc42-1_34_1.a
(and similar pairs for shared libraries and all other variants).
The key item to note is that the two files have the same name. The python version, unlike the GCC version, is not embedded in the library name. So these files cannot both be installed into /usr/lib.
I don't know how such name gristing is generated in Boost.Build, but IMO we should change the naming to account for the Python version. This is really a question for the Boost.Build list, I think.
The question, then, is how to distinguish them once installed? Should we:
1. Rename the resulting libraries to decorate them with the python version in addition to the gcc version? This could generate
libboost_python-py24-gcc42-1_34_1.a libboost_python-py25-gcc42-1_34_1.a
I'm not an expert in this area, but it's been my understanding that simply renaming library files doesn't work for some reason.
2. Similar to above, but use bjam's --buildid option? This has the drawback that the build ID shows up differently than the GCC version decoration, e.g.
libboost_python-gcc42-1_34_1-py24.a libboost_python-gcc42-1_34_1-py25.a
Sounds more likely to work, even though it's ugly.
3. Put the libraries in different subdirectories, e.g.
/usr/lib/python2.4/libboost_python-gcc42-1_34_1.a /usr/lib/python2.5/libboost_python-gcc42-1_34_1.a
You could do that, and then provide symlinks with the python version grist in /usr/lib
4. Put the default version directly in /usr/lib and the others in subdirectories, e.g.
/usr/lib/libboost_python-gcc42-1_34_1.a /usr/lib/python2.5/libboost_python-gcc42-1_34_1.a
The drawback to all these approaches is that client code has to be adjusted to build on Debian. Linking against Boost (for non-bjam projects) is already hard enough with the decorated library names that I fear making the situation worse.
Understandable.
The attraction of #4 is that at least the variants for the default python version are found in a natural manner. This has some appeal to me, personally, but I worry that it is too easy to build an extension to python 2.5 and link in the wrong Boost.Python library. In contrast, the other options have the advantage that it forces the user to declare which version of Python is being used.
I'd appreciate your thoughts. Have I overlooked a better solution? What are the other packagers of Boost.Python doing?
Unfortunately, we don't tend to get a lot of interaction with packagers around here, so we don't tend to know what they're doing. I think Boost raises more packaging issues than most other libraries, so it would be good to have more involvement from y'all.
In principle, it would be nice to have the Boost libraries named the same across distributions.
No doubt! -- Dave Abrahams Boost Consulting http://boost-consulting.com

On Wed, Mar 12, 2008 at 09:11:25PM -0400, David Abrahams wrote:
on Sat Feb 23 2008, "Steve M. Robbins" <steve-AT-sumost.ca> wrote:
[...]
This produces pairs of library files such as
bin.v2/.../link-static/libboost_python-gcc42-1_34_1.a bin.v2/.../link-static/python-2.5/libboost_python-gcc42-1_34_1.a
(and similar pairs for shared libraries and all other variants).
The key item to note is that the two files have the same name. The python version, unlike the GCC version, is not embedded in the library name. So these files cannot both be installed into /usr/lib.
I don't know how such name gristing is generated in Boost.Build, but IMO we should change the naming to account for the Python version.
Perhaps. For the moment, I can get the desired effect using --buildid.
The question, then, is how to distinguish them once installed? Should we:
1. Rename the resulting libraries to decorate them with the python version in addition to the gcc version? This could generate
libboost_python-py24-gcc42-1_34_1.a libboost_python-py25-gcc42-1_34_1.a
I'm not an expert in this area, but it's been my understanding that simply renaming library files doesn't work for some reason.
Partly true. I believe you can rename the static libs (.a files) at will. The problem lies in the shared libs. Subsequent to writing the above, I discovered that the two shared libraries (.so.*) have exactly the same SONAME, so renaming doesn't work for them.
2. Similar to above, but use bjam's --buildid option? This has the drawback that the build ID shows up differently than the GCC version decoration, e.g.
libboost_python-gcc42-1_34_1-py24.a libboost_python-gcc42-1_34_1-py25.a
Sounds more likely to work, even though it's ugly.
Yes, this works because the --buildid is incorporated into the SONAMEs, thus distinguishing them. Using --buildid is therefore the *only* solution short of hacking the soname generation myself.
I'd appreciate your thoughts. Have I overlooked a better solution? What are the other packagers of Boost.Python doing?
Unfortunately, we don't tend to get a lot of interaction with packagers around here, so we don't tend to know what they're doing. I think Boost raises more packaging issues than most other libraries, so it would be good to have more involvement from y'all.
Actually, the only thing about Boost that causes grief to packagers is that the toolset name (e.g. "gcc42") is embedded in the library filename. I just wrote a response on Boost.Build outlining this in some detail [1]. Embedding the compiler version requires Debian to rebuild all the boost packages each time a compiler change is made. This is unnecessary when the compiler ABI is maintained (e.g 4.1 -> 4.2) and also unnecessary when the ABI is broken, as Debian has another mechanism to handle that. Note that rebuilding the boost packages has a huge ripple effect, so it is more than simply "unnecessary", it is a very large headache. The fact that Boost embeds the Boost version in the library name is cause for grief to client software authors. But this is unavoidable as long as Boost doesn't take care to maintain ABI across versions. Fortunately, there is a simple solution to this, which I touch on in my post [1]. Regards, -Steve [1] http://lists.boost.org/boost-build/2008/03/18565.php

on Thu Mar 13 2008, "Steve M. Robbins" <steve-AT-sumost.ca> wrote:
Actually, the only thing about Boost that causes grief to packagers is that the toolset name (e.g. "gcc42") is embedded in the library filename. I just wrote a response on Boost.Build outlining this in some detail [1]. Embedding the compiler version requires Debian to rebuild all the boost packages each time a compiler change is made. This is unnecessary when the compiler ABI is maintained (e.g 4.1 -> 4.2) and also unnecessary when the ABI is broken, as Debian has another mechanism to handle that. Note that rebuilding the boost packages has a huge ripple effect, so it is more than simply "unnecessary", it is a very large headache.
I'm not sure that's the end of the story. One reason we embed the compiler name in the library name is that version-specific workarounds often show up in header files. The result is that when compiled against gcc-4.2, client code will effectively see different header contents than the precompiled boost library that was built with gcc-4.1 did. I think we'd need to work out a discipline of avoiding BOOST_WORKAROUND code in headers that distinguishes compiler minor versions. I'm not very confident that I've thought that issue through completely... is it that simple?
The fact that Boost embeds the Boost version in the library name is cause for grief to client software authors. But this is unavoidable as long as Boost doesn't take care to maintain ABI across versions. Fortunately, there is a simple solution to this, which I touch on in my post [1].
This is just a change to bjam? Have you gotten any traction with that idea? -- Dave Abrahams Boost Consulting http://boost-consulting.com

On Thu, May 01, 2008 at 10:30:32AM -0400, David Abrahams wrote:
on Thu Mar 13 2008, "Steve M. Robbins" <steve-AT-sumost.ca> wrote:
Actually, the only thing about Boost that causes grief to packagers is that the toolset name (e.g. "gcc42") is embedded in the library filename. I just wrote a response on Boost.Build outlining this in some detail [1]. Embedding the compiler version requires Debian to rebuild all the boost packages each time a compiler change is made. This is unnecessary when the compiler ABI is maintained (e.g 4.1 -> 4.2) and also unnecessary when the ABI is broken, as Debian has another mechanism to handle that. Note that rebuilding the boost packages has a huge ripple effect, so it is more than simply "unnecessary", it is a very large headache.
I'm not sure that's the end of the story. One reason we embed the compiler name in the library name is that version-specific workarounds often show up in header files. The result is that when compiled against gcc-4.2, client code will effectively see different header contents than the precompiled boost library that was built with gcc-4.1 did.
Interesting. I didn't know that. Debian's "unstable" distribution has been using the same compiled Boost libraries with a mix of GCC 4.1 and 4.2 since December. GCC 4.3 is being added to the mix, too. I should grep the boost headers for BOOST_WORKAROUND to see what differences might arise.
I think we'd need to work out a discipline of avoiding BOOST_WORKAROUND code in headers that distinguishes compiler minor versions. I'm not very confident that I've thought that issue through completely... is it that simple?
If possible, that would be nice from a library distributor's point of view.
The fact that Boost embeds the Boost version in the library name is cause for grief to client software authors. But this is unavoidable as long as Boost doesn't take care to maintain ABI across versions. Fortunately, there is a simple solution to this, which I touch on in my post [1].
This is just a change to bjam? Have you gotten any traction with that idea?
It turns out to be simpler than that. With a small tweak to boost's Jamroot file, I'm now generating libraries without the toolset and without the Boost version decorations. I will use this for the upcoming Boost 1.35.0 Debian packages. We had a long thread in Boost.Build about this idea [1]. I'm not sure I convinced anyone to my point of view. :-) We'll proceed with Debian-specific changes for the short term and see how it goes. Regards, -Steve [1] Thread starts at http://lists.boost.org/boost-build/2008/04/18835.php The shortest statement of Debian's position is http://lists.boost.org/boost-build/2008/04/18839.php (And please ignore http://lists.boost.org/boost-build/2008/04/18918.php as I've changed my mind again)

Steve, Steve M. Robbins wrote:
It turns out to be simpler than that. With a small tweak to boost's Jamroot file, I'm now generating libraries without the toolset and without the Boost version decorations. I will use this for the upcoming Boost 1.35.0 Debian packages.
By all means, could you please get together with other packagers (fedora, notably) and try to establish a common procedure ? (Ideally this procedure would be christened by boost itself, eventually, but unfortunately I just don't see that happening anytime soon.) This is really a big usability issue for developers who want to develop on platforms such as debian or fedora, but still remain as portable as possible, as they have to be prepared for lots of subtle and not-so-subtle variations, complicating their life unnecessarily. It's bad enough that there is no API & ABI stability guarantee from one boost version to the next... Thanks ! Stefan -- ...ich hab' noch einen Koffer in Berlin...
participants (3)
-
David Abrahams
-
Stefan Seefeld
-
Steve M. Robbins