ABI and API compatibility
Right now, Boost provides not guarantees whatsoever about API and ABI compatibility between any two versions. This might be OK for major releases, but the question is if we really want 1.35.1 to be incompatible to 1.35.0 in any way? If 1.35.1 is a bugfix release, then I would hope that: 1. Applications using 1.35.0 need not be changed at all. 2. If application link to 1.35.0 as dynamic library, then switching them to use 1.35.1 should not break anything. Does this make sense, and if so, can we officially document such a policy? - Volodya
On Thu, Apr 24, 2008 at 09:14:25AM +0400, Vladimir Prus wrote:
Right now, Boost provides not guarantees whatsoever about API and ABI compatibility between any two versions. This might be OK for major releases, but the question is if we really want 1.35.1 to be incompatible to 1.35.0 in any way? If 1.35.1 is a bugfix release, then I would hope that:
1. Applications using 1.35.0 need not be changed at all.
This means the API, the Application Program Interface, does not change (no changes in source code required).
2. If application link to 1.35.0 as dynamic library, then switching them to use 1.35.1 should not break anything.
This means the ABI, the Application Binary Interface, does not change.
Does this make sense, and if so, can we officially document such a policy?
Of course, this would be great! But please note that probably also changes to the build system are required to adjust the soname. Thus there even exist one or do only some distrubutions added the version number as (bogus) soname? I'm really no expert in this domain but just documenting it is *not* sufficient! Jens
Jens Seidel wrote:
On Thu, Apr 24, 2008 at 09:14:25AM +0400, Vladimir Prus wrote:
1. Applications using 1.35.0 need not be changed at all.
This means the API, the Application Program Interface, does not change (no changes in source code required).
This ought to be doable without too much trouble for minor releases.
2. If application link to 1.35.0 as dynamic library, then switching them to use 1.35.1 should not break anything.
This means the ABI, the Application Binary Interface, does not change.
Since most of Boost's libraries are header-only, that's meaningless for them. Mostly, anyway. ABI compatibility is to be maintained by the compiled libraries foremost, they would have to worry about this the most. It can be done with a bit of understanding of the issues and some discipline. The question is, is it worth it? Any application that uses a precompiled Boost component probably uses other Boost components, too. As such, the likelihood is high that upgrading Boost means recompiling the application. Thus, keeping the ABI is kind of pointless. Sebastian
Sebastian Redl wrote:
Jens Seidel wrote:
On Thu, Apr 24, 2008 at 09:14:25AM +0400, Vladimir Prus wrote:
1. Applications using 1.35.0 need not be changed at all.
This means the API, the Application Program Interface, does not change (no changes in source code required).
This ought to be doable without too much trouble for minor releases.
2. If application link to 1.35.0 as dynamic library, then switching them to use 1.35.1 should not break anything.
This means the ABI, the Application Binary Interface, does not change.
Since most of Boost's libraries are header-only, that's meaningless for them. Mostly, anyway. ABI compatibility is to be maintained by the compiled libraries foremost, they would have to worry about this the most. It can be done with a bit of understanding of the issues and some discipline. The question is, is it worth it? Any application that uses a precompiled Boost component probably uses other Boost components, too. As such, the likelihood is high that upgrading Boost means recompiling the application.
Why? If there's no interesting change in that other Boost component in a 1.35.1, I don't really need to recompile. - Volodya
Vladimir Prus wrote:
Why? If there's no interesting change in that other Boost component in a 1.35.1, I don't really need to recompile.
Yes, if. So basically, another thing that's needed is a more precise changelog. Because the evil part is, if there is a change but you miss it, and the dynamic library uses the component too, you might get behaviour discrepancies but no tool errors. Sebastian
On Thu, Apr 24, 2008 at 12:37:09PM +0200, Sebastian Redl wrote:
Jens Seidel wrote:
On Thu, Apr 24, 2008 at 09:14:25AM +0400, Vladimir Prus wrote:
2. If application link to 1.35.0 as dynamic library, then switching them to use 1.35.1 should not break anything.
Since most of Boost's libraries are header-only, that's meaningless for them. Mostly, anyway.
Right!
ABI compatibility is to be maintained by the compiled libraries foremost, they would have to worry about this the most. It can be done with a bit of understanding of the issues and some discipline.
Indeed. I think even changing a public function (ahm, method) from int f() to int f(int c=0) would create a ABI but no API conflict. One has to be careful.
The question is, is it worth it?
I'm very sure it is!
Any application that uses a precompiled Boost component probably uses other Boost components, too. As such, the
OK.
likelihood is high that upgrading Boost means recompiling the application.
Please explain this! Probably you mean that beside Boost libraries also parts of Boost are used which are provided in headers only? Now you assume that one has to recompile to gain from the header only parts? This is right but you forgot (if my assumptions are true) that there is a much more likely case: Every Linux or *BSD distribution (and probably also Cygwin and Fink) contains Boost and applications linked against it. Now assume the Boost maintainer in this distribution wants to update Boost. With a stable ABI the maintainer just uploads a new version of Boost and that's all. Without a stable ABI the maintainer either has to keep the old Boost library (or better librar*ies* in this case) and provide just another version or the upload of a new Boost version requires a coordinated upload with all recompiled programs and libraries which depend on Boost. This is a very big mess!!!! Debian currently discusses to install multiple versions of Boost. Currently libboost-dev installs it's header files in the standard search path /usr/include/boost/*.h. This has to change to support multiple versions and every program which uses Boost requires updates if no link to a default version is created. Such transitions are very time consuming (there could be hundreds of programs affected). For Debian: $ grep-dctrl -FBuild-Depends,Build-Depends-Indep -s Package libboost-dev -n /var/lib/apt/lists/*_Sources ... d4x dc-qt debian-xcontrol deluge-torrent enblend encfs evolvotron exmap fslview geordi glob2 gnash gnuradio gpsdrive inkscape ... That are 50 packages (strange, I expected many more).
Thus, keeping the ABI is kind of pointless.
No, it isn't. I remember also postings (on this list or on a Debian one) that some programs do not intent to use Boost because of no stable API/ABI. Jens
Jens Seidel wrote:
On Thu, Apr 24, 2008 at 12:37:09PM +0200, Sebastian Redl wrote:
Any application that uses a precompiled Boost component probably uses other Boost components, too. As such, the
OK.
likelihood is high that upgrading Boost means recompiling the application.
Please explain this! Probably you mean that beside Boost libraries also parts of Boost are used which are provided in headers only? Now you assume that one has to recompile to gain from the header only parts? This is right but you forgot (if my assumptions are true) that there is a much more likely case:
Every Linux or *BSD distribution (and probably also Cygwin and Fink) contains Boost and applications linked against it. Now assume the Boost maintainer in this distribution wants to update Boost. With a stable ABI the maintainer just uploads a new version of Boost and that's all. Without a stable ABI the maintainer either has to keep the old Boost library (or better librar*ies* in this case) and provide just another version or the upload of a new Boost version requires a coordinated upload with all recompiled programs and libraries which depend on Boost. This is a very big mess!!!!
What I'm saying is that I believe it to be very dangerous to have your own application use Boost header-only libraries of one version and precompiled libraries that link against other versions of the header-only libraries. If components from the header-only version (say, a shared_ptr or an any) are somehow transferred between the main executable and the library, you're in great danger of crashing. Great care would have to be taken that even the header-only libraries don't make changes that break such compatibility. It depends on the kind of changes that go into point releases. Sebastian
Sebastian Redl wrote:
Jens Seidel wrote:
On Thu, Apr 24, 2008 at 12:37:09PM +0200, Sebastian Redl wrote:
Any application that uses a precompiled Boost component probably uses other Boost components, too. As such, the
OK.
likelihood is high that upgrading Boost means recompiling the application.
Please explain this! Probably you mean that beside Boost libraries also parts of Boost are used which are provided in headers only? Now you assume that one has to recompile to gain from the header only parts? This is right but you forgot (if my assumptions are true) that there is a much more likely case:
Every Linux or *BSD distribution (and probably also Cygwin and Fink) contains Boost and applications linked against it. Now assume the Boost maintainer in this distribution wants to update Boost. With a stable ABI the maintainer just uploads a new version of Boost and that's all. Without a stable ABI the maintainer either has to keep the old Boost library (or better librar*ies* in this case) and provide just another version or the upload of a new Boost version requires a coordinated upload with all recompiled programs and libraries which depend on Boost. This is a very big mess!!!!
What I'm saying is that I believe it to be very dangerous to have your own application use Boost header-only libraries of one version and precompiled libraries that link against other versions of the header-only libraries. If components from the header-only version (say, a shared_ptr or an any) are somehow transferred between the main executable and the library, you're in great danger of crashing.
This is not anything specific to header-only libraries. Classes that are in the public API, regardless of whether the actual implementation is defined inline, or compiled in a separate object file, should not change their layout in point releases, in my opinion. - Volodya
Vladimir Prus wrote:
Sebastian Redl wrote:
What I'm saying is that I believe it to be very dangerous to have your own application use Boost header-only libraries of one version and precompiled libraries that link against other versions of the header-only libraries. If components from the header-only version (say, a shared_ptr or an any) are somehow transferred between the main executable and the library, you're in great danger of crashing.
This is not anything specific to header-only libraries. Classes that are in the public API, regardless of whether the actual implementation is defined inline, or compiled in a separate object file, should not change their layout in point releases, in my opinion.
The issues are not only limited with binary layout. ODR violation problems show with functions too, since due to relocations your application may end up using either old or new versions of them. It is well possible to get a mixed set of them in the end. No need to say that the functions may not behave well in such a zoo and you'll get a big can of worms trying to debug the app. Whenever possible I, personally, try to avoid such situation by all means.
Vladimir Prus skrev:
Classes that are in the public API, regardless of whether the actual implementation is defined inline, or compiled in a separate object file, should not change their layout in point releases, in my opinion.
So the layout of those classes in 1 point 35 should be the same as in 1 point 31 in your opinion, we have to wait for boost 2.0 ;-) Sorry, could not resist. If releases become frequent, say every 3 months, maybe full two-dotted releases make less sence than before. Critical fixes for specific platforms may be better handled by a published set of hot fix patches. Les critical issuses can wait until next one-dotted release. -- Bjørn
2008/4/24 Vladimir Prus <ghost@cs.msu.su>:
1. Applications using 1.35.0 need not be changed at all. 2. If application link to 1.35.0 as dynamic library, then switching them to use 1.35.1 should not break anything.
Does this make sense, and if so, can we officially document such a policy?
I think this would be a very good thing, but I'm not sure how practical it would be. There are already at least two changes in the release branch which break ABI compatability (I might be wrong, I only had a quick look at the changesets): http://svn.boost.org/trac/boost/changeset/44414 http://svn.boost.org/trac/boost/changeset/44440 We do so much in headers that this would require a lot of discipline and prevent a lot of fixes. It'd be nice to be able to test for ABI compatibility, perhaps by running the tests with one version, and the libraries with another. Would that be effective? We'd also need more frequent releases so that those fixes could be pushed out sooner but we need that anyway. Daniel
On Thu, Apr 24, 2008 at 09:00:43AM +0100, Daniel James wrote:
I think this would be a very good thing, but I'm not sure how practical it would be. There are already at least two changes in the release branch which break ABI compatability (I might be wrong, I only had a quick look at the changesets):
Isn't it possible to revise every change and to partly revert/find alternatives?
Is this really a problem? It affects only a private header file in detail/ no user should use. The changelog isn't very verbose but "Disable sync use for arm and hppa." could be justified if the previous code was just buggy or unsupported (no need to keep the some compilation bug for a new version as in the old one).
We do so much in headers that this would require a lot of discipline and prevent a lot of fixes. It'd be nice to be able to test for ABI compatibility, perhaps by running the tests with one version, and the libraries with another. Would that be effective? We'd also need more frequent releases so that those fixes could be pushed out sooner but we need that anyway.
Debian (www.debian.org) contains the packages icheck: C interface ABI/API checker abicheck: binary compatibility checking tool Comparing the symbols in the libraries is probably more effective than applying a few tests. Jens
2008/4/24 Jens Seidel <jensseidel@users.sf.net>:
Is this really a problem? It affects only a private header file in detail/ no user should use. The changelog isn't very verbose but "Disable sync use for arm and hppa." could be justified if the previous code was just buggy or unsupported (no need to keep the some compilation bug for a new version as in the old one).
Well, I don't know the story behind that change, but if the 1.35 version compiles then it means that different parts of the program will use different implementations of the counter object, which I assume are incompatible. Also, since the bug is in the support for multi-threading on a particular platform it could be the case that a program is using the buggy version and not aware of it.
We do so much in headers that this would require a lot of discipline and prevent a lot of fixes. It'd be nice to be able to test for ABI compatibility, perhaps by running the tests with one version, and the libraries with another. Would that be effective? We'd also need more frequent releases so that those fixes could be pushed out sooner but we need that anyway.
Debian (www.debian.org) contains the packages icheck: C interface ABI/API checker abicheck: binary compatibility checking tool
Comparing the symbols in the libraries is probably more effective than applying a few tests.
Yes, you're probably right. Although I think it'd still be worth running some sort of tests in addition. I might have made a terminology mistake and was writing about more than just ABI compatibility. A simple example: if I made a change to Boost.Hash that caused it to return different hash values, then different parts of the program would be generating different hash values which would cause problems if they accessed the same hash table. But I don't think it would break ABI compatibility. And we don't have any tests that would detect something like that. That one is pretty obvious, but I worry that there might be more subtle problems. Daniel
Daniel James:
2008/4/24 Jens Seidel <jensseidel@users.sf.net>:
Is this really a problem? It affects only a private header file in detail/ no user should use. The changelog isn't very verbose but "Disable sync use for arm and hppa." could be justified if the previous code was just buggy or unsupported (no need to keep the some compilation bug for a new version as in the old one).
Well, I don't know the story behind that change, but if the 1.35 version compiles...
It doesn't (on ARM and HP PA-RISC). (Actually it compiles but doesn't link.)
2008/4/24 Peter Dimov <pdimov@pdimov.com>:
Daniel James:
2008/4/24 Jens Seidel <jensseidel@users.sf.net>:
Is this really a problem? It affects only a private header file in detail/ no user should use. The changelog isn't very verbose but "Disable sync use for arm and hppa." could be justified if the previous code was just buggy or unsupported (no need to keep the some compilation bug for a new version as in the old one).
Well, I don't know the story behind that change, but if the 1.35 version compiles...
It doesn't (on ARM and HP PA-RISC). (Actually it compiles but doesn't link.)
Does that mean that if GCC adds support for the functions used the sync counted base it'll create an ABI incompatibility? Which isn't a real problem, but such fun.
On Thursday 24 April 2008 12:38:51 Daniel James wrote:
2008/4/24 Jens Seidel <jensseidel@users.sf.net>:
Is this really a problem? It affects only a private header file in detail/ no user should use. The changelog isn't very verbose but "Disable sync use for arm and hppa." could be justified if the previous code was just buggy or unsupported (no need to keep the some compilation bug for a new version as in the old one).
Well, I don't know the story behind that change, but if the 1.35 version compiles then it means that different parts of the program will use different implementations of the counter object, which I assume are incompatible.
Actually directly using some class that changes implementation details (private members are added, modified, deleted) without recompilation of all involved parts leads to violation of ODR and thus UB. I think KDE people have gone through this alot (http://techbase.kde.org/index.php?title=Policies/Binary_Compatibility_Issues...) and they even made a document explaining techniques and policies how to achieve good ABI compatibility and as you can see in there they advise to use the "d-pointer" which seems to me to be another name for the PIMPL idiom. Clearly boost will never accept to add an additional layer of indirection for all the separately compilable API just to achieve good ABI compatibility.
I might have made a terminology mistake and was writing about more than just ABI compatibility. A simple example: if I made a change to Boost.Hash that caused it to return different hash values, then different parts of the program would be generating different hash values which would cause problems if they accessed the same hash table. But I don't think it would break ABI compatibility. And we don't have any tests that would detect something like that. That one is pretty obvious, but I worry that there might be more subtle problems.
Good point, while strict ABI compatibility means you can replace your library with compiled code and it should work it does not also mean it will properly do so. -- Mihai RUSU Email: dizzy@roedu.net "Linux is obsolete" -- AST
On Thursday 24 April 2008 08:06:12 am dizzy wrote:
Actually directly using some class that changes implementation details (private members are added, modified, deleted) without recompilation of all involved parts leads to violation of ODR and thus UB.
If the size of the object changes, all ABI bets are off. If you use boost in a library (not an application) that you supply as headers + precompiled binary library, you must avoid using components from boost libraries (even if they are header-only) as: 1. Members in your classes 2. Direct arguments to member functions If the size of a shared_ptr changed, the size of any class that contains a shared_ptr member also changes. Because of this issue, many libraries (such as those built on SystemC) include their own private (and usually incomplete) copy of boost. When using two such libraries in a project, all hell breaks loose when components in neither library but which could depend on common components in both libraries are used. Worse, when the libraries are header only, it is very easy to overlook this problem, especially since the compiler/linker does not provide any warnings. I have been lucky enough that all issues have been fixable in the past couple of years, but eventually I will get a library vendor who uses boost, but will not provide source even under an NDA (or even worse, out of business), and then it will be a huge problem. It is already pretty hard to maintain internal forks of such libraries (even if the differences are only minor). Regards, Ravi
participants (10)
-
Andrey Semashev -
Bjørn Roald -
Daniel James -
dizzy -
Jens Seidel -
Peter Dimov -
Ravi -
Sebastian Redl -
Vladimir Prus -
Vladimir Prus