[iostreams regressions] patches to fix msvc issues
Since it seems like Jonathan Turkanis is not currently available to work on maintaining the iostreams library, perhaps someone else with CVS write access might apply the attached jamfile patches related to that library to RC_1_34_0 (if the release manager approves) and to HEAD. The description of the patches is below. These, along with the mentioned modification to the bzip2 DEF file, should resolve the remaining iostreams regressions. Once the jamfile patches are applied, I'll ping RudbekAssociates to delete their current test results, since I believe they run the regression tests incrementally and I don't think jamfile changes trigger a rebuild. Thanks, -Dave David Deakins wrote:
Douglas Gregor wrote:
|iostreams| bzip2_test: msvc-7.1 msvc-8.0 gzip_test: msvc-7.1 msvc-8.0 zlib_test: msvc-7.1 msvc-8.0
I spent a little bit of time looking into the msvc-7.1 and msvc-8.0 regressions for this library. It looks like the zlib and bzip2 libraries used in these tests are being built from source files and it appears that there are some omissions in the iostreams jamfile for properly building them as DLLs in Windows.
For zlib, there just needs to be a <link>shared:<define>ZLIB_DLL in its build requirements so that the zlib source files use the __declspec keyword on functions that should be exported. Without this, no functions are exported and the import library file is not created (thus the link errors).
With bzip2, the basic problem is the same but its sources use a .DEF file rather than __declspec (at least in the 1.0.4 sources that I looked at). Adding <link>shared:<def-file>$(BZIP2_SOURCE)/libbz2.def properly included the DEF file that gives the exports, but revealed another problem. The DEF file has a 'LIBRARY LIBBZ2' line in it which wants to give the DLL the name libbz2.DLL. Unfortunately the iostreams jam file typically calls the library something like boost_bzip2-vc80-mt-gd-1_34.DLL and specifies an /OUT directive to the linker to ask for this name. At least on my machine with VC8, the /OUT directive overrides the LIBRARY rule enough to make the intended boost_bzip2-vc80-mt-gd-1_34.DLL file, but not enough to get this name into the import .LIB file that boost_iostreams links to. As a result, the iostreams library will compile and link, but will fail trying locate libbz2.DLL (instead of the correct boost_bzip2-vc80-mt-gd-1_34.DLL). The only way I could easily resolve this was to comment out the LIBRARY rule in the libbz2.DEF file in the bzip2 sources like so:
; LIBRARY LIBBZ2
With this change and the patches to the iostreams jamfile, building zlib and bzip2 from sources works correctly and the 3 tests pass. I attached a patch for jamfile.v2 in libs\iostream\build with my changes. It might be good to put a note about the bzip2 .DEF file change in the iostreams documentation.
Incidently, using <def-file> revealed a small issue in msvc.jam related to DEF file paths that have spaces in them. The linker command does not put quotes around the DEF path like it does with most of the other parameters, and as a result DEF paths with spaces will cause the link to fail. I also included a patch to msvc.jam to add quotes around DEF file paths.
Thanks, -Dave
David Deakins wrote:
Since it seems like Jonathan Turkanis is not currently available to work on maintaining the iostreams library, perhaps someone else with CVS write access might apply the attached jamfile patches related to that library to RC_1_34_0 (if the release manager approves) and to HEAD. The description of the patches is below. These, along with the mentioned modification to the bzip2 DEF file, should resolve the remaining iostreams regressions.
Thanks for the patch! The patch looks good to me, but I have one question -- those tests used to pass before, at least for some msvc testers. Do you know what made them fail now? Was that a change in Jamfile (I don't remember making any), or bzlib/zlib has changed. "No idea" is fine for an answer, but if you know for sure, it would be even better ;-) Thomas, I think this patch is something we should apply, as otherwise functionality of iostream library will some somewhat limited in msvc/(source libz) configuration. What do you think? - Volodya
Vladimir Prus wrote:
Thanks for the patch! The patch looks good to me, but I have one question -- those tests used to pass before, at least for some msvc testers. Do you know what made them fail now? Was that a change in Jamfile (I don't remember making any), or bzlib/zlib has changed.
"No idea" is fine for an answer, but if you know for sure, it would be even better ;-)
It is only when zlib or bzip2 are built from sources that this jamfile issue shows up. If I remember correctly, I think that in the past the Rudbek Associates test system was using already-built LIB or DLL files for zlib and bzip2. I believe that when Victor resurrected the test system a month or so ago, it got configured to built zlib and bzip2 from sources instead. Then these new regressions started showing up. That's my theory at least... -Dave
I am OK with this, but I won't have time to actually patch it myself till tomorrow evening. Thomas David Deakins wrote:
Since it seems like Jonathan Turkanis is not currently available to work on maintaining the iostreams library, perhaps someone else with CVS write access might apply the attached jamfile patches related to that library to RC_1_34_0 (if the release manager approves) and to HEAD. The description of the patches is below. These, along with the mentioned modification to the bzip2 DEF file, should resolve the remaining iostreams regressions.
Once the jamfile patches are applied, I'll ping RudbekAssociates to delete their current test results, since I believe they run the regression tests incrementally and I don't think jamfile changes trigger a rebuild.
Thanks, -Dave
David Deakins wrote:
Douglas Gregor wrote:
|iostreams| bzip2_test: msvc-7.1 msvc-8.0 gzip_test: msvc-7.1 msvc-8.0 zlib_test: msvc-7.1 msvc-8.0
I spent a little bit of time looking into the msvc-7.1 and msvc-8.0 regressions for this library. It looks like the zlib and bzip2 libraries used in these tests are being built from source files and it appears that there are some omissions in the iostreams jamfile for properly building them as DLLs in Windows.
For zlib, there just needs to be a <link>shared:<define>ZLIB_DLL in its build requirements so that the zlib source files use the __declspec keyword on functions that should be exported. Without this, no functions are exported and the import library file is not created (thus the link errors).
With bzip2, the basic problem is the same but its sources use a .DEF file rather than __declspec (at least in the 1.0.4 sources that I looked at). Adding <link>shared:<def-file>$(BZIP2_SOURCE)/libbz2.def properly included the DEF file that gives the exports, but revealed another problem. The DEF file has a 'LIBRARY LIBBZ2' line in it which wants to give the DLL the name libbz2.DLL. Unfortunately the iostreams jam file typically calls the library something like boost_bzip2-vc80-mt-gd-1_34.DLL and specifies an /OUT directive to the linker to ask for this name. At least on my machine with VC8, the /OUT directive overrides the LIBRARY rule enough to make the intended boost_bzip2-vc80-mt-gd-1_34.DLL file, but not enough to get this name into the import .LIB file that boost_iostreams links to. As a result, the iostreams library will compile and link, but will fail trying locate libbz2.DLL (instead of the correct boost_bzip2-vc80-mt-gd-1_34.DLL). The only way I could easily resolve this was to comment out the LIBRARY rule in the libbz2.DEF file in the bzip2 sources like so:
; LIBRARY LIBBZ2
With this change and the patches to the iostreams jamfile, building zlib and bzip2 from sources works correctly and the 3 tests pass. I attached a patch for jamfile.v2 in libs\iostream\build with my changes. It might be good to put a note about the bzip2 .DEF file change in the iostreams documentation.
Incidently, using <def-file> revealed a small issue in msvc.jam related to DEF file paths that have spaces in them. The linker command does not put quotes around the DEF path like it does with most of the other parameters, and as a result DEF paths with spaces will cause the link to fail. I also included a patch to msvc.jam to add quotes around DEF file paths.
Thanks, -Dave
------------------------------------------------------------------------
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
-- Thomas Witt witt@acm.org
Thomas Witt wrote:
I am OK with this, but I won't have time to actually patch it myself till tomorrow evening.
I've checked in both patches to HEAD and RC. Thanks David! Victor, can you remove bin.v2/libs/iostream directory so that we have a clean run? David, you mentioned that manual work is required to tweak DEF file for bz2, is there no way around that? If so, would you be able to post a doc patch, or would rather like somebody else do it? Thanks, Volodya
Vladimir Prus wrote:
Victor, can you remove bin.v2/libs/iostream directory so that we have a clean run?
Victor, after you've cleared out bin.v2/libs/iostream and before you start the new test run, can you also locate the libbz2.DEF file in the bzip2 library sources and comment out the line with the LIBRARY directive like this: ; LIBRARY LIBBZ2
David, you mentioned that manual work is required to tweak DEF file for bz2, is there no way around that? If so, would you be able to post a doc patch, or would rather like somebody else do it?
I spent a little while trying to come up with an alternative to having to modify the libbz2.DEF file but I wasn't successful at doing it. The other path to fix this issue would be to make sure that the iostreams jamfile creates the bzip2 DLL with a name of libbz2.DLL (which is the standard name for it) rather than a name like boost_bzip2-vc80-mt-gd-1_34.DLL. I could get the first part of the name to change to libbz2 instead of boost_bzip2, but I wasn't able to figure out how to turn off the compiler-related decorations at the end. After reflecting on it further, I wasn't sure if turning off those tags was the right thing to do (if I actually knew how to do it) since that would mean all builds of boost_iostreams (potentially with different compilers and compiler options) would try to link to the same libbz2 file. That may not be a big deal in this case, but I wasn't sure, so I stuck with the somewhat ugly option of just changing the bzip2 DEF file. Perhaps someone with better Boost.Build knowledge and insight into the iostreams/bzip2 inner workings might be able to make the other path work. In case we stay with modifying the DEF file, I attached a patch for the iostreams/doc/installation.html file describing the modification. Thanks, -Dave
At 13:00 2007-03-20, David Deakins wrote:
Vladimir Prus wrote:
Victor, can you remove bin.v2/libs/iostream directory so that we have a clean run?
got that part done....
Victor, after you've cleared out bin.v2/libs/iostream and before you start the new test run, can you also locate the libbz2.DEF file in the bzip2 library sources and comment out the line with the LIBRARY directive like this:
looking for the file now
; LIBRARY LIBBZ2
David, you mentioned that manual work is required to tweak DEF file for bz2, is there no way around that? If so, would you be able to post a doc patch, or would rather like somebody else do it?
I spent a little while trying to come up with an alternative to having to modify the libbz2.DEF file but I wasn't successful at doing it. The other path to fix this issue would be to make sure that the iostreams jamfile creates the bzip2 DLL with a name of libbz2.DLL (which is the standard name for it) rather than a name like boost_bzip2-vc80-mt-gd-1_34.DLL. I could get the first part of the name to change to libbz2 instead of boost_bzip2, but I wasn't able to figure out how to turn off the compiler-related decorations at the end. After reflecting on it further, I wasn't sure if turning off those tags was the right thing to do (if I actually knew how to do it) since that would mean all builds of boost_iostreams (potentially with different compilers and compiler options) would try to link to the same libbz2 file. That may not be a big deal in this case, but I wasn't sure, so I stuck with the somewhat ugly option of just changing the bzip2 DEF file. Perhaps someone with better Boost.Build knowledge and insight into the iostreams/bzip2 inner workings might be able to make the other path work. In case we stay with modifying the DEF file, I attached a patch for the iostreams/doc/installation.html file describing the modification.
Thanks, -Dave
Index: installation.html =================================================================== RCS file: /cvsroot/boost/boost/libs/iostreams/doc/installation.html,v retrieving revision 1.8.2.1 diff -u -r1.8.2.1 installation.html --- installation.html 7 Jan 2007 22:40:33 -0000 1.8.2.1 +++ installation.html 20 Mar 2007 18:37:03 -0000 @@ -81,6 +81,15 @@ <P> On most UNIX systems, it should not be necessary to set any of these variables, since the zlib and libbz2 headers and binaries are already installed in locations where they will be found automatically. On Windows, the zlib and bzip2 filters are disabled by default; to enable support for these filters, you will need to set variables indicating the location of the source files, if building from the source, or the location of the headers and binaries, if using pre-built binaries. </P> +<P> + NOTE: When building the bzip2 library from the source as a DLL on Windows, a small change needs to be made in the bzip2 source files. As of version 1.0.4 of bzip2, a libbz2.DEF file is used to indicate the exported functions for Windows builds. This file also contains a line with a LIBRARY directive indicating the standard library name of LIBBZ2. Since the Boost.Build jamfiles prefer to build the bzip2 library with a boost_bzip2 name, there is a conflict with this LIBRARY directive. To allow the boost_iostreams DLL to successfully find the bzip2 DLL with its boost_bzip2 name, the LIBRARY directive line in libbz2.DEF should be commented out by placing a semicolon at the beginning of the line as follows: +</P> +<P> + ; LIBRARY LIBBZ2 +</P> +<P> + With this change to the DEF file, Boost.Build should be able to successfully generate a DLL version of the bzip2 library from its sources. +</P>
<TABLE STYLE="margin-bottom:1em" BORDER="1" CELLPADDING="6" CELLSPACING="1"> <TR>
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Victor A. Wagner Jr. http://rudbek.com The five most dangerous words in the English language: "There oughta be a law"
participants (4)
-
David Deakins -
Thomas Witt -
Victor A. Wagner Jr. -
Vladimir Prus