[python] partially resolved: the never ending story of vc8/win32 linker settings desaster
Hi! So now since I wasted 2 days figuring out the interaction of Python with boost::python and Visual Studio 2005 SP1 (VC8) I finally managed to resolve the "VC8/Debug Crashes" problem. To be honest: I am shocked and I need someone to explain why things are intentionally broken the way they are. All boost libraries play nicely with *any* build system, but the settings for Boost.Python are twisted in ways that make me feel absolutely uncomfortable. Admittedly some of the headaches have their reason inside Python itself and somehow Visual Studio plays ugly, too, but IMHO it is time to rework the settings shipped with Boost.Python. I managed to resolve some of the troubles I had with VS2005. If you try to run a debug version of Boost.Python with Debug-Python you are on your own: Boost.Python(Debug) deliberately links to Python(Release) which is the source of major headache: 1. With VS2005 it is NOT a GoodIdea(TM) to mix different types of binaries for one executable, since the (stupid) linker of vs2005 succeeds without error sometimes creating executables that fail to be debugged and/or crash at random points during execution. Summary: VC8 linker sucks in many aspects, so don't do that. This is a request to change the settings of the debug version Boost.Python, even if this forces the user to download the python sources. What do you think? 2. You can, of course, build Python with _DEBUG settings. This again breaks everything: Python-2.5.2, sets Py_DEBUG and then in file Include/object.h, line 58 decides that 54 /* Py_DEBUG implies Py_TRACE_REFS. */ 55 #if defined(Py_DEBUG) && !defined(Py_TRACE_REFS) 56 #define Py_TRACE_REFS // WE HAVE A PROBLEM HERE!!!!!!!! 57 #endif Autodefined Py_TRACE_REFS itself yields Py_InitModule4 be renamed to Py_InitModule4TraceRefs in modsupport.h: #ifdef Py_TRACE_REFS /* When we are tracing reference counts, rename Py_InitModule4 so modules compiled with incompatible settings will generate a link-time error. */ ... #define Py_InitModule4 Py_InitModule4TraceRefs #endif Unfortunately two things lead to a headache here: First, boost::python::detail::init_module *always* calls Py_InitModule4 regardless of the compiler settings, which I consider critical here. See libs/python/src/module.cpp for what I mean. Second, vc8 linker - for reasons beyond my limited horizon - during search of symbols in my self-created python_d.lib/dll fails to find Py_InitModule4, requested by boost::python::detail::init_module, but happens to find python.lib/dll in the same directory and (BOOM) silently links against python.lib/dll (OUCH!) which yields a truly broken executable, the nightmare of debugging stuff that should not have compiled. I suspect Boost.Python settings to be resposible for this. Please correct me if I am wrong here (of course I may have missed something): Summary so far: If you want to debug an embedded Boost.Python/Python with vc8 you have to fetch the Python sources, change file Include/object.h, line 56 such that 54 /* Py_DEBUG implies Py_TRACE_REFS. */ 55 #if defined(Py_DEBUG) && !defined(Py_TRACE_REFS) 56 // #define Py_TRACE_REFS // WE HAVE A PROBLEM HERE!!!!!!!! 57 #endif then after rebuilding the "Debug" version of PCbuild8\pcbuild.sln go to your own project, Open the project settings linker settings and add python.lib to the "Ignore Specific Library" section. Copy the python_d.dll and .lib files to your project and everything is fine. Phew. Done. Compiles. Runs. Does not Crash. Don't you think we can do better? IMHO the content of libs/python/src/module.cpp belongs into a header that allows users to control its behaviour and that nicely interacts with the original Pythion settings. To be resolved: I failed to create a static executable that statically links to boost::python and Python, since boost::python static libs ask for python.lib/dll symbols beginning with __imp__ instead of taking those in my self-named libpython-mt-sgd.lib. Can you provide a resolution for this task? Boost.Build settings? Wrong library included? Markus
I think you can always choose which libraries to link with. If you don't want python.lib/dll then don't include it or add it to the ignore library list. If adding explicitely python_d.lib leads to unresolved symbols then you are compiling the library and your source code using different settings. I suggest you to add the linker option /VERBOSE:LIB, it will show you the searched libraries, useful for "linker debugging". -----Messaggio originale----- Second, vc8 linker - for reasons beyond my limited horizon - during search of symbols in my self-created python_d.lib/dll fails to find Py_InitModule4, requested by boost::python::detail::init_module, but happens to find python.lib/dll in the same directory and (BOOM) silently links against python.lib/dll (OUCH!) which yields a truly broken executable, the nightmare of debugging stuff that should not have compiled. I suspect Boost.Python settings to be resposible for this. Please correct me if I am wrong here (of course I may have missed something):
Andrea Denzler <andrea <at> andreaplanet.com> writes:
I think you can always choose which libraries to link with. If you don't want python.lib/dll then don't include it or add it to the ignore library list.
You completely missed the point.
I suggest you to add the linker option /VERBOSE:LIB
Of course I use /VERBOSE to find out what is going on. Markus
Sorry. Generally I have no problems linking (Debug executables) to Release libraries. For example in the linker options I specify icuuc.lib or icuucd.lib to link boost::regex to the ICU Debug or Release Library. Both cases works well in Debugging. So the idea to force the debug python libraries is a overkill. Now, when using boost::python in _DEBUG then it is better to use python too in _DEBUG or it's standard release version? I think it's better to use the release version because else as you said you have to download and rebuild the python library only because I want boost in _DEBUG. Of course the best would be to modify boost::python so that we can choose if to use python debug or release, even better if we can choose this at link time like I do with other libraries. IMHO... -----Messaggio originale----- Da: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] Per conto di Markus Werle Inviato: mercoledì 13 agosto 2008 8.22 A: boost-users@lists.boost.org Oggetto: Re: [Boost-users] [python] partially resolved: the never endingstoryof vc8/win32 linker settings desaster Andrea Denzler <andrea <at> andreaplanet.com> writes:
I think you can always choose which libraries to link with. If you don't want python.lib/dll then don't include it or add it to the ignore library list.
You completely missed the point.
I suggest you to add the linker option /VERBOSE:LIB
Of course I use /VERBOSE to find out what is going on. Markus _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Andrea Denzler <andrea <at> andreaplanet.com> writes:
Sorry. Generally I have no problems linking (Debug executables) to Release libraries.
Then you are lucky, or your executables are smaller than 2MB. I have internal compiler/linker errors every day (nice to see VS is built below drive F:), not only crashing Visual Studio 2005, but sometimes also forcing me to reboot. If only this mad behaviour was reproducible. It starts to remind me of gcc-3.0 on hpux-10.2 (2 alpha systems in unholy combination)
For example in the linker options I specify icuuc.lib or icuucd.lib to link boost::regex to the ICU Debug or Release Library. Both cases works well in Debugging.
How can you debug into the library, if the library has no debug symbols?
So the idea to force the debug python libraries is a overkill.
I strongly disagree. It is the normal thing to do. If I debug, I debug and if I run release code I run release lib versions. Also it is not about forcing anyone, but about the fact that it is impossible due to the symbol incompatibility.
Now, when using boost::python in _DEBUG then it is better to use python too in _DEBUG or it's standard release version? I think it's better to use the release version because else as you said you have to download and rebuild the python library only because I want boost in _DEBUG.
This is what it is all about. Boost.Python tries to fix the broken release scheme of Python and by this introduces silent breaks and endless headache with uninituitive behaviour. What is wrong with downloading python sources, clicking on a predefined .sln file, pressing the key combination SHIFT-B, waiting 3 minutes, opening the explorer and copyying the libs to the correct place?
Of course the best would be to modify boost::python so that we can choose if to use python debug or release,
This is excatly what I want. I am looking forward to comments whether my solution idea of moving init calls to a header is OK or not.
even better if we can choose this at link time like I do with other libraries.
Yes. Markus
Good to know that I am lucky with the compiler/linker (exe's over 2 MB). It's always a bad experience to get internal compiler/linker errors. I can accept a bug in a Office application but not in the compiler. When I debug my code I want to find my bugs, so I am not always interested in debugging other libraries, especially if they are huge and complex. If I use the library release version of course I don't have any debug info on that library but I have my code (and other libraries) in debug. Also a library build in debug mode will have a different behavior than in release mode. Maybe a hard to discover bug in my code is present only when using the library in release mode due to different compiler optimizations. At last a library in debug build is slower, sometimes much slower causing an unacceptable overhead when testing heavy computation tasks. So I definitively want always choose to use the debug or release version when I am debugging my application. I don't think there is a incompatibility when using libraries in release build and your application in debug. Building in _DEBUG has three effects: 1) adding symbols: how this can create an incompatibility is unknown to me. 2) different compiler settings: internal optimizations also don't create incompatibilities. Only the boundaries between the library and your source code is affected, see next point. 3) different library behaviors: if the library use for example different data structures when building in debug/release (i.e. adding extra members to classes for debug info) and those structures are handled by your source code too then your application will likely crash. But this is a library bug not a VS bug. If the library use different structures this should be uniquely identified using global symbols like Py_DEBUG so that the headers define correctly the internal structures of the library, the used function calls, the correct byte alignment, etc. Sure, if VS is full of bugs and the workaround for you is to use all libraries in _DEBUG then let's go on with this, but without forcing it to everyone. Developers should be able to choose if they can use python.lib in release or debug when compiling the application in _DEBUG. Which one is the default option is not that important. One point is that you had troubles using python in debug build, and this should definitively be fixed in the boost::python. And probably you are one of the few trying to do this else whoever use python is unable to build the application in debug mode. But sometimes we are one of the few, like when I wanted to use boost::regex and Unicode ICU, but boost 1.35 was unable to do a complete build (already fixed in the current svn code). Andrea -----Messaggio originale----- Da: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] Per conto di Markus Werle Inviato: giovedì 14 agosto 2008 9.34 A: boost-users@lists.boost.org Oggetto: Re: [Boost-users][python] partially resolved: the never endingstoryof vc8/win32 linker settings desaster Andrea Denzler <andrea <at> andreaplanet.com> writes:
Sorry. Generally I have no problems linking (Debug executables) to Release libraries.
Then you are lucky, or your executables are smaller than 2MB. I have internal compiler/linker errors every day (nice to see VS is built below drive F:), not only crashing Visual Studio 2005, but sometimes also forcing me to reboot. If only this mad behaviour was reproducible. It starts to remind me of gcc-3.0 on hpux-10.2 (2 alpha systems in unholy combination)
For example in the linker options I specify icuuc.lib or icuucd.lib to link boost::regex to the ICU Debug or Release Library. Both cases works well in Debugging.
How can you debug into the library, if the library has no debug symbols?
So the idea to force the debug python libraries is a overkill.
I strongly disagree. It is the normal thing to do. If I debug, I debug and if I run release code I run release lib versions. Also it is not about forcing anyone, but about the fact that it is impossible due to the symbol incompatibility.
Now, when using boost::python in _DEBUG then it is better to use python too in _DEBUG or it's standard release version? I think it's better to use the release version because else as you said you have to download and rebuild the python library only because I want boost in _DEBUG.
This is what it is all about. Boost.Python tries to fix the broken release scheme of Python and by this introduces silent breaks and endless headache with uninituitive behaviour. What is wrong with downloading python sources, clicking on a predefined .sln file, pressing the key combination SHIFT-B, waiting 3 minutes, opening the explorer and copyying the libs to the correct place?
Of course the best would be to modify boost::python so that we can choose if to use python debug or release,
even better if we can choose this at link time like I do with other
This is excatly what I want. I am looking forward to comments whether my solution idea of moving init calls to a header is OK or not. libraries. Yes. Markus _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On Tue, Aug 12, 2008 at 4:47 PM, Markus Werle <numerical.simulation@web.de> wrote:
This is a request to change the settings of the debug version Boost.Python, even if this forces the user to download the python sources. What do you think?
From what you write I'm not sure if you know about the option "python-debugging=on", which you can pass to bjam. I have only used this with shared libraries for everything, so I'm not sure whether it solves all your problems.
Just mentioning it, in case it helps. The resulting boost libs are then named "gyd" instead of the usual "gd", and (the shared libs) link to python25_d. Then you need to define BOOST_DEBUG_PYTHON when building your project. Does this make any difference for your issues? Thomas
Thomas Berg <merlin66b <at> gmail.com> writes:
On Tue, Aug 12, 2008 at 4:47 PM, Markus Werle <numerical.simulation <at> web.de> wrote:
This is a request to change the settings of the debug version Boost.Python, even if this forces the user to download the python sources. What do you think?
From what you write I'm not sure if you know about the option "python-debugging=on", which you can pass to bjam.
Thanks for that hint. Unfortunately this option builds *all* boost libraries for this option which makes absolutely no sense at all.
I have only used this with shared libraries for everything, so I'm not sure whether it solves all your problems.
Just mentioning it, in case it helps. The resulting boost libs are then named "gyd" instead of the usual "gd", and (the shared libs) link to python25_d.
Then you need to define BOOST_DEBUG_PYTHON when building your project. Does this make any difference for your issues?
This thing still lacks full user control via PREPOCESSOR MACRO, such that auto-link really works seamlessly for the three options that people want to use. 1. boost-python release version with Python release version 2. boost-python debug version with Python debug version 3. boost-python debug version with Python release version. I'll dig further into this in order to propose some changes. Markus
participants (3)
-
Andrea Denzler -
Markus Werle -
Thomas Berg