[python] threading-related failures on Linux
The OSL2-V2 test results show a few link failiures for Python tests, for example the 'bases' test: http://tinyurl.com/nyqv7 Looks like Boost.Python library wants multi-threading for some reason. In V1, those tests are linked with -pthread, and in V2, they are not. The logic responsible for this in V1 is in python.jam: # Force statically-linked embedding applications to be multithreaded # on UNIX. rule python-static-multithread ( toolset variant : properties * ) { if ! $(PYTHON_WINDOWS) { local x = <define>BOOST_PYTHON_STATIC_LIB <threading>single ; if $(x) in $(properties) { properties = [ difference $(properties) : <threading>single ] <threading>multi ; } } return $(properties) ; } And I don't understand some things: 1. What's the point of this at all? The comment is not very clear, it seems it means static linking to Boost.Python, not statically linked application in the usual sense (no shared libs at all). The comment does not say why threading multi is needed. 2. In V2, shared linking to Boost.Python is used, and the application is fully shared. So, where does the link error come from? Is comment even more inaccurate? 3. On my system, the test works just fine. Why does it fail elsewhere/ 4. The test in question, bases.cpp, has nothing but static asserts. It's not clear to me why it should link to Boost.Python at all. Can somebody explain this to me? Thanks in advance, Volodya
Dave, can you comment on the below message? The problem shows up not only for OSL2 but also for Martin, and I'd rather implement the right solution, not one that happens to work with just the regression tests. Thanks in advance, Volodya
The OSL2-V2 test results show a few link failiures for Python tests, for example the 'bases' test:
Looks like Boost.Python library wants multi-threading for some reason. In V1, those tests are linked with -pthread, and in V2, they are not. The logic responsible for this in V1 is in python.jam:
# Force statically-linked embedding applications to be multithreaded # on UNIX. rule python-static-multithread ( toolset variant : properties * ) { if ! $(PYTHON_WINDOWS) { local x = <define>BOOST_PYTHON_STATIC_LIB <threading>single ; if $(x) in $(properties) { properties = [ difference $(properties) : <threading>single ] <threading>multi ; } } return $(properties) ; }
And I don't understand some things:
1. What's the point of this at all? The comment is not very clear, it seems it means static linking to Boost.Python, not statically linked application in the usual sense (no shared libs at all). The comment does not say why threading multi is needed.
2. In V2, shared linking to Boost.Python is used, and the application is fully shared. So, where does the link error come from? Is comment even more inaccurate?
3. On my system, the test works just fine. Why does it fail elsewhere/
4. The test in question, bases.cpp, has nothing but static asserts. It's not clear to me why it should link to Boost.Python at all.
Can somebody explain this to me?
Thanks in advance, Volodya
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Vladimir Prus <ghost@cs.msu.su> writes:
The OSL2-V2 test results show a few link failiures for Python tests, for example the 'bases' test:
Isn't this just the same issue we've seen several times: the Jamfile.v2 fails to properly translate the v1 logic (in this case leaving out the effects of <template>py-unit-test)?
Looks like Boost.Python library wants multi-threading for some reason.
It's not just -lpthread, but also -ldl that is missing.
In V1, those tests are linked with -pthread, and in V2, they are not. The logic responsible for this in V1 is in python.jam:
# Force statically-linked embedding applications to be multithreaded # on UNIX. rule python-static-multithread ( toolset variant : properties * ) { if ! $(PYTHON_WINDOWS) { local x = <define>BOOST_PYTHON_STATIC_LIB <threading>single ; if $(x) in $(properties) { properties = [ difference $(properties) : <threading>single ] <threading>multi ; } } return $(properties) ; }
And I don't understand some things:
1. What's the point of this at all?
It made things work :) IIRC, libpythonXX.a on *nix requires these libraries. Why is this mysterious?
The comment is not very clear, it seems it means static linking to Boost.Python,
Yes.
not statically linked application in the usual sense (no shared libs at all). The comment does not say why threading multi is needed.
There are two usual ways to use Python on *nix: 1. extending, where the Python executable dynamically loads Python extension modules and they use the Python API that's statically linked into the Python executable. In this model, the Python executable is linked to whatever libraries it needs, such as pthreads and dl. 2. embedding, where a user-written executable uses the Python API in libpythonXX.a. In that case, who is going to link to pthreads and dl if not the user-written executable.
2. In V2, shared linking to Boost.Python is used, and the application is fully shared. So, where does the link error come from? Is comment even more inaccurate?
Than what?
3. On my system, the test works just fine. Why does it fail elsewhere/
Different unices and Python builds have slightly different requirements.
4. The test in question, bases.cpp, has nothing but static asserts. It's not clear to me why it should link to Boost.Python at all.
In this case you could probably turn it from a "run" test into a "compile" test, then.
Can somebody explain this to me?
-- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Looks like Boost.Python library wants multi-threading for some reason.
It's not just -lpthread, but also -ldl that is missing.
The 'dl' is already added in CVS, we've talked with Martin about it and figured out that on some Unixes Python links to dl and on others don't.
In V1, those tests are linked with -pthread, and in V2, they are not. The logic responsible for this in V1 is in python.jam:
# Force statically-linked embedding applications to be multithreaded # on UNIX. rule python-static-multithread ( toolset variant : properties * ) { if ! $(PYTHON_WINDOWS) { local x = <define>BOOST_PYTHON_STATIC_LIB <threading>single ; if $(x) in $(properties) { properties = [ difference $(properties) : <threading>single ] <threading>multi ; } } return $(properties) ; }
And I don't understand some things:
1. What's the point of this at all?
It made things work :)
IIRC, libpythonXX.a on *nix requires these libraries. Why is this mysterious?
Why it requires -pthread only whem <define>BOOST_PYTHON_STATIC_LIB is in properties. Everything else is not a mistery.
The comment is not very clear, it seems it means static linking to Boost.Python,
Yes.
not statically linked application in the usual sense (no shared libs at all). The comment does not say why threading multi is needed.
There are two usual ways to use Python on *nix:
1. extending, where the Python executable dynamically loads Python extension modules and they use the Python API that's statically linked into the Python executable. In this model, the Python executable is linked to whatever libraries it needs, such as pthreads and dl.
2. embedding, where a user-written executable uses the Python API in libpythonXX.a. In that case, who is going to link to pthreads and dl if not the user-written executable.
Then, how <define>BOOST_PYTHON_STATIC_LIB is related to those cases. It's quite feasible to embed python and use shared linking to Boost.Python at the same time.
2. In V2, shared linking to Boost.Python is used, and the application is fully shared. So, where does the link error come from? Is comment even more inaccurate?
Than what?
I mean, -pthread is required for all embedding, even if linking to Boost.Python is shared. Right?
4. The test in question, bases.cpp, has nothing but static asserts. It's not clear to me why it should link to Boost.Python at all.
In this case you could probably turn it from a "run" test into a "compile" test, then.
Ok. Thanks for explanation. I'll think about best way to distinguish between extending and embedding. - Volodya
Vladimir Prus wrote:
David Abrahams wrote:
Looks like Boost.Python library wants multi-threading for some reason.
It's not just -lpthread, but also -ldl that is missing.
The 'dl' is already added in CVS, we've talked with Martin about it and figured out that on some Unixes Python links to dl and on others don't.
Not only on "some Unixes"; the way libpython gets built on Linux systems differs among the different distributions. We also know of having to add -lutil for pty related functions. Regards, m Send instant messages to your online friends http://au.messenger.yahoo.com
Martin Wille wrote:
Vladimir Prus wrote:
David Abrahams wrote:
Looks like Boost.Python library wants multi-threading for some reason.
It's not just -lpthread, but also -ldl that is missing.
The 'dl' is already added in CVS, we've talked with Martin about it and figured out that on some Unixes Python links to dl and on others don't.
Not only on "some Unixes"; the way libpython gets built on Linux systems differs among the different distributions.
We also know of having to add -lutil for pty related functions.
The way this is typically solved is by querying the library / tool about its dependencies (via autotools, i.e. configure, say). I'm not sure whether boost.build has something equivalent... Regards, Stefan
Stefan Seefeld wrote:
Martin Wille wrote:
Vladimir Prus wrote:
David Abrahams wrote:
Looks like Boost.Python library wants multi-threading for some reason.
It's not just -lpthread, but also -ldl that is missing.
The 'dl' is already added in CVS, we've talked with Martin about it and figured out that on some Unixes Python links to dl and on others don't.
Not only on "some Unixes"; the way libpython gets built on Linux systems differs among the different distributions.
We also know of having to add -lutil for pty related functions.
The way this is typically solved is by querying the library / tool about its dependencies (via autotools, i.e. configure, say). I'm not sure whether boost.build has something equivalent...
In case of Python, everything should be built in. Using python -c "import distutils.sysconfig; print\ distutils.sysconfig.get_config_var('LIBS') one can query a lot of information from the installed Python package. Among the interesting variables to query are: LIBS, LIBPL, VERSION, LIBDIR, LIBDEST Using those, the build system could figure out most, probably even all, it needs. Regards, m Send instant messages to your online friends http://au.messenger.yahoo.com
Martin Wille wrote:
In case of Python, everything should be built in.
Using
python -c "import distutils.sysconfig; print\ distutils.sysconfig.get_config_var('LIBS')
one can query a lot of information from the installed Python package. Among the interesting variables to query are:
LIBS, LIBPL, VERSION, LIBDIR, LIBDEST
Using those, the build system could figure out most, probably even all, it needs.
In theory you are right. In practice, however, distutils reported wrong libraries in one particular case (cygwin, IIRC). Oh well... Regards, Stefan
Stefan Seefeld wrote:
Not only on "some Unixes"; the way libpython gets built on Linux systems differs among the different distributions.
We also know of having to add -lutil for pty related functions.
The way this is typically solved is by querying the library / tool about its dependencies (via autotools, i.e. configure, say). I'm not sure whether boost.build has something equivalent...
We don't have any configure-like solution at the moment. Care to work on one? ;-) - Volodya
Vladimir Prus <ghost@cs.msu.su> writes:
Stefan Seefeld wrote:
Not only on "some Unixes"; the way libpython gets built on Linux systems differs among the different distributions.
We also know of having to add -lutil for pty related functions.
The way this is typically solved is by querying the library / tool about its dependencies (via autotools, i.e. configure, say). I'm not sure whether boost.build has something equivalent...
We don't have any configure-like solution at the moment. Care to work on one? ;-)
Well, we do have the SHELL rule. Not much more than that is needed, is it? -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Vladimir Prus <ghost@cs.msu.su> writes:
Stefan Seefeld wrote:
Not only on "some Unixes"; the way libpython gets built on Linux systems differs among the different distributions.
We also know of having to add -lutil for pty related functions.
The way this is typically solved is by querying the library / tool about its dependencies (via autotools, i.e. configure, say). I'm not sure whether boost.build has something equivalent...
We don't have any configure-like solution at the moment. Care to work on one? ;-)
Well, we do have the SHELL rule. Not much more than that is needed, is it?
There are two approaches. 1. You have separate configure step. Some tests are compiled and run (much like C++ Boost regression tests) and produce some files. Then, Boost.Build reads those files, decides if tests failed or passed and makes some decision. For this, SHELL is fine. 2. Ideally, the tests are automatically run when out-of-date. The results of tests immediate affect Jamfile logic. For example: if [ configure.run-test have_dlopen.cpp ] { sources += dlopen_using_code.cpp ; } This requires that we be able to update some targets before we've finished reading all Jamfiles, and is not supported at the moment. - Volodya
Vladimir Prus <ghost@cs.msu.su> writes:
There are two approaches.
1. You have separate configure step. Some tests are compiled and run (much like C++ Boost regression tests) and produce some files. Then, Boost.Build reads those files, decides if tests failed or passed and makes some decision. For this, SHELL is fine.
2. Ideally, the tests are automatically run when out-of-date. The results of tests immediate affect Jamfile logic. For example:
if [ configure.run-test have_dlopen.cpp ] { sources += dlopen_using_code.cpp ; }
This requires that we be able to update some targets before we've finished reading all Jamfiles, and is not supported at the moment.
Is it a bad idea to think about using a recursive bjam invocation for this sort of thing? That is, configure.run-test could invoke bjam to update the target associated with have_dlopen. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Vladimir Prus <ghost@cs.msu.su> writes:
There are two approaches.
1. You have separate configure step. Some tests are compiled and run (much like C++ Boost regression tests) and produce some files. Then, Boost.Build reads those files, decides if tests failed or passed and makes some decision. For this, SHELL is fine.
2. Ideally, the tests are automatically run when out-of-date. The results of tests immediate affect Jamfile logic. For example:
if [ configure.run-test have_dlopen.cpp ] { sources += dlopen_using_code.cpp ; }
This requires that we be able to update some targets before we've finished reading all Jamfiles, and is not supported at the moment.
Is it a bad idea to think about using a recursive bjam invocation for this sort of thing? That is, configure.run-test could invoke bjam to update the target associated with have_dlopen.
This is possible, but I'm concerned about performance implications of this. It's quite feasible to have 100, or 200 tests, running separate bjam for each one might be too slow. Maybe updating targets in the middle will be easier. - Volodya
Vladimir Prus <ghost@cs.msu.su> writes:
David Abrahams wrote:
This requires that we be able to update some targets before we've finished reading all Jamfiles, and is not supported at the moment.
Is it a bad idea to think about using a recursive bjam invocation for this sort of thing? That is, configure.run-test could invoke bjam to update the target associated with have_dlopen.
This is possible, but I'm concerned about performance implications of this. It's quite feasible to have 100, or 200 tests, running separate bjam for each one might be too slow.
You could run one bjam for all the tests
Maybe updating targets in the middle will be easier.
Maybe; seems dicey to me. -- Dave Abrahams Boost Consulting www.boost-consulting.com
Do I understand that the issue is only relevant to systems that require static linking for boost::python testing?
David Abrahams wrote:
Vladimir Prus <ghost@cs.msu.su> writes:
David Abrahams wrote:
This requires that we be able to update some targets before we've finished reading all Jamfiles, and is not supported at the moment.
Is it a bad idea to think about using a recursive bjam invocation for this sort of thing? That is, configure.run-test could invoke bjam to update the target associated with have_dlopen.
This is possible, but I'm concerned about performance implications of this. It's quite feasible to have 100, or 200 tests, running separate bjam for each one might be too slow.
You could run one bjam for all the tests
It's tricky. Say you have: if [ configure.has-lib foo ] { } if [ configure.has-lib bar ] { } Them if you invoke bjam once for configuration, from 'configure.has-lib', how that configure step knows that it should check not only for 'foo' but also for 'bar', when 'has-lib' for bar is not yet called. So, you'd have to have a separate file describing configure checks, not the above syntax. This is passable, but not ideal. - Volodya
Vladimir Prus <ghost@cs.msu.su> writes:
David Abrahams wrote:
Vladimir Prus <ghost@cs.msu.su> writes:
This is possible, but I'm concerned about performance implications of this. It's quite feasible to have 100, or 200 tests, running separate bjam for each one might be too slow.
You could run one bjam for all the tests
It's tricky. Say you have:
if [ configure.has-lib foo ] { }
if [ configure.has-lib bar ] { }
Them if you invoke bjam once for configuration, from 'configure.has-lib', how that configure step knows that it should check not only for 'foo' but also for 'bar', when 'has-lib' for bar is not yet called.
So, you'd have to have a separate file describing configure checks, not the above syntax. This is passable, but not ideal.
It's more important to make things work than to make them ideal. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
It's tricky. Say you have:
if [ configure.has-lib foo ] { }
if [ configure.has-lib bar ] { }
Them if you invoke bjam once for configuration, from 'configure.has-lib', how that configure step knows that it should check not only for 'foo' but also for 'bar', when 'has-lib' for bar is not yet called.
So, you'd have to have a separate file describing configure checks, not the above syntax. This is passable, but not ideal.
It's more important to make things work than to make them ideal.
Anyway, this is more about doing work, then which approach is better. I don't have a pressing need for configure step yet, but if I run into a personal project where configure is really needed, I'll see what can be done. - Volodya
Vladimir Prus <ghost@cs.msu.su> writes:
David Abrahams wrote:
It made things work :)
IIRC, libpythonXX.a on *nix requires these libraries. Why is this mysterious?
Why it requires -pthread only whem <define>BOOST_PYTHON_STATIC_LIB is in properties.
I don't know, unfortunately.
There are two usual ways to use Python on *nix:
1. extending, where the Python executable dynamically loads Python extension modules and they use the Python API that's statically linked into the Python executable. In this model, the Python executable is linked to whatever libraries it needs, such as pthreads and dl.
2. embedding, where a user-written executable uses the Python API in libpythonXX.a. In that case, who is going to link to pthreads and dl if not the user-written executable.
Then, how <define>BOOST_PYTHON_STATIC_LIB is related to those cases.
It shouldn't be.
It's quite feasible to embed python and use shared linking to Boost.Python at the same time.
Yes.
2. In V2, shared linking to Boost.Python is used, and the application is fully shared. So, where does the link error come from? Is comment even more inaccurate?
Than what?
I mean, -pthread is required for all embedding, even if linking to Boost.Python is shared. Right?
I believe so. -- Dave Abrahams Boost Consulting www.boost-consulting.com
Vladimir Prus <ghost@cs.msu.su> writes:
Thanks for explanation. I'll think about best way to distinguish between extending and embedding.
Seems to me that <lib>pthread (or however we spell it) should be a usage-requirement of the target representing the python library... or if we're not going to have a target for libpythonxx.a, it should be a usage-requirement of the Boost.Python library. -- Dave Abrahams Boost Consulting www.boost-consulting.com
participants (5)
-
David Abrahams -
Martin Wille -
Neal Becker -
Stefan Seefeld -
Vladimir Prus