Exposing C++ Instances to Python

older
[interprocess] user application...

Leonhard Weber

25 May 2010 25 May '10

12:44 p.m.

Hello, I'm in the same bind this topic describes: http://lists.boost.org/boost-users/2009/02/44749.php Unfortunately the subject didn't tell about the resolution or approach he took to succeed. Can anyone help me? Basically: I want to expose a C++ Instance to Python at runtime. I have a pointer to the Instance and need to expose that instance to Python. Thankful for any help, Best regards, ~lwk

Attachments:

attachment.html (text/html — 522 bytes)

Show replies by date

Steven Watanabe

25 May 25 May

1:40 p.m.

AMDG Leonhard Weber wrote:

...

I'm in the same bind this topic describes: http://lists.boost.org/boost-users/2009/02/44749.php Unfortunately the subject didn't tell about the resolution or approach he took to succeed. Can anyone help me?

Basically: I want to expose a C++ Instance to Python at runtime. I have a pointer to the Instance and need to expose that instance to Python.

Have you tried the Boost.Python documentation? http://www.boost.org/libs/python/ In Christ, Steven Watanabe

Leonhard Weber

2:30 p.m.

Yes, but it seems I'm unable to see the wood for the trees. The concrete situation: Im developing a Interface that supposedly is going to enable AIs being written in Python for a C/C++ environment. What I get is a pointer to a Callback Instance I have to deliver to the AI dev (Python side). The Instance enables the AI dev to ask question about the Applications state so the AI reacts accordingly. Now I have a boost::module wrapper for the Class Im getting a reference of and successfully built a callback so the embedded Python is able to fetch the Callback Instance coming from the App. But I run into... Sandbox: // The Callback coming from the Application of which I am not allowed to taint the Source Tree struct EngineCallback { void setInt(int number) { this->number = number; } void setLoop(bool loop) { this->loop = loop; } int getInt() { return this->number; } bool getLoop() { return this->loop; } int number; bool loop; }; EngineCallback* test = new EngineCallback; // Entrypoint for Python to come ant fetch the Application Callback shared_ptr<EngineCallback> PyCallback( EngineCallback* &test ) { return shared_ptr<EngineCallback>(test); } // The Problem? ...how to convert? // The Wrapper for all this BOOST_PYTHON_MODULE(EngineCallback) { class_("EngineCallback", no_init) .def("setInt", &EngineCallback::setInt) .def("setLoop", &EngineCallback::setLoop) .def("getInt", &EngineCallback::getInt) .def("getLoop", &EngineCallback::getLoop) ; def("PyCallback", &PyCallback); } # test.py from EngineCallback import PyCallback from EngineCallback import EngineCallback test = PyCallback() dir(test)

...

...
...
...
...
...
ends in: Traceback (most recent call last): File "test.py", line 5, in <module> test = PyCallback() Boost.Python.ArgumentError: Python argument types in EngineCallback.PyCallback() did not match C++ signature: PyCallback(EngineCallback* {lvalue})

Conversions in PyCallback missing?, how/with what do I accomplish them? I think Im having trouble with C/C++ more in general than anything else, but I would appreciate if you could just point to the things I should look at to understand and come up with a solution. Thanks in advance, lwk On 25 May 2010 15:40, Steven Watanabe wrote:

...

AMDG

Leonhard Weber wrote:

...
I'm in the same bind this topic describes: http://lists.boost.org/boost-users/2009/02/44749.php Unfortunately the subject didn't tell about the resolution or approach he took to succeed. Can anyone help me?

Basically: I want to expose a C++ Instance to Python at runtime. I have a pointer to the Instance and need to expose that instance to Python.

Have you tried the Boost.Python documentation? http://www.boost.org/libs/python/

In Christ, Steven Watanabe

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org

http://lists.boost.org/mailman/listinfo.cgi/boost-users

Steven Watanabe

2:56 p.m.

AMDG Leonhard Weber wrote:

...

EngineCallback* test = new EngineCallback;

// Entrypoint for Python to come ant fetch the Application Callback shared_ptr<EngineCallback> PyCallback( EngineCallback* &test ); <snip> # test.py from EngineCallback import PyCallback from EngineCallback import EngineCallback

test = PyCallback()

PyCallback takes one argument. Hence the exception below. Did you intend to use the global "test" instead of taking test as an argument in PyCallback?

...

Traceback (most recent call last): File "test.py", line 5, in <module> test = PyCallback() Boost.Python.ArgumentError: Python argument types in EngineCallback.PyCallback() did not match C++ signature: PyCallback(EngineCallback* {lvalue})

In Christ, Steven Watanabe

Leonhard Weber

3:34 p.m.

Made a mistake... srry for my lacking knowledge/experience in C++. Trying to improve :D. For test purposes I changed the Python Script so it inits an instance, and then tries to fetch the instance through the PyCallback: ##### from EngineCallback import * a = EngineCallback() b = PyCallback(a)

...

...
...
a b

##### On the C++ side I had to change this, its now passing the argument by value (which I dont want) shared_ptr<EngineCallback> PyCallback( EngineCallback* test ) { return shared_ptr<EngineCallback>(test); } If I passed it per reference EngineCallback* &test (hope Im getting the jargon right and not mixing up definitions) it would say: did not match C++ signature: PyCallback(EngineCallback* {lvalue}) by changing it to pass by value it works, though isnt it making a copy of the instance? how do I get just a reference to the instance passed to Python? On 25 May 2010 16:56, Steven Watanabe wrote:

...

AMDG

Leonhard Weber wrote:

...
EngineCallback* test = new EngineCallback;

// Entrypoint for Python to come ant fetch the Application Callback shared_ptr<EngineCallback> PyCallback( EngineCallback* &test ); <snip>

# test.py from EngineCallback import PyCallback from EngineCallback import EngineCallback

test = PyCallback()

PyCallback takes one argument. Hence the exception below. Did you intend to use the global "test" instead of taking test as an argument in PyCallback?

Traceback (most recent call last):

...
File "test.py", line 5, in <module> test = PyCallback() Boost.Python.ArgumentError: Python argument types in EngineCallback.PyCallback() did not match C++ signature: PyCallback(EngineCallback* {lvalue})

In Christ, Steven Watanabe

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Steven Watanabe

4:14 p.m.

AMDG Leonhard Weber wrote:

...

Made a mistake... srry for my lacking knowledge/experience in C++. Trying to improve :D.

For test purposes I changed the Python Script so it inits an instance, and then tries to fetch the instance through the PyCallback: ##### from EngineCallback import *

a = EngineCallback() b = PyCallback(a)

...
...
...
a

...
...
...
b

##### On the C++ side I had to change this, its now passing the argument by value (which I dont want) shared_ptr<EngineCallback> PyCallback( EngineCallback* test ) { return shared_ptr<EngineCallback>(test); }

If I passed it per reference EngineCallback* &test (hope Im getting the jargon right and not mixing up definitions) it would say:

did not match C++ signature: PyCallback(EngineCallback* {lvalue})

by changing it to pass by value it works, though isnt it making a copy of the instance? how do I get just a reference to the instance passed to Python?

Since you take a pointer, the pointer will be copied, the object that it points to won't be copied. However, you should probably make the argument a shared_ptr. Otherwise, you'll run into trouble because the new shared_ptr thinks that it has exclusive ownership. In Christ, Steven Watanabe

XiongJia Le

4:16 p.m.

Are you finding "Boost.Python.register_ptr_to_python" ? Please look this document http://www.boost.org/doc/libs/1_41_0/libs/python/doc/v2/register_ptr_to_pyth... Is this sample same as you require ? Thanks, Xiongjia Le ( http://notes.xj-labs.net/ ) On Tue, May 25, 2010 at 11:34 PM, Leonhard Weber wrote:

...

Made a mistake... srry for my lacking knowledge/experience in C++. Trying to improve :D.

For test purposes I changed the Python Script so it inits an instance, and then tries to fetch the instance through the PyCallback: ##### from EngineCallback import *

a = EngineCallback() b = PyCallback(a)

...
...
...
a b

##### On the C++ side I had to change this, its now passing the argument by value (which I dont want) shared_ptr<EngineCallback> PyCallback( EngineCallback* test ) { return shared_ptr<EngineCallback>(test); }

If I passed it per reference EngineCallback* &test (hope Im getting the jargon right and not mixing up definitions) it would say:

did not match C++ signature: PyCallback(EngineCallback* {lvalue})

by changing it to pass by value it works, though isnt it making a copy of the instance? how do I get just a reference to the instance passed to Python?

On 25 May 2010 16:56, Steven Watanabe wrote:

...
AMDG

Leonhard Weber wrote:

...
EngineCallback* test = new EngineCallback;

// Entrypoint for Python to come ant fetch the Application Callback shared_ptr<EngineCallback> PyCallback( EngineCallback* &test ); <snip>

# test.py from EngineCallback import PyCallback from EngineCallback import EngineCallback

test = PyCallback()

PyCallback takes one argument. Hence the exception below. Did you intend to use the global "test" instead of taking test as an argument in PyCallback?

Traceback (most recent call last):

...
File "test.py", line 5, in <module> test = PyCallback() Boost.Python.ArgumentError: Python argument types in EngineCallback.PyCallback() did not match C++ signature: PyCallback(EngineCallback* {lvalue})

In Christ, Steven Watanabe

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Leonhard Weber

28 May 28 May

4:53 p.m.

Hi again, Thank you for your precious help up to now. I figured I had to see more and take more time working things out by myself before actually making use of your precious time. I've spent the last days building a testing environment to simulate the requirements of the Interface Implementation that is waiting for me. The Idea: An Interface that should make communication between the SpringRTS Engine and Python AI developers possible without tainting the SpringRTS sourcetree, and encapsulating as much as possible without invading or dictating "how to do it" on the Python side. 1) Engine starts the Interface (C/C++) 2) Engine tells the Interface to start a AI, passes a Callback that the Interface has to make accessible for the Python side 3) Interface starts the Python Interpreter and starts the actual AI 4) From Python, through the __init__.py the Client of the Interface fetches for the Callback instance and also registers some functions that are called from the Engine on ongoing happening Events (The topic that comes after finishing the current one which is encapsulating/automating the SAIFloat3 to and from PyTuple's) 5) The rest is up to the Client and the Engine For now the Interface is initialized from python. Here is the Code: (The entire thing is a OpenSource Project, so I don't really mind on publicly making this an example for other inexperienced users as myself) ################################## C++ Start #include #include #include #include #include using namespace boost; using namespace boost::python; // ============================================================= // Defining the to be passed Callback and SAIFloat3. // Callback: // Testing reference passing etc... // // SAIFloat3: // Testing of a nested custom Type (How to Wrap) // // Also some automatic converters for SAIFloat3 <--> Tuple // ============================================================= struct SAIFloat3 // As defined in SAIFloat3.h (SpringRTS ExternalAI Interface) { float x, y, z; }; // DUMMY CALLBACK FOR TESTING PURPOSES struct EngineCallback { void setInt(int number) { this->number = number; } void setLoop(bool loop) { this->loop = loop; } int getInt() const { return this->number; } bool getLoop() const { return this->loop; } void setVec1(float x, float y, float z) { vec1.x = x; vec1.y = y; vec1.z = z; } void setVec2(float x, float y, float z) { vec2.x = x; vec2.y = y; vec2.z = z; } SAIFloat3 getVec1() const { return vec1; } SAIFloat3 getVec2() const { return vec2; } SAIFloat3 vec1; SAIFloat3 vec2; int number; bool loop; }; // EngineCallback struct float3_to_python_tuple { static PyObject* convert(SAIFloat3 const& float3) { boost::python::list result; result.append(object(float3.x)); result.append(object(float3.y)); result.append(object(float3.z)); return boost::python::incref(boost::python::tuple(result).ptr()); } static const PyTypeObject* get_pytype() { return &PyTuple_Type; } }; struct float3_from_python_tuple { float3_from_python_tuple() { boost::python::converter::registry::push_back(&convertible, &construct, boost::python::type_id<SAIFloat3>()); } static void* convertible(PyObject* obj_ptr) { if (!(PyTuple_Check(obj_ptr))) return 0; int obj_size = PyObject_Length(obj_ptr); if (!(obj_size == 3) ) { // must contain 3 elements PyErr_Clear(); return 0; } return obj_ptr; } static void construct(PyObject* obj_ptr, boost::python::converter::rvalue_from_python_stage1_data* data) { void* storage = ( (boost::python::converter::rvalue_from_python_storage<SAIFloat3>*) data)->storage.bytes; new (storage) SAIFloat3(); data->convertible = storage; SAIFloat3& result = *((SAIFloat3*)storage); result.x = extract<float>(PyTuple_GetItem(obj_ptr, 0)); result.y = extract<float>(PyTuple_GetItem(obj_ptr, 1)); result.z = extract<float>(PyTuple_GetItem(obj_ptr, 2)); if (PyErr_Occurred()) boost::python::throw_error_already_set(); } }; // ============================================================= // Initializator and Instanceholder of a EngineCallback // object. // ============================================================= class PythonInterface { public: PythonInterface(); EngineCallback* exportCallback(); private: EngineCallback* instance_callback; }; PythonInterface::PythonInterface() { this->instance_callback = new EngineCallback; this->instance_callback->setInt(5); this->instance_callback->setLoop(true); } EngineCallback* PythonInterface::exportCallback() { return this->instance_callback; } // =================== // Wrapping all up // =================== BOOST_PYTHON_MODULE(PythonInterface) { to_python_converter< SAIFloat3, float3_to_python_tuple #ifdef BOOST_PYTHON_SUPPORTS_PY_SIGNATURES , true #endif >(); float3_from_python_tuple(); class_< SAIFloat3 >( "SAIFloat3" ) .def_readwrite("x", &SAIFloat3::x) .def_readwrite("y", &SAIFloat3::y) .def_readwrite("z", &SAIFloat3::z) ; class_< PythonInterface, shared_ptr<PythonInterface>

...

("PythonInterface") .def("exportCallback", &PythonInterface::exportCallback, return_value_policy()) ;

class_< EngineCallback >("EngineCallback") .def("setInt", &EngineCallback::setInt) .def("setLoop", &EngineCallback::setLoop) .def("getInt", &EngineCallback::getInt) .def("getLoop", &EngineCallback::getLoop) // RESPONSIBLE FOR THE COMPILE ERROR .def("setVec1", &EngineCallback::setVec1) .def("setVec2", &EngineCallback::setVec2) .def("getVec1", &EngineCallback::getVec1, return_value_policy()) .def("getVec2", &EngineCallback::getVec2, return_value_policy()) // .def("getVec1", &EngineCallback::getVec1, return_value_policy()) // .def("getVec2", &EngineCallback::getVec2, return_value_policy()) // .def("getVec1", &EngineCallback::getVec1, return_value_policy()) // .def("getVec2", &EngineCallback::getVec2, return_value_policy()) ; } // copy_const_reference ################################## C++ End Python testscript: ################################## Python Start from PythonInterface import PythonInterface

...

...
...
__main__:2: RuntimeWarning: to-Python converter for SAIFloat3 already registered; second conversion method ignored.

interface = PythonInterface() callback = interface.exportCallback() vec1 = callback.getVec1()

...

...
...
vec1 type(vec1) # Not exactly what I have in mind ################################## Python End

With that Error I don't even get close to testing the actual conversion code which is an adapted version of scitbx/boost_python/container_conversions.h. The main modification being, cutting on the generics and the endless Python Types checking (reduced to 'is it a tuple?') To be honest am not experienced enough in C/C++ to see if it works without actually feeling the pain. Another thing is the different return_value_policies: by_value, copy_const_reference(recommendet in the documentation for its efficiency), reference_existing_object (which gives a reference to the actual object omitting the desired conversion). On it shrieks at compiletime: ################################## Error Start In file included from /usr/include/boost/preprocessor/iteration/detail/iter/forward1.hpp:47, from /usr/include/boost/python/detail/invoke.hpp:63, from /usr/include/boost/python/detail/caller.hpp:16, from /usr/include/boost/python/object/function_handle.hpp:8, from /usr/include/boost/python/converter/arg_to_python.hpp:19, from /usr/include/boost/python/call.hpp:15, from /usr/include/boost/python/object_core.hpp:12, from /usr/include/boost/python/args.hpp:25, from /usr/include/boost/python.hpp:11, from PythonInterface.cpp:2: /usr/include/boost/python/detail/invoke.hpp: In function ‘PyObject* boost::python::detail::invoke(boost::python::detail::invoke_tag_, const RC&, F&, TC&) [with RC = boost::python::detail::caller_arity<1u>::impl::operator()(PyObject*, PyObject*) [with F = SAIFloat3 (EngineCallback::*)()const, Policies = boost::python::return_value_policy, Sig = boost::mpl::vector2]::result_converter, F = SAIFloat3 (EngineCallback::*)()const, TC = boost::python::detail::caller_arity<1u>::impl::operator()(PyObject*, PyObject*) [with F = SAIFloat3 (EngineCallback::*)()const, Policies = boost::python::return_value_policy, Sig = boost::mpl::vector2]::c_t0]’: /usr/include/boost/python/detail/caller.hpp:223: instantiated from ‘PyObject* boost::python::detail::caller_arity<1u>::impl::operator()(PyObject*, PyObject*) [with F = SAIFloat3 (EngineCallback::*)()const, Policies = boost::python::return_value_policy, Sig = boost::mpl::vector2]’ /usr/include/boost/python/object/py_function.hpp:38: instantiated from ‘PyObject* boost::python::objects::caller_py_function_impl<Caller>::operator()(PyObject*, PyObject*) [with Caller = boost::python::detail::caller, boost::mpl::vector2 >]’ PythonInterface.cpp:167: instantiated from here /usr/include/boost/python/detail/invoke.hpp:88: error: no match for call to ‘(const boost::python::detail::copy_const_reference_expects_a_const_reference_return_type<SAIFloat3>) (SAIFloat3)’ In file included from /usr/include/boost/python/object/function_handle.hpp:8, from /usr/include/boost/python/converter/arg_to_python.hpp:19, from /usr/include/boost/python/call.hpp:15, from /usr/include/boost/python/object_core.hpp:12, from /usr/include/boost/python/args.hpp:25, from /usr/include/boost/python.hpp:11, from PythonInterface.cpp:2: /usr/include/boost/python/detail/caller.hpp: In static member function ‘static const PyTypeObject* boost::python::detail::converter_target_type<ResultConverter>::get_pytype() [with ResultConverter = boost::python::detail::copy_const_reference_expects_a_const_reference_return_type<SAIFloat3>]’: /usr/include/boost/python/detail/caller.hpp:242: instantiated from ‘static boost::python::detail::py_func_sig_info boost::python::detail::caller_arity<1u>::impl::signature() [with F = SAIFloat3 (EngineCallback::*)()const, Policies = boost::python::return_value_policy, Sig = boost::mpl::vector2]’ /usr/include/boost/python/object/py_function.hpp:48: instantiated from ‘boost::python::detail::py_func_sig_info boost::python::objects::caller_py_function_impl<Caller>::signature() const [with Caller = boost::python::detail::caller, boost::mpl::vector2 >]’ PythonInterface.cpp:167: instantiated from here /usr/include/boost/python/detail/caller.hpp:102: error: ‘struct boost::python::detail::caller_arity<1u>::impl::operator()(PyObject*, PyObject*) [with F = SAIFloat3 (EngineCallback::*)()const, Policies = boost::python::return_value_policy, Sig = boost::mpl::vector2]::result_converter’ has no member named ‘get_pytype’ ################################## Error End Where are the mistakes? Where did I screw again? What concepts am I not understanding? Thanks, ~lwk On 25 May 2010 18:16, XiongJia Le wrote:

...

Are you finding "Boost.Python.register_ptr_to_python" ? Please look this document http://www.boost.org/doc/libs/1_41_0/libs/python/doc/v2/register_ptr_to_pyth...

Is this sample same as you require ?

Thanks, Xiongjia Le ( http://notes.xj-labs.net/ )

On Tue, May 25, 2010 at 11:34 PM, Leonhard Weber wrote:

...
Made a mistake... srry for my lacking knowledge/experience in C++. Trying to improve :D.

For test purposes I changed the Python Script so it inits an instance, and then tries to fetch the instance through the PyCallback: ##### from EngineCallback import *

a = EngineCallback() b = PyCallback(a)

...
...
...
a b

##### On the C++ side I had to change this, its now passing the argument by value (which I dont want) shared_ptr<EngineCallback> PyCallback( EngineCallback* test ) { return shared_ptr<EngineCallback>(test); }

If I passed it per reference EngineCallback* &test (hope Im getting the jargon right and not mixing up definitions) it would say:

did not match C++ signature: PyCallback(EngineCallback* {lvalue})

by changing it to pass by value it works, though isnt it making a copy of the instance? how do I get just a reference to the instance passed to Python?

On 25 May 2010 16:56, Steven Watanabe wrote:

...
AMDG

Leonhard Weber wrote:

...
EngineCallback* test = new EngineCallback;

// Entrypoint for Python to come ant fetch the Application Callback shared_ptr<EngineCallback> PyCallback( EngineCallback* &test ); <snip>

# test.py from EngineCallback import PyCallback from EngineCallback import EngineCallback

test = PyCallback()

PyCallback takes one argument. Hence the exception below. Did you intend to use the global "test" instead of taking test as an argument in PyCallback?

Traceback (most recent call last):

...
File "test.py", line 5, in <module> test = PyCallback() Boost.Python.ArgumentError: Python argument types in EngineCallback.PyCallback() did not match C++ signature: PyCallback(EngineCallback* {lvalue})

In Christ, Steven Watanabe

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

5280

Age (days ago)

5283

Last active (days ago)

List overview

Download

7 comments

3 participants

participants (3)

Leonhard Weber
Steven Watanabe
XiongJia Le