Re: [Boost-users] Exposing C++ Instances to Python

28 May 2010

      Hi again,

Thank you for your precious help up to now. I figured I had to see more and
take more time working things out by myself before actually making use of
your precious time.
I've spent the last days building a testing environment to simulate the
requirements of the Interface Implementation that is waiting for me.

The Idea:
An Interface that should make communication between the SpringRTS Engine and
Python AI developers possible without tainting the SpringRTS sourcetree, and
encapsulating as much as possible without invading or dictating "how to do
it" on the Python side.
1) Engine starts the Interface (C/C++)
2) Engine tells the Interface to start a AI, passes a Callback that the
Interface has to make accessible for the Python side
3) Interface starts the Python Interpreter and starts the actual AI
4) From Python, through the __init__.py the Client of the Interface fetches
for the Callback instance and also registers some functions that are called
from the Engine on ongoing happening Events (The topic that comes after
finishing the current one which is encapsulating/automating the SAIFloat3 to
and from PyTuple's)
5) The rest is up to the Client and the Engine

For now the Interface is initialized from python.

Here is the Code: (The entire thing is a OpenSource Project, so I don't
really mind on publicly making this an example for other inexperienced users
as myself)
################################## C++ Start
#include 
#include 
#include 
#include 
#include 

using namespace boost;
using namespace boost::python;

// =============================================================
//     Defining the to be passed Callback and SAIFloat3.
//         Callback:
//             Testing reference passing etc...
//
//         SAIFloat3:
//             Testing of a nested custom Type (How to Wrap)
//
//      Also some automatic converters for SAIFloat3 <--> Tuple
// =============================================================
struct SAIFloat3 // As defined in SAIFloat3.h (SpringRTS ExternalAI
Interface)
{
    float x, y, z;
};

// DUMMY CALLBACK FOR TESTING PURPOSES
struct EngineCallback
{
    void setInt(int number) { this->number = number; }
    void setLoop(bool loop) { this->loop = loop; }
    int getInt() const { return this->number; }
    bool getLoop() const { return this->loop; }

    void setVec1(float x, float y, float z) { vec1.x = x; vec1.y = y; vec1.z
= z; }
    void setVec2(float x, float y, float z) { vec2.x = x; vec2.y = y; vec2.z
= z; }
    SAIFloat3 getVec1() const { return vec1; }
    SAIFloat3 getVec2() const { return vec2; }

    SAIFloat3 vec1;
    SAIFloat3 vec2;
    int number;
    bool loop;
}; // EngineCallback

struct float3_to_python_tuple
{
    static PyObject* convert(SAIFloat3 const& float3)
    {
        boost::python::list result;
        result.append(object(float3.x));
        result.append(object(float3.y));
        result.append(object(float3.z));

        return boost::python::incref(boost::python::tuple(result).ptr());
    }

    static const PyTypeObject* get_pytype() { return &PyTuple_Type; }
};

struct float3_from_python_tuple
{
    float3_from_python_tuple()
    {
        boost::python::converter::registry::push_back(&convertible,
                                                      &construct,

boost::python::type_id<SAIFloat3>());
    }

    static void* convertible(PyObject* obj_ptr)
    {
        if (!(PyTuple_Check(obj_ptr)))
            return 0;

        int obj_size = PyObject_Length(obj_ptr);
        if (!(obj_size == 3) ) { // must contain 3 elements
            PyErr_Clear();
            return 0;
        }

        return obj_ptr;
    }

    static void construct(PyObject* obj_ptr,

boost::python::converter::rvalue_from_python_stage1_data* data)
    {
        void* storage = (

(boost::python::converter::rvalue_from_python_storage<SAIFloat3>*)
            data)->storage.bytes;
        new (storage) SAIFloat3();
        data->convertible = storage;
        SAIFloat3& result = *((SAIFloat3*)storage);

        result.x = extract<float>(PyTuple_GetItem(obj_ptr, 0));
        result.y = extract<float>(PyTuple_GetItem(obj_ptr, 1));
        result.z = extract<float>(PyTuple_GetItem(obj_ptr, 2));

        if (PyErr_Occurred())
            boost::python::throw_error_already_set();
    }
};

// =============================================================
//     Initializator and Instanceholder of a EngineCallback
//  object.
// =============================================================
class PythonInterface {
    public:
        PythonInterface();

        EngineCallback* exportCallback();

    private:
        EngineCallback* instance_callback;
};

PythonInterface::PythonInterface() {
    this->instance_callback = new EngineCallback;
    this->instance_callback->setInt(5);
    this->instance_callback->setLoop(true);
}

EngineCallback* PythonInterface::exportCallback() {
    return this->instance_callback;
}

//  ===================
//     Wrapping all up
//  ===================
BOOST_PYTHON_MODULE(PythonInterface)
{
    to_python_converter< SAIFloat3, float3_to_python_tuple
#ifdef BOOST_PYTHON_SUPPORTS_PY_SIGNATURES
        , true
#endif
        >();
    float3_from_python_tuple();

    class_< SAIFloat3 >( "SAIFloat3" )
        .def_readwrite("x", &SAIFloat3::x)
        .def_readwrite("y", &SAIFloat3::y)
        .def_readwrite("z", &SAIFloat3::z)
        ;

    class_< PythonInterface, shared_ptr<PythonInterface>
...
("PythonInterface")
        .def("exportCallback", &PythonInterface::exportCallback,
return_value_policy())
    ;
class_< EngineCallback >("EngineCallback")
        .def("setInt", &EngineCallback::setInt)
        .def("setLoop", &EngineCallback::setLoop)
        .def("getInt", &EngineCallback::getInt)
        .def("getLoop", &EngineCallback::getLoop)

        // RESPONSIBLE FOR THE COMPILE ERROR
        .def("setVec1", &EngineCallback::setVec1)
        .def("setVec2", &EngineCallback::setVec2)
        .def("getVec1", &EngineCallback::getVec1,
return_value_policy())
        .def("getVec2", &EngineCallback::getVec2,
return_value_policy())
//         .def("getVec1", &EngineCallback::getVec1,
return_value_policy())
//         .def("getVec2", &EngineCallback::getVec2,
return_value_policy())
//         .def("getVec1", &EngineCallback::getVec1,
return_value_policy())
//         .def("getVec2", &EngineCallback::getVec2,
return_value_policy())
    ;
}
// copy_const_reference
################################## C++ End

Python testscript:
################################## Python Start
from PythonInterface import PythonInterface
...
...
...
__main__:2: RuntimeWarning: to-Python converter for SAIFloat3 already
registered; second conversion method ignored.
interface = PythonInterface()
callback = interface.exportCallback()

vec1 = callback.getVec1()
...
...
...
vec1

type(vec1)
 # Not exactly what I have in mind
################################## Python End
With that Error I don't even get close to testing the actual conversion code
which is an adapted version of
scitbx/boost_python/container_conversions.h. The main modification being,
cutting on the generics and the endless Python
Types checking (reduced to 'is it a tuple?')
To be honest am not experienced enough in C/C++ to see if it works without
actually feeling the pain.

Another thing is the different return_value_policies: by_value,
copy_const_reference(recommendet in the documentation for its efficiency),
reference_existing_object (which gives a reference to the actual object
omitting the desired conversion).
On  it shrieks at compiletime:

################################## Error Start
In file included from
/usr/include/boost/preprocessor/iteration/detail/iter/forward1.hpp:47,
                 from /usr/include/boost/python/detail/invoke.hpp:63,
                 from /usr/include/boost/python/detail/caller.hpp:16,
                 from
/usr/include/boost/python/object/function_handle.hpp:8,
                 from
/usr/include/boost/python/converter/arg_to_python.hpp:19,
                 from /usr/include/boost/python/call.hpp:15,
                 from /usr/include/boost/python/object_core.hpp:12,
                 from /usr/include/boost/python/args.hpp:25,
                 from /usr/include/boost/python.hpp:11,
                 from PythonInterface.cpp:2:
/usr/include/boost/python/detail/invoke.hpp: In function ‘PyObject*
boost::python::detail::invoke(boost::python::detail::invoke_tag_, const RC&, F&, TC&) [with RC =
boost::python::detail::caller_arity<1u>::impl::operator()(PyObject*, PyObject*) [with F = SAIFloat3
(EngineCallback::*)()const, Policies =
boost::python::return_value_policy, Sig = boost::mpl::vector2]::result_converter, F = SAIFloat3
(EngineCallback::*)()const, TC =
boost::python::detail::caller_arity<1u>::impl::operator()(PyObject*, PyObject*) [with F = SAIFloat3
(EngineCallback::*)()const, Policies =
boost::python::return_value_policy, Sig = boost::mpl::vector2]::c_t0]’:
/usr/include/boost/python/detail/caller.hpp:223:   instantiated from
‘PyObject* boost::python::detail::caller_arity<1u>::impl::operator()(PyObject*, PyObject*) [with F = SAIFloat3
(EngineCallback::*)()const, Policies =
boost::python::return_value_policy, Sig = boost::mpl::vector2]’
/usr/include/boost/python/object/py_function.hpp:38:   instantiated from
‘PyObject*
boost::python::objects::caller_py_function_impl<Caller>::operator()(PyObject*,
PyObject*) [with Caller = boost::python::detail::caller, boost::mpl::vector2 >]’
PythonInterface.cpp:167:   instantiated from here
/usr/include/boost/python/detail/invoke.hpp:88: error: no match for call to
‘(const
boost::python::detail::copy_const_reference_expects_a_const_reference_return_type<SAIFloat3>)
(SAIFloat3)’
In file included from
/usr/include/boost/python/object/function_handle.hpp:8,
                 from
/usr/include/boost/python/converter/arg_to_python.hpp:19,
                 from /usr/include/boost/python/call.hpp:15,
                 from /usr/include/boost/python/object_core.hpp:12,
                 from /usr/include/boost/python/args.hpp:25,
                 from /usr/include/boost/python.hpp:11,
                 from PythonInterface.cpp:2:
/usr/include/boost/python/detail/caller.hpp: In static member function
‘static const PyTypeObject*
boost::python::detail::converter_target_type<ResultConverter>::get_pytype()
[with ResultConverter =
boost::python::detail::copy_const_reference_expects_a_const_reference_return_type<SAIFloat3>]’:
/usr/include/boost/python/detail/caller.hpp:242:   instantiated from ‘static
boost::python::detail::py_func_sig_info
boost::python::detail::caller_arity<1u>::impl::signature()
[with F = SAIFloat3 (EngineCallback::*)()const, Policies =
boost::python::return_value_policy, Sig = boost::mpl::vector2]’
/usr/include/boost/python/object/py_function.hpp:48:   instantiated from
‘boost::python::detail::py_func_sig_info
boost::python::objects::caller_py_function_impl<Caller>::signature() const
[with Caller = boost::python::detail::caller, boost::mpl::vector2 >]’
PythonInterface.cpp:167:   instantiated from here
/usr/include/boost/python/detail/caller.hpp:102: error: ‘struct
boost::python::detail::caller_arity<1u>::impl::operator()(PyObject*, PyObject*) [with F = SAIFloat3
(EngineCallback::*)()const, Policies =
boost::python::return_value_policy, Sig = boost::mpl::vector2]::result_converter’ has no member named ‘get_pytype’
################################## Error End

Where are the mistakes? Where did I screw again? What concepts am I not
understanding?

Thanks,
~lwk

On 25 May 2010 18:16, XiongJia Le  wrote:
...
Are you finding  "Boost.Python.register_ptr_to_python" ?
Please look this document
http://www.boost.org/doc/libs/1_41_0/libs/python/doc/v2/register_ptr_to_pyth...
Is this sample same as you require ?
Thanks,
Xiongjia Le ( http://notes.xj-labs.net/ )
On Tue, May 25, 2010 at 11:34 PM, Leonhard Weber wrote:
...
Made a mistake... srry for my lacking knowledge/experience in C++. Trying
to improve :D.
For test purposes I changed the Python Script so it inits an instance, and
then tries to fetch the instance through the PyCallback:
#####
from EngineCallback import *
a = EngineCallback()
b = PyCallback(a)
...
...
...
a

b

#####
On the C++ side I had to change this, its now passing the argument by
value (which I dont want)
shared_ptr<EngineCallback> PyCallback( EngineCallback* test ) { return
shared_ptr<EngineCallback>(test); }
If I passed it per reference EngineCallback* &test (hope Im getting the
jargon right and not mixing up definitions) it would say:
did not match C++ signature:
    PyCallback(EngineCallback* {lvalue})
by changing it to pass by value it works, though isnt it making a copy of
the instance? how do I get just a reference to the instance passed to
Python?
On 25 May 2010 16:56, Steven Watanabe  wrote:
...
AMDG
Leonhard Weber wrote:
...
EngineCallback* test = new EngineCallback;
// Entrypoint for Python to come ant fetch the Application Callback
shared_ptr<EngineCallback> PyCallback( EngineCallback* &test );
<snip>
# test.py
from EngineCallback import PyCallback
from EngineCallback import EngineCallback
test = PyCallback()
PyCallback takes one argument.  Hence the exception below.
Did you intend to use the global "test" instead of taking test
as an argument in PyCallback?
Traceback (most recent call last):
...
File "test.py", line 5, in <module>
   test = PyCallback()
Boost.Python.ArgumentError: Python argument types in
   EngineCallback.PyCallback()
did not match C++ signature:
   PyCallback(EngineCallback* {lvalue})
In Christ,
Steven Watanabe
_______________________________________________
Boost-users mailing list
Boost-users@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________
Boost-users mailing list
Boost-users@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________
Boost-users mailing list
Boost-users@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users