Pooyan Dadvand wrote:

Hi again,

In this project the variable is in charge of storing all information about any data (like DISPLACEMENT) which is its name, default value, unique key, etc, and perform all necessary operations with its type of data. In serializing std::pair who knows the type of the value pointed by the void* is the Variable pointed by a base class pointer. So for serializing I need to call a method of Variable (which has the type information) and pass the void* which is the value to be store. This value can be a simple double, ublas matrix or even more complex classes. I know that this looks strange from OO point of view but the variable itself don't have any pointer to its value because there are few hundred variables in Kratos and in a normal analysis we have to store millions of values in the data structure. So using the variable as handler of its value will cause a very big overhead of memory which is not acceptable.

Thanks,

Pooyan.

-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Robert Ramey Sent: 18 August 2010 18:18 To: boost-users@lists.boost.org Subject: Re: [Boost-users] Boost Serialization calling a custom serialize method

Pooyan Dadvand wrote:

...

Hi,

I'm trying to use the boost serialization in our project "Kratos" (a Multi-physics finite element code) in which I found the following problem, I have a class Variable which is derived from VariableData as follow:

Class VariableData { std::size_t mKey; public: ..... };

template<class TDataType> class Variable : public VariableData { .... }

Seems to me that what would work is:

Class VariableData{ std::size_t m_Key; public: virtual ~VariableData() = 0; };

template<class TDataType> class Veriable : public VariableData { ... }

This would work with the library in a normal way.

...

Now I have a container which holds a pairs, consist of VariableData pointer to the Variable and a void* pointer to its value:

std::vector >

I can't understand what the "void *" could possibly be useful for here. Nor can I understand why you would use it. The whole idea of trying to serialize a "void *" seems totally non-sensical to me. I have no idea how or way one would want to do this.

Robert Ramey

maybe what you want is to replace: template<class TArchiveType> Variable::serialize(TArchiveType &rArchive, const unsigned int Version, void* pValue) { rArchive & static_cast<TDataType>(pVAlue); } with template<class TArchiveType> Variable::serialize(TArchiveType &rArchive, const unsigned int Version, void* pValue) { rArchive & static_cast(pVAlue); } which still looks like a bad idea to me. But it might be what you want. Robert Ramey

Consider that the data.first is a pointer to the base class and I want to call the serialize of the derived class which is not virtual

...

which still looks like a bad idea to me. But it might be what you want.

Really I'm not aware of a storage mechanism for a heterogeneous container without using virtual functions in time of accessing data which means less performace for us. In this sudo code the accessing mechanism is not menssioned but it is fast and typesafe (it can be seen here http://kratos.cimne.upc.es/trac/browser/kratos/kratos/containers/data_value_...)

Of course any suggestion will be very welcome!!

First of all - focusing on this aspect of performance at this stage in development is a bad idea. Try the following: a) Make your container interface. b) Implement it using the the easiest method c) Make a test program for it d) Profile it e) Enhance the implementation to improve performance bottlenecks. That is the fastest way to make the fastest code. To do this (e), you need to know more about how the serialization library works internally. Most of this is available in the documentation - but it would also require investment of effort in studying some of the dozens of examples and tests included with the library. The serialization library only uses virtual types for a couple of special types inside the library implemenation. Even re-constitution of virtual types created by the user don't use any calls to virtual functions. This should be apparent from study of the document tests and examples. Also, for a heterogenous container, you might look at several approaches. One is a container of boost.variant or boost.any. The former doesn't use virtual functions at all. Study of all of this seems like a lot of work - and it is. But ultimately it will be faster and give a better result than trying to speculate on an optimal solution and then trying to make the serialization library fit it. The easiest and best way to make a 12 inch telescope lens is to make a 2 inch one first. Robert Ramey

Pooyan Dadvand wrote:

Hi,

First of all thank you for your kind suggestion. I completely agree about suggested methodology for optimizing. (Indeed this is the methodology I use in my work) However here my concern is the performance in normal use of container and not in seriliaztion and optimizing the serialization will be a secondary objective.

What I need to know is how to call a MyCostumSerialize method of my derive object with an additional parameter via a pointer to the base class:

Class VariableData{ std::size_t m_Key; public:

template<class TArchiveType> MyCostumSerialize(TArchiveType &rArchive, const unsigned int Version, void* pValue) {}

};

template<class TDataType> class Veriable : public VariableData { public:

template<class TArchiveType> MyCostumSerialize(TArchiveType &rArchive, const unsigned int Version, void* pValue) { rArchive & static_cast(pVAlue); }

}

Let say I don't how to call the MyCostumSerialize method of "Variable" class above when I have a "VariableData*" pointer in my container:

std::vector >

Just rename MyCostumeSerialize to serialize Robert Ramey

Pooyan Dadvand wrote:

The question is not the name of the method, the problem is the additional parameter which boost::serialization cannot pass to me. For this reason I have to call my own serialize method inside the standard serialize method prepared for boost::serialization.

template<class TArchiveType> MyContainer::serialize(TArchiveType &rArchive, const unsigned int Version) { p_variable_data->MyCustomSerialize(rArchive, Version, MyData); }

try the following: template<class TArchiveType> MyContainer::serialize(TArchiveType &rArchive, const unsigned int Version) { rArchive << p_variable_data; } or template<class TArchiveType> MyContainer::serialize(TArchiveType &rArchive, const unsigned int Version) { rArchive << std::pair<?, ?>(p_variable_data, MyData); } I'm sure that there is an easy way to use the library to get the effect you want. But it will require effort to see it. Robert Ramey

_______________________________________________________________________________________________________________________________________________________________________________ Dr. Pooyan Dadvand International Center for Numerical Methods in Engineering - CIMNE Campus Norte, Edificio C1 c/ Gran Capitán s/n 08034 Barcelona, Spain Tel: (+34) 93 401 56 96 Fax: (+34) 93 401 65 17 web: www.cimne.com <http://www.cimne.com http://www.cimne.com/> _______________________________________________________________________________________________________________________________________________________________________________ *AVISO IMPORTANTE *Los datos de carácter personal contenidos en el mensaje, se registrarán en un fichero para facilitar la gestión de las comunicaciones de CIMNE. Se pueden ejercitar los derechos de acceso, rectificación, cancelación y oposición por escrito, dirigiéndose a nuestras oficinas de CIMNE, Gran Capitán s/n, Edificio C1 - Campus Norte UPC, 08034 Barcelona, España. *AVÍS IMPORTANT *Les dades de caràcter personal contingudes en aquest missatge es registraran en un fitxer per facilitar la gestió de les comunicacions del CIMNE. Es poden exercir els drets d'accés, rectificació, cancel·lació i oposició, per escrit a les nostres oficines del CIMNE, Gran Capità s/n, Edifici C1, Campus Nord UPC, 08034 Barcelona, Espanya. *IMPORTANT NOTICE *All personal data contained in this mail will be processed confidentially and stored in a file property of CIMNE in order to manage corporate communications. You may exercise the right of access, rectification, deletion and objection by letter sent to CIMNE, Gran Capitán, Edificio C1 - Campus Norte UPC, 08034 Barcelona, Spain. On 08/25/2010 09:39 PM, Robert Ramey wrote:

Pooyan Dadvand wrote:

...

The question is not the name of the method, the problem is the additional parameter which boost::serialization cannot pass to me. For this reason I have to call my own serialize method inside the standard serialize method prepared for boost::serialization.

template<class TArchiveType> MyContainer::serialize(TArchiveType&rArchive, const unsigned int Version) { p_variable_data->MyCustomSerialize(rArchive, Version, MyData); }

try the following:

template<class TArchiveType> MyContainer::serialize(TArchiveType&rArchive, const unsigned int Version) { rArchive<< p_variable_data; }

or

template<class TArchiveType> MyContainer::serialize(TArchiveType&rArchive, const unsigned int Version) { rArchive<< std::pair<?, ?>(p_variable_data, MyData); }

I'm sure that there is an easy way to use the library to get the effect you want.

I'm sure too. But How? Do you know any boost tool to get the pointer to the derive class like this: template<class TArchiveType> MyContainer::serialize(TArchiveType&rArchive, const unsigned int Version) { boost_tool_to_get_drived_class_pointer(p_variable_data)->MyCustomSerialize(rArchive, Version, MyData); } Or tell me how the following code in oserializer.hpp can be used for my case template<class T> static void save( Archive&ar, T& t ){ BOOST_DEDUCED_TYPENAME boost::serialization::type_info_implementation<T>::type const & i = boost::serialization::singleton< BOOST_DEDUCED_TYPENAME boost::serialization::type_info_implementation<T>::type >::get_const_instance(); boost::serialization::extended_type_info const * const this_type =& i; // retrieve the true type of the object pointed to // if this assertion fails its an error in this library assert(NULL != this_type); const boost::serialization::extended_type_info * true_type = i.get_derived_extended_type_info(t); // note:if this exception is thrown, be sure that derived pointer // is either registered or exported. if(NULL == true_type){ boost::serialization::throw_exception( archive_exception( archive_exception::unregistered_class, true_type->get_debug_info() ) ); } // if its not a pointer to a more derived type const void *vp = static_cast(&t); if(*this_type == *true_type){ const basic_pointer_oserializer * bpos = register_type(ar, t); ar.save_pointer(vp, bpos); return; } // convert pointer to more derived type. if this is thrown // it means that the base/derived relationship hasn't be registered vp = serialization::void_downcast( *true_type, *this_type, static_cast(&t) ); if(NULL == vp){ boost::serialization::throw_exception( archive_exception( archive_exception::unregistered_cast, true_type->get_debug_info(), this_type->get_debug_info() ) ); } // since true_type is valid, and this only gets made if the // pointer oserializer object has been created, this should never // fail const basic_pointer_oserializer * bpos = static_cast( boost::serialization::singleton< archive_serializer_map<Archive> >::get_const_instance().find(*true_type) ); assert(NULL != bpos); if(NULL == bpos) boost::serialization::throw_exception( archive_exception( archive_exception::unregistered_class, bpos->get_debug_info() ) ); ar.save_pointer(vp, bpos); } };

But it will require effort to see it.

Robert Ramey

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Thanks, Pooyan.

_______________________________________________________________________________________________________________________________________________________________________________ Dr. Pooyan Dadvand International Center for Numerical Methods in Engineering - CIMNE Campus Norte, Edificio C1 c/ Gran Capitán s/n 08034 Barcelona, Spain Tel: (+34) 93 401 56 96 Fax: (+34) 93 401 65 17 web: www.cimne.com <http://www.cimne.com http://www.cimne.com/> _______________________________________________________________________________________________________________________________________________________________________________ *AVISO IMPORTANTE *Los datos de carácter personal contenidos en el mensaje, se registrarán en un fichero para facilitar la gestión de las comunicaciones de CIMNE. Se pueden ejercitar los derechos de acceso, rectificación, cancelación y oposición por escrito, dirigiéndose a nuestras oficinas de CIMNE, Gran Capitán s/n, Edificio C1 - Campus Norte UPC, 08034 Barcelona, España. *AVÍS IMPORTANT *Les dades de caràcter personal contingudes en aquest missatge es registraran en un fitxer per facilitar la gestió de les comunicacions del CIMNE. Es poden exercir els drets d'accés, rectificació, cancel·lació i oposició, per escrit a les nostres oficines del CIMNE, Gran Capità s/n, Edifici C1, Campus Nord UPC, 08034 Barcelona, Espanya. *IMPORTANT NOTICE *All personal data contained in this mail will be processed confidentially and stored in a file property of CIMNE in order to manage corporate communications. You may exercise the right of access, rectification, deletion and objection by letter sent to CIMNE, Gran Capitán, Edificio C1 - Campus Norte UPC, 08034 Barcelona, Spain. On 08/27/2010 07:55 PM, Robert Ramey wrote:

Pooyan Dadvand wrote:

...

Do you know any boost tool to get the pointer to the derive class like this:

template<class TArchiveType> MyContainer::serialize(TArchiveType&rArchive, const unsigned int Version) {

boost_tool_to_get_drived_class_pointer(p_variable_data)->MyCustomSerialize(rArchive, Version, MyData); }

I'm not going to tell you how to do this because doing this won't help you !

Reread the section of the documentation which addresses serialization of pointers. It discusses the serialization of an object through a pointer to the virtual base class

I have already read it!

so all you have to do is

base_class *m_bc; // where base class is a virtual base class.

ar<< m_dc

I'm already using this for serializing a derived class and it works fine, but it's not the case here.

The serialization library keeps track of the "true" (most derived) class and call the serialize function on THAT class. You don't have to do anything to get what you want.

It keep track of "serialize" method with two parameters and I need an additional parameter!

I would suggest you look through the tests/examples (there are 50+ in the library!) and find one that does this and study it.

...

Or tell me how the following code in oserializer.hpp can be used for my case

<snip>

the above example ar<< m_dc will in fact use that code.

One time we had a similar thread. The final result was

"Oh - this is really simple - once you see it!"

So if what you want to do is NOT simple, you're doing it wrong.

Of course I'm doing something wrong, for this reason I'm asking the correct way. however I don't see an easy solution to my problem looking to the examples for the very standard cases.

Start make taking one of your types serialize it make a small test to prove that it works.

I have already done this.

Add the next higher type and repeat

Exactly the methodology I'm using.

make a test which does the above through a pointer and repeat

I have already done this and it works.

add a virtual base class and repeat

I have already serialized several classes in my library with pointer to the base classes and they are working. But it's not what I'm asking. What I'm asking is a very special case in which the serialize don't have enough information to deal with my void* in std::pair and for this reason I need the help of VariableData derived class to give the serialization the necessary type information. Actually I get the following error using the normal serialization method as you suggest: /usr/local/include/boost/archive/detail/oserializer.hpp:465: error: 'void*' is not a pointer-to-object type

Robert Ramey

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Thanks, Pooyan.

_______________________________________________________________________________________________________________________________________________________________________________ Dr. Pooyan Dadvand International Center for Numerical Methods in Engineering - CIMNE Campus Norte, Edificio C1 c/ Gran Capitán s/n 08034 Barcelona, Spain Tel: (+34) 93 401 56 96 Fax: (+34) 93 401 65 17 web: www.cimne.com <http://www.cimne.com http://www.cimne.com/> _______________________________________________________________________________________________________________________________________________________________________________ *AVISO IMPORTANTE *Los datos de carácter personal contenidos en el mensaje, se registrarán en un fichero para facilitar la gestión de las comunicaciones de CIMNE. Se pueden ejercitar los derechos de acceso, rectificación, cancelación y oposición por escrito, dirigiéndose a nuestras oficinas de CIMNE, Gran Capitán s/n, Edificio C1 - Campus Norte UPC, 08034 Barcelona, España. *AVÍS IMPORTANT *Les dades de caràcter personal contingudes en aquest missatge es registraran en un fitxer per facilitar la gestió de les comunicacions del CIMNE. Es poden exercir els drets d'accés, rectificació, cancel·lació i oposició, per escrit a les nostres oficines del CIMNE, Gran Capità s/n, Edifici C1, Campus Nord UPC, 08034 Barcelona, Espanya. *IMPORTANT NOTICE *All personal data contained in this mail will be processed confidentially and stored in a file property of CIMNE in order to manage corporate communications. You may exercise the right of access, rectification, deletion and objection by letter sent to CIMNE, Gran Capitán, Edificio C1 - Campus Norte UPC, 08034 Barcelona, Spain. On 08/27/2010 08:50 PM, Robert Ramey wrote:

Pooyan Dadvand wrote:

...

But it's not what I'm asking. What I'm asking is a very special case in which the serialize don't have enough information to deal with my void* in std::pair and for this reason I need the help of VariableData derived class to give the serialization the necessary type information. Actually I get the following error using the normal serialization method as you suggest:

/usr/local/include/boost/archive/detail/oserializer.hpp:465: error: 'void*' is not a pointer-to-object type

ahh - Ok

std::pair m_x ... ar<< m_x

of course won't work- my mistake - since there is no way to determine what the true type is.

So how about one of the following?

struct extradata { ... };

std::pair m_x std::pair m_x std::pair m_x std::pair m_x ... ar<< m_x;

Can you please explain me more the idea of extradata structure? Consider that my extradata can be a double, ublas::vector, or even std::vector or any new type of variables. I have: std::vector > mData ... ar<< mData; and each element of vector can have a different type of data with different size or even have pointers. (It's a heterogeneous container) In this situation I don't know how to implement the extradata structure without knowing the type of variable I'm storing in it.

Robert Ramey

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Bests, Pooyan.

Pooyan Dadvand wrote:

Can you please explain me more the idea of extradata structure? Consider that my extradata can be a double, ublas::vector, or even std::vector or any new type of variables. I have:

std::vector > mData ... ar << mData;

and each element of vector can have a different type of data with different size or even have pointers. (It's a heterogeneous container)

The above will work as written. Maybe what you want is: template<class T> class mytype : std::vector > { template<class Archive> void serialize(Archive &ar, const unsigned int version){ ar << boost::serialization::base_object(*this); } }; then you can mytype<double> m_t ... ar << m_t The possibilities are endless. This really isn't about the serialization library but about C++ Robert Ramey