[Serialization] Zero-sized arrays
Hi, i am planning to use the boost serialization library to implement checkpointing for an ASP/Sat solver. Unfortunately, the solver implements one (very special) optimization that makes use of a zero-sized array at the end of a class. I have found no way to store something like this using the library. There is a small program in the attachment that contains data structures similar to the ones used in the solver code. This would need a mechanism to control the allocation of memory during deserialization. Do you see any possibility do serialize something like this? Regards and thanks, Roland
On 3 Aug 2009, at 10:51, Roland Kaminski wrote:
Hi, i am planning to use the boost serialization library to implement checkpointing for an ASP/Sat solver. Unfortunately, the solver implements one (very special) optimization that makes use of a zero-sized array at the end of a class. I have found no way to store something like this using the library. There is a small program in the attachment that contains data structures similar to the ones used in the solver code. This would need a mechanism to control the allocation of memory during deserialization. Do you see any possibility do serialize something like this?
Regards and thanks, Roland
Uhhhh .... that is dangerous code. and this object is neither Assignable nor Copyable. Do you really need to make use of that optimization? In essence you need to control the memory allocation: Base *x = new(new unsigned char[sizeof(Derived) + 42 * sizeof(int)]) Derived(42); and have the serialization library do such a new operation instead of the default one. Robert, is there a hook for that? Matthias
The code is full of such "dangerous" optimizations but most of them can be somehow handled. It is optimized to be as cache-friendly as possible that is why I would like to not remove this one. A hook during deserialization would be exactly what I need. I could also imagine other worarounds but all of them are very ugly. Regards, Roland On Monday 03 August 2009 23:12:50 Matthias Troyer wrote:
On 3 Aug 2009, at 10:51, Roland Kaminski wrote:
Hi, i am planning to use the boost serialization library to implement checkpointing for an ASP/Sat solver. Unfortunately, the solver implements one (very special) optimization that makes use of a zero-sized array at the end of a class. I have found no way to store something like this using the library. There is a small program in the attachment that contains data structures similar to the ones used in the solver code. This would need a mechanism to control the allocation of memory during deserialization. Do you see any possibility do serialize something like this?
Regards and thanks, Roland
Uhhhh .... that is dangerous code. and this object is neither Assignable nor Copyable. Do you really need to make use of that optimization? In essence you need to control the memory allocation:
Base *x = new(new unsigned char[sizeof(Derived) + 42 * sizeof(int)]) Derived(42);
and have the serialization library do such a new operation instead of the default one. Robert, is there a hook for that?
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On 3 Aug 2009, at 15:34, Roland Kaminski wrote:
The code is full of such "dangerous" optimizations but most of them can be somehow handled. It is optimized to be as cache-friendly as possible that is why I would like to not remove this one. A hook during deserialization would be exactly what I need. I could also imagine other worarounds but all of them are very ugly.
Look at this part of the documentation: http://www.boost.org/doc/libs/1_39_0/libs/serialization/doc/serialization.ht... and overload load_construct_data and save_construct_data for your type Matthias
I do not see how these could be used for this case. load_construct_data provides a pre-allocated pointer that can then be initialized. And after save_construct_data is called the pointer is already registered but without the additional size information. Do I miss something? Regards, Roland On Tuesday 04 August 2009 02:38:01 Matthias Troyer wrote:
On 3 Aug 2009, at 15:34, Roland Kaminski wrote:
The code is full of such "dangerous" optimizations but most of them can be somehow handled. It is optimized to be as cache-friendly as possible that is why I would like to not remove this one. A hook during deserialization would be exactly what I need. I could also imagine other worarounds but all of them are very ugly.
Look at this part of the documentation:
http://www.boost.org/doc/libs/1_39_0/libs/serialization/doc/serialization.h tml#constructors
and overload load_construct_data and save_construct_data for your type
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
I see your point. Here is one thing you could do: you could wrap your pointer in a special class and have the serialize function of that class do the allocation, loading of the object, and registration of the pointer for you. Would that work? Matthias On 4 Aug 2009, at 03:04, Roland Kaminski wrote:
I do not see how these could be used for this case. load_construct_data provides a pre-allocated pointer that can then be initialized. And after save_construct_data is called the pointer is already registered but without the additional size information. Do I miss something?
Regards, Roland
On Tuesday 04 August 2009 02:38:01 Matthias Troyer wrote:
On 3 Aug 2009, at 15:34, Roland Kaminski wrote:
The code is full of such "dangerous" optimizations but most of them can be somehow handled. It is optimized to be as cache-friendly as possible that is why I would like to not remove this one. A hook during deserialization would be exactly what I need. I could also imagine other worarounds but all of them are very ugly.
Look at this part of the documentation:
http://www.boost.org/doc/libs/1_39_0/libs/serialization/doc/serialization.h tml#constructors
and overload load_construct_data and save_construct_data for your type
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Note that the library now supports serialization of pointers to classes which have a new operator defined. Wouldn't that do the trick? Robert Ramey Matthias Troyer wrote:
I see your point. Here is one thing you could do: you could wrap your pointer in a special class and have the serialize function of that class do the allocation, loading of the object, and registration of the pointer for you. Would that work?
Matthias
On 4 Aug 2009, at 03:04, Roland Kaminski wrote:
I do not see how these could be used for this case. load_construct_data provides a pre-allocated pointer that can then be initialized. And after save_construct_data is called the pointer is already registered but without the additional size information. Do I miss something?
Regards, Roland
On Tuesday 04 August 2009 02:38:01 Matthias Troyer wrote:
On 3 Aug 2009, at 15:34, Roland Kaminski wrote:
The code is full of such "dangerous" optimizations but most of them can be somehow handled. It is optimized to be as cache-friendly as possible that is why I would like to not remove this one. A hook during deserialization would be exactly what I need. I could also imagine other worarounds but all of them are very ugly.
Look at this part of the documentation:
http://www.boost.org/doc/libs/1_39_0/libs/serialization/doc/serialization.h tml#constructors
and overload load_construct_data and save_construct_data for your type
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
I tried that too. But I need the size in the new operator and this requires reading some global variable. It works but is not very "elegant" either. Btw: I had to define the struct has_new_operator because I needed two new operators, which led to an ambigous call in has_new_operator_impl. namespace boost { template<> struct has_new_operator<Derived> : public true_type { }; } On Thursday 06 August 2009 23:14:45 Robert Ramey wrote:
Note that the library now supports serialization of pointers to classes which have a new operator defined. Wouldn't that do the trick?
Robert Ramey
Matthias Troyer wrote:
I see your point. Here is one thing you could do: you could wrap your pointer in a special class and have the serialize function of that class do the allocation, loading of the object, and registration of the pointer for you. Would that work?
Matthias
On 4 Aug 2009, at 03:04, Roland Kaminski wrote:
I do not see how these could be used for this case. load_construct_data provides a pre-allocated pointer that can then be initialized. And after save_construct_data is called the pointer is already registered but without the additional size information. Do I miss something?
Regards, Roland
On Tuesday 04 August 2009 02:38:01 Matthias Troyer wrote:
On 3 Aug 2009, at 15:34, Roland Kaminski wrote:
The code is full of such "dangerous" optimizations but most of them can be somehow handled. It is optimized to be as cache-friendly as possible that is why I would like to not remove this one. A hook during deserialization would be exactly what I need. I could also imagine other worarounds but all of them are very ugly.
Look at this part of the documentation:
http://www.boost.org/doc/libs/1_39_0/libs/serialization/doc/serializati on.h tml#constructors
and overload load_construct_data and save_construct_data for your type
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
That is exactly what I am trying to avoid because this would force me to reinvent all the nice features that are already implemented in the library (to store pointers only once, to automatically store the right type). Regards, Roland On Thursday 06 August 2009 19:25:42 Matthias Troyer wrote:
I see your point. Here is one thing you could do: you could wrap your pointer in a special class and have the serialize function of that class do the allocation, loading of the object, and registration of the pointer for you. Would that work?
Matthias
On 4 Aug 2009, at 03:04, Roland Kaminski wrote:
I do not see how these could be used for this case. load_construct_data provides a pre-allocated pointer that can then be initialized. And after save_construct_data is called the pointer is already registered but without the additional size information. Do I miss something?
Regards, Roland
On Tuesday 04 August 2009 02:38:01 Matthias Troyer wrote:
On 3 Aug 2009, at 15:34, Roland Kaminski wrote:
The code is full of such "dangerous" optimizations but most of them can be somehow handled. It is optimized to be as cache-friendly as possible that is why I would like to not remove this one. A hook during deserialization would be exactly what I need. I could also imagine other worarounds but all of them are very ugly.
Look at this part of the documentation:
http://www.boost.org/doc/libs/1_39_0/libs/serialization/doc/serializatio n.h tml#constructors
and overload load_construct_data and save_construct_data for your type
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
We just added support for overloaded new operator for most compilers. That might be an option. The wrapper binary_option might be what you want. The of course there is the save/load construct data. Robert Ramey Matthias Troyer wrote:
On 3 Aug 2009, at 10:51, Roland Kaminski wrote:
Hi, i am planning to use the boost serialization library to implement checkpointing for an ASP/Sat solver. Unfortunately, the solver implements one (very special) optimization that makes use of a zero-sized array at the end of a class. I have found no way to store something like this using the library. There is a small program in the attachment that contains data structures similar to the ones used in the solver code. This would need a mechanism to control the allocation of memory during deserialization. Do you see any possibility do serialize something like this?
Regards and thanks, Roland
Uhhhh .... that is dangerous code. and this object is neither Assignable nor Copyable. Do you really need to make use of that optimization? In essence you need to control the memory allocation:
Base *x = new(new unsigned char[sizeof(Derived) + 42 * sizeof(int)]) Derived(42);
and have the serialization library do such a new operation instead of the default one. Robert, is there a hook for that?
Matthias
On Tuesday 04 August 2009 07:28:39 Robert Ramey wrote:
We just added support for overloaded new operator for most compilers. Could you point me to some documentation please?
The wrapper binary_object might be what you want. This might work. The object itself contains no pointers and has already some virtual methods. Adding a new virtual size method would allow for saving the raw binary object. Of course this would be a very ugly hack and everything but portable.
The of course there is the save/load construct data. I think those cannot be used.
Regards, Roland
Hi, I dived into the source code of the library and found a workaround to allocate such objects. Unfortunately, I had to specialize a class in the Detail namespace. With a hook in the pointer_iserializer and pointer_oserializer it would have been very easy to implement this. I don't know whether there are other cases where such an extension could be useful. Just wanted to let you know that i found some solution. (see attachment) Regards and thanks, Roland On Monday 03 August 2009 23:12:50 Matthias Troyer wrote:
Uhhhh .... that is dangerous code. and this object is neither Assignable nor Copyable. Do you really need to make use of that optimization? In essence you need to control the memory allocation:
Base *x = new(new unsigned char[sizeof(Derived) + 42 * sizeof(int)]) Derived(42);
and have the serialization library do such a new operation instead of the default one. Robert, is there a hook for that?
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
participants (3)
-
Matthias Troyer -
Robert Ramey -
Roland Kaminski