Using boost::serialization in real-time without allocating memory

newer
boost zlib problem on linux (works...

Peter Soetens

17 Sep 2009 17 Sep '09

1:10 p.m.

Hi, I'm trying to find out if boost::serialization can be used in real-time applications to stream data into a fifo to another process. It is mandatory that no memory allocations happen during the serialization. I tested this with a std::vector<double> of 10 elements in combination with the boost::iostreams library. The aim is that all memory allocations happen during construction of the archive object, while the serialisation itself causes none. <code> #include #include #include #include namespace io = boost::iostreams; int main(int argc, char *argv[]) { vector<double> d(10, 1.1); char sink[1000]; memset( sink, 0, 1000); io::streamio::array_sink out(sink,1000); boost::archive::binary_oarchive oa(out); //oa << d; // should not allocate return 0; } </code> The setup code does 10 memory allocations according to valgrind: ==12995== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 1) ==12995== malloc/free: in use at exit: 0 bytes in 0 blocks. ==12995== malloc/free: 10 allocs, 10 frees, 913 bytes allocated. ==12995== For counts of detected errors, rerun with: -v ==12995== All heap blocks were freed -- no leaks are possible. If we uncomment 'oa << d' we get 2 more (unwanted) allocations: ==13010== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 1) ==13010== malloc/free: in use at exit: 0 bytes in 0 blocks. ==13010== malloc/free: 12 allocs, 12 frees, 1,001 bytes allocated. ==13010== For counts of detected errors, rerun with: -v ==13010== All heap blocks were freed -- no leaks are possible I'm guessing that the 2 allocations in the serialisation path come from a temporay std::string object, when writing the 'serialization::archive' string into the archive. Wouldn't it be possible to rewrite this library code as such that there are no allocations / strings created ? A similar pattern is observed for deserializing: 2 allocs in the serialisation path. Peter PS: I tried to find out who defines the macro BOOST_ARCHIVE_SIGNATURE() but I couldn't find the definition/#define !?

Show replies by date

Stefan Strasser

17 Sep 17 Sep

12:11 p.m.

Am Thursday 17 September 2009 15:10:11 schrieb Peter Soetens:

...

Hi,

I'm trying to find out if boost::serialization can be used in real-time applications to stream data into a fifo to another process. It is mandatory that no memory allocations happen during the serialization. I tested this with a std::vector<double> of 10 elements in combination with the boost::iostreams library.

...

I'm guessing that the 2 allocations in the serialisation path come from a temporay std::string object, when writing the 'serialization::archive' string into the archive.

you can supress that by passing no_header to the archive. however, that won't solve the problem. the archives internally use STL containers for type registration and object tracking (and maybe other things, I don't know all the details). so even if you can avoid those 2 allocations for a vector<double>, there is no way to avoid allocations in the general case, for any type, until boost.serialization accepts a custom allocator, or a traits class that handles the registration stuff. that code is statically linked right now so if you wanted to implement that you'd also have to refactor boost.serialization. also note that one archive can only be used for one serialization process - I'm guessing one object in your case. if you serialize 2 objects into 1 archive they can only be read in that order from the stream. so you'd have to take the allocations of archive construction into account, too. the only simple way I see right now is using an own archive that doesn't derive from boost.serialization's common_?archive. but that comes close to an implementing an new serialization system.

Peter Soetens

2:33 p.m.

On Thu, Sep 17, 2009 at 14:11, Stefan Strasser wrote:

...

Am Thursday 17 September 2009 15:10:11 schrieb Peter Soetens:

...
Hi,

I'm trying to find out if boost::serialization can be used in real-time applications to stream data into a fifo to another process. It is mandatory that no memory allocations happen during the serialization. I tested this with a std::vector<double> of 10 elements in combination with the boost::iostreams library.

...
I'm guessing that the 2 allocations in the serialisation path come from a temporay std::string object, when writing the 'serialization::archive' string into the archive.

you can supress that by passing no_header to the archive. however, that won't solve the problem. the archives internally use STL containers for type registration and object tracking (and maybe other things, I don't know all the details). so even if you can avoid those 2 allocations for a vector<double>, there is no way to avoid allocations in the general case, for any type, until boost.serialization accepts a custom allocator, or a traits class that handles the registration stuff. that code is statically linked right now so if you wanted to implement that you'd also have to refactor boost.serialization.

also note that one archive can only be used for one serialization process - I'm guessing one object in your case. if you serialize 2 objects into 1 archive they can only be read in that order from the stream. so you'd have to take the allocations of archive construction into account, too.

the only simple way I see right now is using an own archive that doesn't derive from boost.serialization's common_?archive. but that comes close to an implementing an new serialization system.

Thanks for the detailed answer. I missed the point that an archive could serialize only once. This is indeed incompatible with my design requirements. By design, our serialisation requirements are PODs and std::vector<POD> since we only exchange 'pure' data structures (and use std::vector in a limited way, almost similar to array<POD>). Much of the features in boost::serialization aren't required for us. I might do what you suggest or look for another serialization solution. Peter

Stefan Strasser

12:57 p.m.

Am Thursday 17 September 2009 16:33:16 schrieben Sie: ting an new serialization system.

...

Thanks for the detailed answer. I missed the point that an archive could serialize only once. This is indeed incompatible with my design requirements. By design, our serialisation requirements are PODs and std::vector<POD> since we only exchange 'pure' data structures (and use std::vector in a limited way, almost similar to array<POD>). Much of the features in boost::serialization aren't required for us.

PODs and POD-containers is pretty simple. just pass a type that is compatible with boost.serialization's archive types to the container's serialize() function. here's part of the archive type I use for types that support this kind of serialization. it will fail statically if the container tries to serialize anything non-POD. class serializer{ public: explicit serializer(... void save_binary(void const *data,std::size_t size){ ... } template<class T> serializer &operator<<(T const &t){ BOOST_STATIC_ASSERT(serialization::implementation_level<T>::value == serialization::primitive_type); ... return *this; } ... template<class T> serializer &operator&(T const &t){ return this->operator<<(t); } private: ... };

Robert Ramey

4:33 p.m.

New subject: Using boost::serialization in real-time withoutallocating memory

Peter Soetens wrote:

...

I might do what you suggest or look for another serialization solution.

Given the limited scope of what you want to do, it wouldn't be THAT hard to make your own archive. Take a look at "trivial_archive" in the documentation and build from that.

...

Peter

Peter Soetens

18 Sep 18 Sep

12:30 p.m.

New subject: Using boost::serialization in real-time withoutallocating memory

On Thu, Sep 17, 2009 at 18:33, Robert Ramey wrote:

...

Peter Soetens wrote:

...
I might do what you suggest or look for another serialization solution.

Given the limited scope of what you want to do, it wouldn't be THAT hard to make your own archive. Take a look at "trivial_archive" in the documentation and build from that.

Encouraged by your and Stefan's suggestions, I've spent another day at looking what the possibilities are. 1. I first created my own binary_?archive classes which inherited from common_?archive. This resulted in pulling in the library with the 'behind the scenes type tracking'. It seems, inheriting from common is not an option for me if I want 'zero-allocation' serialization. Correct ? (I also tested the demo_fast_archive.cpp extended with no_header, but that one also allocated). 2. I then tried to work further on Stefan's code snippet. It works, but indeed only for 'primitive_type's. I *think* I need functionality upto the 'object_serializable' if I want std::vector<T> as well. Since Stefan's operator& consumes every T, I'll need one operator& for each primitive type, and then an operator& that tries to save/load using free functions of the not primitive type. 3. It looks that I'll have to implement the Archive concept from scratch or at least based on Stefan's code and the load_binary/save_binary code from binary_?archive. Am I on the right track ? Peter

Peter Soetens

3:18 p.m.

New subject: Using boost::serialization in real-time withoutallocating memory

On Fri, Sep 18, 2009 at 14:30, Peter Soetens wrote:

...

On Thu, Sep 17, 2009 at 18:33, Robert Ramey wrote:

...
Peter Soetens wrote:

...
I might do what you suggest or look for another serialization solution.

Given the limited scope of what you want to do, it wouldn't be THAT hard to make your own archive. Take a look at "trivial_archive" in the documentation and build from that.

Encouraged by your and Stefan's suggestions, I've spent another day at looking what the possibilities are.

1. I first created my own binary_?archive classes which inherited from common_?archive. This resulted in pulling in the library with the 'behind the scenes type tracking'. It seems, inheriting from common is not an option for me if I want 'zero-allocation' serialization. Correct ? (I also tested the demo_fast_archive.cpp extended with no_header, but that one also allocated).

2. I then tried to work further on Stefan's code snippet. It works, but indeed only for 'primitive_type's. I *think* I need functionality upto the 'object_serializable' if I want std::vector<T> as well. Since Stefan's operator& consumes every T, I'll need one operator& for each primitive type, and then an operator& that tries to save/load using free functions of the not primitive type.

3. It looks that I'll have to implement the Archive concept from scratch or at least based on Stefan's code and the load_binary/save_binary code from binary_?archive.

I'm got so far that I 'hacked' an allocation free output and output archive, and output works like a charm. But for the input side, I can't seem to get the ADL to work for serializing the nvt<T> case. The point I'm struggling with is that sometimes save() is a member function and sometimes it's a free function, and I can't get to the selection logic. The archiver in attachment fails to compile on line 68 where it says 'no matching function for call to ‘load(boost::archive::mqueue_tiny_iarchive&, const boost::serialization::nvp<double>&, int)', when reading back in an std::vector<double>. I'm aware that this is no longer an serialization question, but I'm just one compile error away from the real solution :-) Maybe it wouldn't hurt to include some examples for serializing level 1/2 objects which just implement the Archive concept interface. Any hints greatly appreciated. Peter

Robert Ramey

6 p.m.

New subject: Using boost::serialization in real-timewithoutallocating memory

Peter Soetens wrote:

...

I'm got so far that I 'hacked' an allocation free output and output archive, and output works like a charm.

hmmm - you're making it look easy.

...

But for the input side, I can't seem to get the ADL to work for serializing the nvt<T> case.

The point I'm struggling with is that sometimes save() is a member function and sometimes it's a free function, and I can't get to the selection logic. The archiver in attachment fails to compile on line 68 where it says 'no matching function for call to load(boost::archive::mqueue_tiny_iarchive&, const boost::serialization::nvp<double>&, int)', when reading back in an std::vector<double>.

Double check the namespaces in which you've declared and implemented your save/load functions. It's very easy to get that wrong. Also double check your arguments. Note that the serialization library uses an obscure technique to implement the equivalent of partial function template ordering. This could also be confusing.

...

I'm aware that this is no longer an serialization question, but I'm just one compile error away from the real solution :-) Maybe it wouldn't hurt to include some examples for serializing level 1/2 objects which just implement the Archive concept interface.

Feel free to submit your example as a "case study" to the serialization library documentation. Robert Ramey

Stefan Strasser

6:08 p.m.

New subject: Using boost::serialization in real-time withoutallocating memory

Am Friday 18 September 2009 17:18:56 schrieb Peter Soetens:

...

On Fri, Sep 18, 2009 at 14:30, Peter Soetens wrote:

...
On Thu, Sep 17, 2009 at 18:33, Robert Ramey wrote:

...
Peter Soetens wrote:

...
I might do what you suggest or look for another serialization solution.

Given the limited scope of what you want to do, it wouldn't be THAT hard to make your own archive. Take a look at "trivial_archive" in the documentation and build from that.

Encouraged by your and Stefan's suggestions, I've spent another day at looking what the possibilities are.

1. I first created my own binary_?archive classes which inherited from common_?archive. This resulted in pulling in the library with the 'behind the scenes type tracking'. It seems, inheriting from common is not an option for me if I want 'zero-allocation' serialization. Correct ? (I also tested the demo_fast_archive.cpp extended with no_header, but that one also allocated).

2. I then tried to work further on Stefan's code snippet. It works, but indeed only for 'primitive_type's. I *think* I need functionality upto the 'object_serializable' if I want std::vector<T> as well. Since Stefan's operator& consumes every T, I'll need one operator& for each primitive type, and then an operator& that tries to save/load using free functions of the not primitive type.

3. It looks that I'll have to implement the Archive concept from scratch or at least based on Stefan's code and the load_binary/save_binary code from binary_?archive.

I'm got so far that I 'hacked' an allocation free output and output archive, and output works like a charm. But for the input side, I can't seem to get the ADL to work for serializing the nvt<T> case.

line 68: boost::serialization::load(*this, t, version); don't you mean archive::load? I'm not aware of a serialization::load function. however, this is not an ADL lookup, as it has a namespace specifier. something like this should do the trick: namespace boost{ namespace archive{ void tinyarchive_load_adl(...){ load(...); } } } boost::archive::tinyarchive_load_adl(...); I'm not sure if the call is required to originate within the archive namespace in this case (I think it does, because none of the arguments is in that namespace) but in any way, an ADL call needs to be without a namespace specifier.

Peter Soetens

8:22 p.m.

New subject: Using boost::serialization in real-time withoutallocating memory

On Fri, Sep 18, 2009 at 20:08, Stefan Strasser wrote:

...

Am Friday 18 September 2009 17:18:56 schrieb Peter Soetens:

...
I'm got so far that I 'hacked' an allocation free output and output archive, and output works like a charm. But for the input side, I can't seem to get the ADL to work for serializing the nvt<T> case.

line 68: boost::serialization::load(*this, t, version);

don't you mean archive::load? I'm not aware of a serialization::load function.

That's it ! I used the wrong load/save functions. I cleaned up the archives and documented a bit. The only obscurity is that both classes required a 'save/load_object()' function, while this is not documented as such in the Archive Concept pages. Is this a recent addition (I'm using Boost 1.37.0 btw) ? What about save/load_pointer etc ? I left most/all of these exotic functions empty. I wonder in which situations they will be called ? Should I put an assert(false) in them ? I renamed the class to binary_object_archive.hpp This file was exactly what I needed :-) Peter

Robert Ramey

9:44 p.m.

New subject: Using boost::serialization in real-timewithoutallocating memory

Peter Soetens wrote:

...

...
On Fri, Sep 18, 2009 at 20:08, Stefan Strasser wrote:

...
Am Friday 18 September 2009 17:18:56 schrieb Peter Soetens:

...
I'm got so far that I 'hacked' an allocation free output and output archive, and output works like a charm. But for the input side, I can't seem to get the ADL to work for serializing the nvt<T> case.

line 68: boost::serialization::load(*this, t, version);

don't you mean archive::load? I'm not aware of a serialization::load function.

That's it ! I used the wrong load/save functions. I cleaned up the archives and documented a bit. The only obscurity is that both classes required a 'save/load_object()' function, while this is not documented as such in the Archive Concept pages. Is this a recent addition (I'm using Boost 1.37.0 btw) ? What about save/load_pointer etc ?

I left most/all of these exotic functions empty. I wonder in which situations they will be called ? Should I put an assert(false) in them ?

I renamed the class to binary_object_archive.hpp

This file was exactly what I needed :-)

Peter

...
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Robert Ramey

9:56 p.m.

New subject: Using boost::serialization in real-timewithoutallocating memory

Peter Soetens wrote:

...

...
That's it ! I used the wrong load/save functions. I cleaned up the archives and documented a bit. The only obscurity is that both classes required a 'save/load_object()' function, while this is not documented as such in the Archive Concept pages. Is this a recent addition (I'm using Boost 1.37.0 btw)

This would be due to the fact that even the "trivial_archive" inherits functionality from common_?archive. The archive concept doesn't require this - but most archives will want to do this - then override the implementation of common_?archive.

...

...
? What about save/load_pointer etc ?

truth is I don't remember all of this stuff. But it would seem that the serialization of pointers is a lower level. This makes sense since there is more than one idea about how to save/restore a pointer. The current implementation creates a new object when a pointer is de-serialized. But it didn't have to be this way. Someone else might want to define it as to just restore the value pointed to. Someone else might want to prohibit it. et.

...

...
I left most/all of these exotic functions empty. I wonder in which situations they will be called ? Should I put an assert(false) in them ?

If it were me, I would leave them undeclared.

...

...
I renamed the class to binary_object_archive.hpp

and I would have chose a different name that reflects the more specialized nature of this implementation. Also binary_object is used in the library to mean something specifice. Also the archives in the the library are templated on stream type binary/text - so binary isn't such a great name either. maybe something like (binary | text) _static_(i/o)archive static sort of suggests that there is no dynamic memory allocation.

...

...
This file was exactly what I needed :-)

Peter

...
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Peter Soetens

21 Sep 21 Sep

10:10 a.m.

New subject: Using boost::serialization in real-timewithoutallocating memory

On Fri, Sep 18, 2009 at 23:56, Robert Ramey wrote:

...

Peter Soetens wrote:

...
...
That's it ! I used the wrong load/save functions. I cleaned up the archives and documented a bit. The only obscurity is that both classes required a 'save/load_object()' function, while this is not documented as such in the Archive Concept pages. Is this a recent addition (I'm using Boost 1.37.0 btw)

This would be due to the fact that even the "trivial_archive" inherits functionality from common_?archive. The archive concept doesn't require this - but most archives will want to do this - then override the implementation of common_?archive.

That's not my point. As I understand the 'Loading/Storing Archive Concept', any archive that implements the listed functions should be able to process 'ar << data;' if the serialization handler for data is defined. My archive implemented the concept fully, but did not compile because of a missing function. The only conclusion I can make is that the concept is not complete, or violated because people only tested it with the common base class and started to assume some functions were there while they are not required to be.

...

...
...
? What about save/load_pointer etc ?

truth is I don't remember all of this stuff. But it would seem that the serialization of pointers is a lower level. This makes sense since there is more than one idea about how to save/restore a pointer. The current implementation creates a new object when a pointer is de-serialized. But it didn't have to be this way. Someone else might want to define it as to just restore the value pointed to. Someone else might want to prohibit it. et.

...
...
I left most/all of these exotic functions empty. I wonder in which situations they will be called ? Should I put an assert(false) in them ?

If it were me, I would leave them undeclared.

Such that the user gets a compile time warning when he want's to use these features ?

...

...
...
I renamed the class to binary_object_archive.hpp

and I would have chose a different name that reflects the more specialized nature of this implementation. Also binary_object is used in the library to mean something specifice. Also the archives in the the library are templated on stream type binary/text - so binary isn't such a great name either. maybe something like

(binary | text) _static_(i/o)archive

static sort of suggests that there is no dynamic memory allocation.

I'm merely suggesting it to add to the examples directory, so templating on a stream type did not seem necessary (other examples take shortcuts as well). I would agree with many more shortcomings if this was to be added to the library itself. IMHO 'static' is an effect of the level 2-only + binary serialization of this archive. I wanted to stress in the name that 'class'/polymorphic types can not be serialized with this archive (I'd need to add the boost static assert, but I don't know if std::vector would still work then). I was suggesting before that we could make examples for each serialization level. Then we could also name them 'binary_level1_?archive', 'binary_level2_?archive', 'binary_level3_?archive'. YMMV. Peter

Robert Ramey

5:10 p.m.

New subject: Using boost::serialization inreal-timewithoutallocating memory

Peter Soetens wrote:

...

On Fri, Sep 18, 2009 at 23:56, Robert Ramey wrote:

...
Peter Soetens wrote:

...
...
That's it ! I used the wrong load/save functions. I cleaned up the archives and documented a bit. The only obscurity is that both classes required a 'save/load_object()' function, while this is not documented as such in the Archive Concept pages. Is this a recent addition (I'm using Boost 1.37.0 btw)

This would be due to the fact that even the "trivial_archive" inherits functionality from common_?archive. The archive concept doesn't require this - but most archives will want to do this - then override the implementation of common_?archive.

That's not my point. As I understand the 'Loading/Storing Archive Concept', any archive that implements the listed functions should be able to process 'ar << data;' if the serialization handler for data is defined. My archive implemented the concept fully, but did not compile because of a missing function. The only conclusion I can make is that the concept is not complete, or violated because people only tested it with the common base class and started to assume some functions were there while they are not required to be.

I would have to look into this in more detail to really provide an answer. Clearly, the fact that that "trivial_archive" inherits from common_archive would suggest you're correct about this or maybe that something else is out of whack. I'll look into it when I have nothing else to do. Given you've already wrapped your head around this whole issue - maybe you want to suggest how something might be changed. My first approach would be to figure out why save/load object is required to compile. Perhaps this is just an implementation artifact. Ideally the implementation could be slightly refactored to support the original concept.

...

...
...
...
? What about save/load_pointer etc ?

truth is I don't remember all of this stuff. But it would seem that the serialization of pointers is a lower level. This makes sense since there is more than one idea about how to save/restore a pointer. The current implementation creates a new object when a pointer is de-serialized. But it didn't have to be this way. Someone else might want to define it as to just restore the value pointed to. Someone else might want to prohibit it. et.

...
...
I left most/all of these exotic functions empty. I wonder in which situations they will be called ? Should I put an assert(false) in them ?

If it were me, I would leave them undeclared.

Such that the user gets a compile time warning when he want's to use these features ?

That would seem correct to me. There is nothing in the archive concept which suggests that all potential operate must be supported in some specific way. That is, it's not a bug if some archive doesn't implement some function like serializing pointers in some specific way. Of course it turns out that they have been implemented in the archives provided in the way that to me seemed most useful. And judging from way the library seems to be used - others argree with me. However, I don't/didn't want to close off other implementations such as yours. I've strived mightely to keep archive and serialization concepts orthogonal. I believe that I have been mostly successful.

...

...
...
...
I renamed the class to binary_object_archive.hpp

and I would have chose a different name that reflects the more specialized nature of this implementation. Also binary_object is used in the library to mean something specifice. Also the archives in the the library are templated on stream type binary/text - so binary isn't such a great name either. maybe something like

(binary | text) _static_(i/o)archive

static sort of suggests that there is no dynamic memory allocation.

I sort of assumed that you wanted to avoid that. Truth is I didn't go into detail about your requirements, I just pointed you in the direction which I thought would do the most good. I am gratified that you've been able to accomplish something that I never forsaw while including just parts of the library which you needed. This makes me thing that I got it mostly right.

...

I'm merely suggesting it to add to the examples directory, so templating on a stream type did not seem necessary (other examples take shortcuts as well). I would agree with many more shortcomings if this was to be added to the library itself.

I would love to include such an example (case study). However I have a few observations based on years of practicle experience in helping users with the library. * Focused - it should address one - and only one - specific problem * Simple - easy to follow * Short - not require a lot of time to follow through * Useful - should provide a really useful function rather than a "toy problem". This provides more motivation for users to look at it detail and will likely be the basis. * Should be templated where appropriate. See "Useful" above. But there's a larger point. Presenting it as a template fits in with the other boost libraries. Also - template parameters tend to show the separation of the "algorithm" from the "arguments" (template parameters) which actually helps make things more clear. Just one man's opinion. * When the example is added to the package - it is added to the test suite so that it can be tweaked for all the compilers run in the test trunk. * Documentation - a nice page describing the motiviation for the example etc. If an example doesn't do the above, it creates a lot more user queries than it solves. It is my intention that any addition to the library result in: * no loss in functionality * no increase in requirements for user support * no new bugs * improvement in at least one of the above. That is, any addition to the library and/or it's documentation should result in a strict improvement. I realise that it makes it look that adding a simple example is a lot more work than it first appears. That is in fact the case. Having said all that - I still would like to see something like this included. Other examples demo_fast_archive and demo_portable_binary have result in full blown archives of thier own.

...

IMHO 'static' is an effect of the level 2-only + binary serialization of this archive. I wanted to stress in the name that 'class'/polymorphic types can not be serialized with this archive (I'd need to add the boost static assert, but I don't know if std::vector would still work then).

I was suggesting before that we could make examples for each serialization level. Then we could also name them 'binary_level1_?archive', 'binary_level2_?archive', 'binary_level3_?archive'.

I havn't look into it enough to really understand this. It goes to show that a simple idea can't help evolving to something non-trivial as one examines it more closely. Robert Ramey

Stefan Strasser

5:57 p.m.

New subject: Using boost::serialization inreal-timewithoutallocating memory

Am Monday 21 September 2009 19:10:15 schrieb Robert Ramey:

...

Peter Soetens wrote:

...
On Fri, Sep 18, 2009 at 23:56, Robert Ramey wrote:

...
Peter Soetens wrote:

...
...
That's it ! I used the wrong load/save functions. I cleaned up the archives and documented a bit. The only obscurity is that both classes required a 'save/load_object()' function, while this is not documented as such in the Archive Concept pages. Is this a recent addition (I'm using Boost 1.37.0 btw)

This would be due to the fact that even the "trivial_archive" inherits functionality from common_?archive. The archive concept doesn't require this - but most archives will want to do this - then override the implementation of common_?archive.

That's not my point. As I understand the 'Loading/Storing Archive Concept', any archive that implements the listed functions should be able to process 'ar << data;' if the serialization handler for data is defined. My archive implemented the concept fully, but did not compile because of a missing function. The only conclusion I can make is that the concept is not complete, or violated because people only tested it with the common base class and started to assume some functions were there while they are not required to be.

I would have to look into this in more detail to really provide an answer. Clearly, the fact that that "trivial_archive" inherits from common_archive would suggest you're correct about this or maybe that something else is out of whack. I'll look into it when I have nothing else to do. Given you've already wrapped your head around this whole issue - maybe you want to suggest how something might be changed. My first approach would be to figure out why save/load object is required to compile. Perhaps this is just an implementation artifact. Ideally the implementation could be slightly refactored to support the original concept.

I think to support this an other use cases like it common_archive would have to be moved outside of the linked library into a header and be made more generic. right now it's full-blown type-registering/object-tracking, object-graph with class-hierarchy serialization, or nothing - write your own archive from scratch. I've also had my problems with that. although I didn't have to avoid allocation completely, the construction of a common_archive took too long and you couldn't configure away the parts you don't need. such a generalization could be based e.g. on what is described as "levels of sophistication" of serialization, here: http://www.parashift.com/c%2B%2B-faq-lite/serialization.html#faq-36.2

Robert Ramey

6:27 p.m.

New subject: Using boost::serializationinreal-timewithoutallocating memory

Stefan Strasser wrote:

...

I think to support this an other use cases like it common_archive would have to be moved outside of the linked library into a header and be made more generic.

common_archive is a template - it's not in the linked library.

...

right now it's full-blown type-registering/object-tracking, object-graph with class-hierarchy serialization, or nothing - write your own archive from scratch.

Hmmm - seems to me that that this thread shows the opposite. Also note, that since the "higher level" functionality is mostly implemented in templates, you (mostly) don't generate code for those functions that your archive class doesn't use.

...

I've also had my problems with that. although I didn't have to avoid allocation completely, the construction of a common_archive took too long and you couldn't configure away the parts you don't need.

I would like to see improvements in this area. Mostly this is because I would like to see the library more "formal" and hence with less bugs, more robust, and more extensible. It's not been a huge priority since most users find the the archive classes included in the package seem addequate to their needs.

...

such a generalization could be based e.g. on what is described as "levels of sophistication" of serialization, here: http://www.parashift.com/c%2B%2B-faq-lite/serialization.html#faq-36.2

I looked at this and found it to be very unhelpful and misleading two those who want to actually implement serialization. The items really address the tradeoffs in implementing a serialization system. I don't think that that these items consider the power of C++ combined with things like other boost libraries. Anyone following this advice would endup writing new serialization code for every application - exactly the wrong direction to go. It also also lacks and notion of good design - factoring data types from algorithms (text vs binary discussion). Any user following this advice would without at least taking a hard look at the boost serialization library would be wasting tons of his employers money at a minimum. It might even be considered by some to be a breach of professional ethics - unless he didn't know better in which case I suppose it would be excusable. Robert Ramey

Stefan Strasser

7:24 p.m.

New subject: Using boost::serializationinreal-timewithoutallocating memory

Am Monday 21 September 2009 20:27:59 schrieb Robert Ramey:

...

Stefan Strasser wrote:

...
I think to support this an other use cases like it common_archive would have to be moved outside of the linked library into a header and be made more generic.

common_archive is a template - it's not in the linked library.

ok, I must have mixed up some types. the point was that the ultimate base class, that handles the type registration and object tracking etc., is linked .cpp and not very configurable. the optimum case imho would be that you could pass a traits class to that archive base class, that gets called whenever there is an object or type to be registered and is queried for existing objects and types later on. that way you could disable registration, save type registration outside of the archive itself, let type registration span multiple archives etc, make sure type registration doesn't allocate etc.

...

I would like to see improvements in this area. Mostly this is because I would like to see the library more "formal" and hence with less bugs, more robust, and more extensible. It's not been a huge priority since most users find the the archive classes included in the package seem addequate to their needs.

sure, it's the best serialization solution I know and the template technique trumps any reflection based solution (even if c++ had reflection). this is not meant as a critique.

...

...
such a generalization could be based e.g. on what is described as "levels of sophistication" of serialization, here: http://www.parashift.com/c%2B%2B-faq-lite/serialization.html#faq-36.2

I looked at this and found it to be very unhelpful and misleading two those who want to actually implement serialization. The items

I didn't mean to suggest that FAQ gave good advice for serialization or a boost serialization system should be modelled after this. I just wanted to highlight what I meant by "configure away the parts you don't need", and the FAQ answer I linked includes a list of techniques that don't support serialization of object graphs with class hierarchy, but still may be sufficient for many cases.

...

really address the tradeoffs in implementing a serialization system. I don't think that that these items consider the power of C++ combined with things like other boost libraries. Anyone following this advice would endup writing new serialization code for every application - exactly the wrong direction to go. It also also lacks and notion of good design - factoring data types from algorithms (text vs binary discussion). Any user following this advice would without at least taking a hard look at the boost serialization library would be wasting tons of his employers money at a minimum. It might even be considered by some to be a breach of professional ethics - unless he didn't know better in which case I suppose it would be excusable.

I think that FAQ is pretty old, at least part of it. it may have been good advice at the time, if you had to build serialization from scratch.

Robert Ramey

7:25 p.m.

New subject: [Boost-users] Usingboost::serializationinreal-timewithoutallocating memory

Stefan Strasser wrote:

...

Am Monday 21 September 2009 20:27:59 schrieb Robert Ramey:

...
Stefan Strasser wrote:

...
I think to support this an other use cases like it common_archive would have to be moved outside of the linked library into a header and be made more generic.

common_archive is a template - it's not in the linked library.

ok, I must have mixed up some types. the point was that the ultimate base class, that handles the type registration and object tracking etc., is linked .cpp and not very configurable. the optimum case imho would be that you could pass a traits class to that archive base class, that gets called whenever there is an object or type to be registered and is queried for existing objects and types later on. that way you could disable registration, save type registration outside of the archive itself, let type registration span multiple archives etc, make sure type registration doesn't allocate etc.

as I alluded to before - what I would like to know is why I derived trivial_archive from common_archive. I don't remember what made me do this. It seems that if this isn't done then save/load_object is necessary. So the question is - why is this? It is an oversight somewhere? or does the archive concept have to be changed. Personally, I generally don't like the "traits class" as an argument. It sort of implies that that the template has a lot of "if -" code which navigates the traits. A "policy class" is better but can still make things complex. Personally I prefer composition of funtionality through inheritance. This can be seen in the class diagram for the serialization library. And your example makes me more convinced than ever that this is basically right. Now the only loose end it investigate why save/load object is required. Robert Ramey

Stefan Strasser

8:26 p.m.

New subject: [Boost-users] Usingboost::serializationinreal-timewithoutallocating memory

Am Monday 21 September 2009 21:25:28 schrieb Robert Ramey:

...

as I alluded to before - what I would like to know is why I derived trivial_archive from common_archive. I don't remember what made me do this. It seems that if this isn't done then save/load_object is necessary. So the question is - why is this? It is an oversight somewhere? or does the archive concept have to be changed.

common_?archive derives from basic_?archive, which implements the save_object/save_pointer functions: type registration and object tracking. so if you need to serialize graphs (as opposed to trees) you have to derive from basic_?archive, unless you want to implement your own registration system.

...

Personally, I generally don't like the "traits class" as an argument. It sort of implies that that the template has a lot of "if -" code which navigates the traits. A "policy class" is better but can still make things complex. Personally I prefer composition of funtionality through inheritance.

I'm not sure if we talk about the same thing when we say "traits class". it does accomplish composition of functionality, just not through inheritance. see http://www.boost.org/doc/libs/1_38_0/doc/html/intrusive/node_algorithms.html... for an example from boost.intrusive. the traits class isn't a configuration class, requiring if-then-else code in the archive class, but provides actual functionality - type registration and object tracking. so it would have functions like register_object, get_object, etc. the default argument to the traits template parameter would implement the current type registration and object tracknig system, using the STL maps that have caused trouble for some of us. maybe that could also be inserted somewhere in the inheritance tree and a traits class be avoided if you prefer that. the goal in any case is to be able to store that object tracking/type registration information somewhere other than STL containers that are hardcoded in the archive base class, without having to reimplement serialization logic.

Peter Soetens

9:44 p.m.

New subject: [Boost-users] Usingboost::serializationinreal-timewithoutallocating memory

On Mon, Sep 21, 2009 at 21:25, Robert Ramey wrote:

...

Stefan Strasser wrote:

...
Am Monday 21 September 2009 20:27:59 schrieb Robert Ramey:

...
Stefan Strasser wrote:

...
I think to support this an other use cases like it common_archive would have to be moved outside of the linked library into a header and be made more generic.

common_archive is a template - it's not in the linked library.

ok, I must have mixed up some types. the point was that the ultimate base class, that handles the type registration and object tracking etc., is linked .cpp and not very configurable. the optimum case imho would be that you could pass a traits class to that archive base class, that gets called whenever there is an object or type to be registered and is queried for existing objects and types later on. that way you could disable registration, save type registration outside of the archive itself, let type registration span multiple archives etc, make sure type registration doesn't allocate etc.

as I alluded to before - what I would like to know is why I derived trivial_archive from common_archive. I don't remember what made me do this. It seems that if this isn't done then save/load_object is necessary. So the question is - why is this? It is an oversight somewhere? or does the archive concept have to be changed.

Personally, I generally don't like the "traits class" as an argument. It sort of implies that that the template has a lot of "if -" code which navigates the traits. A "policy class" is better but can still make things complex. Personally I prefer composition of funtionality through inheritance. This can be seen in the class diagram for the serialization library. And your example makes me more convinced than ever that this is basically right. Now the only loose end it investigate why save/load object is required.

I found a lead to that. When the archive calls archive::save, a chain of 'invoke' calls is done to select the right serialization method for type T. When I pass std::vector<double> as T, it encounters a branch in oserializer.hpp:245, in struct 'save_conditional::invoke' . <code> // adds class information to the archive. This includes // serialization level and class version struct save_conditional { static void invoke(Archive &ar, const T &t){ //if(0 == (ar.get_flags() & no_tracking)) save_standard::invoke(ar, t); //else // save_only::invoke(ar, t); } }; </code> The alternative branch is commented out, but let that be the path that needs to be taken by my code. Current code calls save_standard, which calls directly ' ar.save_object'. save_object in turn depends on basic_oarchive to do its job. If save_only was taken, the 'non-basic_oarchive' serialization code path would have been taken. For some reason (I've spent an hour staring at it), my unit test did not discover this flaw in my code and does as if serialisation of the std::vector was fine. I'm overlooking something stupid clearly... Anyway, as it seems now, the code path taken by archive::save/load mandates derivation of basic_?archive. Maybe I should have called another 'selection' function ? like archive::save_only directly ? Peter

Peter Soetens

10:11 p.m.

New subject: [Boost-users] Usingboost::serializationinreal-timewithoutallocating memory

On Mon, Sep 21, 2009 at 23:44, Peter Soetens wrote:

...

On Mon, Sep 21, 2009 at 21:25, Robert Ramey wrote:

...
Stefan Strasser wrote:

...
Am Monday 21 September 2009 20:27:59 schrieb Robert Ramey:

...
Stefan Strasser wrote:

...
I think to support this an other use cases like it common_archive would have to be moved outside of the linked library into a header and be made more generic.

common_archive is a template - it's not in the linked library.

ok, I must have mixed up some types. the point was that the ultimate base class, that handles the type registration and object tracking etc., is linked .cpp and not very configurable. the optimum case imho would be that you could pass a traits class to that archive base class, that gets called whenever there is an object or type to be registered and is queried for existing objects and types later on. that way you could disable registration, save type registration outside of the archive itself, let type registration span multiple archives etc, make sure type registration doesn't allocate etc.

as I alluded to before - what I would like to know is why I derived trivial_archive from common_archive. I don't remember what made me do this. It seems that if this isn't done then save/load_object is necessary. So the question is - why is this? It is an oversight somewhere? or does the archive concept have to be changed.

Personally, I generally don't like the "traits class" as an argument. It sort of implies that that the template has a lot of "if -" code which navigates the traits. A "policy class" is better but can still make things complex. Personally I prefer composition of funtionality through inheritance. This can be seen in the class diagram for the serialization library. And your example makes me more convinced than ever that this is basically right. Now the only loose end it investigate why save/load object is required.

I found a lead to that. When the archive calls archive::save, a chain of 'invoke' calls is done to select the right serialization method for type T. When I pass std::vector<double> as T, it encounters a branch in oserializer.hpp:245, in struct 'save_conditional::invoke' .

<code> // adds class information to the archive. This includes // serialization level and class version struct save_conditional { static void invoke(Archive &ar, const T &t){ //if(0 == (ar.get_flags() & no_tracking)) save_standard::invoke(ar, t); //else // save_only::invoke(ar, t); } }; </code>

The alternative branch is commented out, but let that be the path that needs to be taken by my code. Current code calls save_standard, which calls directly ' ar.save_object'. save_object in turn depends on basic_oarchive to do its job. If save_only was taken, the 'non-basic_oarchive' serialization code path would have been taken.

For some reason (I've spent an hour staring at it), my unit test did not discover this flaw in my code and does as if serialisation of the std::vector was fine. I'm overlooking something stupid clearly...

Anyway, as it seems now, the code path taken by archive::save/load mandates derivation of basic_?archive. Maybe I should have called another 'selection' function ? like archive::save_only directly ?

I tried using this instead of archive::save() : <code> boost::archive::detail::load_non_pointer_type::load_only::invoke(*this,t); </code> But the compiler (gcc 4.3.3-5ubuntu4) refuses to compile this when the nvt types are in use. Basically, this is not fine: <code> int count; ar >> BOOST_SERIALIZATION_NVP(count); </code> While this is: <code> int count; nvp<int> n = BOOST_SERIALIZATION_NVP(count); ar >> n </code> You can try this out in plain user code. Beats me. The errors I get for the first case are in the line of: usr/include/boost/serialization/access.hpp: In static member function ‘static void boost::serialization::access::serialize(Archive&, T&, unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T = const boost::serialization::nvp<int>]’: /usr/include/boost/serialization/serialization.hpp:74: instantiated from ‘void boost::serialization::serialize(Archive&, T&, unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T = const boost::serialization::nvp<int>]’ /usr/include/boost/serialization/serialization.hpp:133: instantiated from ‘void boost::serialization::serialize_adl(Archive&, T&, unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T = const boost::serialization::nvp<int>]’ /usr/include/boost/archive/detail/iserializer.hpp:316: instantiated from ‘static void boost::archive::detail::load_non_pointer_type::load_only::invoke(Archive&, T&) [with Archive = RTT::marsh::binary_data_iarchive, T = const boost::serialization::nvp<int>]’ /home/kaltan/src/git/orocos-rtt/src/marsh/binary_data_archive.hpp:198: instantiated from ‘RTT::marsh::binary_data_iarchive& RTT::marsh::binary_data_iarchive::load_a_type(T&, mpl_::false_) [with T = const boost::serialization::nvp<int>]’ /home/kaltan/src/git/orocos-rtt/src/marsh/binary_data_archive.hpp:131: instantiated from ‘RTT::marsh::binary_data_iarchive& RTT::marsh::binary_data_iarchive::operator>>(T&) [with T = const boost::serialization::nvp<int>]’ /home/kaltan/src/git/orocos-rtt/tests/mqueue_archive_test.cpp:71: instantiated from here /usr/include/boost/serialization/access.hpp:109: error: passing ‘const boost::serialization::nvp<int>’ as ‘this’ argument of ‘void boost::serialization::nvp<T>::serialize(Archive&, unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T = int]’ discards qualifiers Passing to operator>> the nvt struct as a return value of make_nvt() makes gcc add a const qualifier, which gets propagated all the way...I have no idea why the other archive implementations don't suffer this. Peter

Peter Soetens

22 Sep 22 Sep

10:02 a.m.

New subject: [Boost-users] Usingboost::serializationinreal-timewithoutallocating memory

On Tue, Sep 22, 2009 at 00:11, Peter Soetens wrote:

...

On Mon, Sep 21, 2009 at 23:44, Peter Soetens wrote:

...
On Mon, Sep 21, 2009 at 21:25, Robert Ramey wrote:

...
Stefan Strasser wrote:

...
Am Monday 21 September 2009 20:27:59 schrieb Robert Ramey:

...
Stefan Strasser wrote:

...
I think to support this an other use cases like it common_archive would have to be moved outside of the linked library into a header and be made more generic.

common_archive is a template - it's not in the linked library.

ok, I must have mixed up some types. the point was that the ultimate base class, that handles the type registration and object tracking etc., is linked .cpp and not very configurable. the optimum case imho would be that you could pass a traits class to that archive base class, that gets called whenever there is an object or type to be registered and is queried for existing objects and types later on. that way you could disable registration, save type registration outside of the archive itself, let type registration span multiple archives etc, make sure type registration doesn't allocate etc.

as I alluded to before - what I would like to know is why I derived trivial_archive from common_archive. I don't remember what made me do this. It seems that if this isn't done then save/load_object is necessary. So the question is - why is this? It is an oversight somewhere? or does the archive concept have to be changed.

Personally, I generally don't like the "traits class" as an argument. It sort of implies that that the template has a lot of "if -" code which navigates the traits. A "policy class" is better but can still make things complex. Personally I prefer composition of funtionality through inheritance. This can be seen in the class diagram for the serialization library. And your example makes me more convinced than ever that this is basically right. Now the only loose end it investigate why save/load object is required.

I found a lead to that. When the archive calls archive::save, a chain of 'invoke' calls is done to select the right serialization method for type T. When I pass std::vector<double> as T, it encounters a branch in oserializer.hpp:245, in struct 'save_conditional::invoke' .

<code> // adds class information to the archive. This includes // serialization level and class version struct save_conditional { static void invoke(Archive &ar, const T &t){ //if(0 == (ar.get_flags() & no_tracking)) save_standard::invoke(ar, t); //else // save_only::invoke(ar, t); } }; </code>

The alternative branch is commented out, but let that be the path that needs to be taken by my code. Current code calls save_standard, which calls directly ' ar.save_object'. save_object in turn depends on basic_oarchive to do its job. If save_only was taken, the 'non-basic_oarchive' serialization code path would have been taken.

For some reason (I've spent an hour staring at it), my unit test did not discover this flaw in my code and does as if serialisation of the std::vector was fine. I'm overlooking something stupid clearly...

Anyway, as it seems now, the code path taken by archive::save/load mandates derivation of basic_?archive. Maybe I should have called another 'selection' function ? like archive::save_only directly ?

I tried using this instead of archive::save() : <code> boost::archive::detail::load_non_pointer_type::load_only::invoke(*this,t); </code>

But the compiler (gcc 4.3.3-5ubuntu4) refuses to compile this when the nvt types are in use. Basically, this is not fine: <code> int count; ar >> BOOST_SERIALIZATION_NVP(count); </code> While this is: <code> int count; nvp<int> n = BOOST_SERIALIZATION_NVP(count); ar >> n </code>

You can try this out in plain user code. Beats me. The errors I get for the first case are in the line of:

usr/include/boost/serialization/access.hpp: In static member function ‘static void boost::serialization::access::serialize(Archive&, T&, unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T = const boost::serialization::nvp<int>]’: /usr/include/boost/serialization/serialization.hpp:74: instantiated from ‘void boost::serialization::serialize(Archive&, T&, unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T = const boost::serialization::nvp<int>]’ /usr/include/boost/serialization/serialization.hpp:133: instantiated from ‘void boost::serialization::serialize_adl(Archive&, T&, unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T = const boost::serialization::nvp<int>]’ /usr/include/boost/archive/detail/iserializer.hpp:316: instantiated from ‘static void boost::archive::detail::load_non_pointer_type::load_only::invoke(Archive&, T&) [with Archive = RTT::marsh::binary_data_iarchive, T = const boost::serialization::nvp<int>]’ /home/kaltan/src/git/orocos-rtt/src/marsh/binary_data_archive.hpp:198: instantiated from ‘RTT::marsh::binary_data_iarchive& RTT::marsh::binary_data_iarchive::load_a_type(T&, mpl_::false_) [with T = const boost::serialization::nvp<int>]’ /home/kaltan/src/git/orocos-rtt/src/marsh/binary_data_archive.hpp:131: instantiated from ‘RTT::marsh::binary_data_iarchive& RTT::marsh::binary_data_iarchive::operator>>(T&) [with T = const boost::serialization::nvp<int>]’ /home/kaltan/src/git/orocos-rtt/tests/mqueue_archive_test.cpp:71: instantiated from here /usr/include/boost/serialization/access.hpp:109: error: passing ‘const boost::serialization::nvp<int>’ as ‘this’ argument of ‘void boost::serialization::nvp<T>::serialize(Archive&, unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T = int]’ discards qualifiers

Passing to operator>> the nvt struct as a return value of make_nvt() makes gcc add a const qualifier, which gets propagated all the way...I have no idea why the other archive implementations don't suffer this.

I solved it, and came to the conclusion that things got worse. It seems that, even when only calling <code> boost::archive::detail::load_non_pointer_type::load_only::invoke(*this,t); </code> I need to implement more function overloads (but not the load_object one), which are present in the binary archive case, but not in my class. I needed to add 3 load_override overloads in the iarchive: <code> template<class T> void load_override(T & t, BOOST_PFTO int){ load_a_type(t, boost::mpl::bool_() ); //archive::load(* this->This(), t); } void load_override(const boost::serialization::nvpboost::serialization::collection_size_type & t, int){ size_t x=0; * this >> x; t.value() = boost::serialization::collection_size_type(x); } template<class T> void load_override(const boost::serialization::nvp<T> & t, int){ T x=0; * this >> x; t.value() = x; } template<class T> Archive & operator>>(T & t){ this->load_override(t, 0); return * this; } </code> If you try to delete any of these three, serializing an std::vector will no longer work. In some way, the basic_iarchive accomodates for this case. So I traded one overload for load_object with three overloads for load_override. Because my load_object function was empty, it's very likely that the load_overrides were not required because the marshalling was not done at all. Someone with strong knowledge of all this needs to take my example + the existing codebase to something that does what is promised by the concept docs. The attached file is how the class lives in my current project, in the RTT::marsh namespace. I fixed my unit tests, and the class in attachment now works 100% for: * marshalling primitive types (non pointer) * marshalling structs/classes similar to std::vector, but which contain non-pointer data. It's not looking how I had imagined/hoped for, but it does the trick for me now. I'm happy to stick on this thread, but I'm not going to be able to solve this or make this an example that meets your expectations. It also looks that this thread no longer belongs on boost-users but on the boost list. To end with a positive note: I'm very happy how the separation between serialization and archiving has been (or has intended to be) done. It allows me to tap in a real-time archiver and re-using all serialization code. That's so cool. I checked the now working implementation again (stepped through the debugger as well) and it truely works for my test set. Peter

Robert Ramey

4:52 p.m.

New subject: [Boost-users] Usingboost::serializationinreal-timewithoutallocating memory

Peter Soetens wrote:

...

On Tue, Sep 22, 2009 at 00:11, Peter Soetens wrote:

...
On Mon, Sep 21, 2009 at 23:44, Peter Soetens wrote:

...
On Mon, Sep 21, 2009 at 21:25, Robert Ramey wrote:

...
Stefan Strasser wrote:

...
Am Monday 21 September 2009 20:27:59 schrieb Robert Ramey:

...
Stefan Strasser wrote: > I think to support this an other use cases like it > common_archive would have to be moved outside of the linked > library into a header and be made more generic.

common_archive is a template - it's not in the linked library.

ok, I must have mixed up some types. the point was that the ultimate base class, that handles the type registration and object tracking etc., is linked .cpp and not very configurable. the optimum case imho would be that you could pass a traits class to that archive base class, that gets called whenever there is an object or type to be registered and is queried for existing objects and types later on. that way you could disable registration, save type registration outside of the archive itself, let type registration span multiple archives etc, make sure type registration doesn't allocate etc.

as I alluded to before - what I would like to know is why I derived trivial_archive from common_archive. I don't remember what made me do this. It seems that if this isn't done then save/load_object is necessary. So the question is - why is this? It is an oversight somewhere? or does the archive concept have to be changed.

Personally, I generally don't like the "traits class" as an argument. It sort of implies that that the template has a lot of "if -" code which navigates the traits. A "policy class" is better but can still make things complex. Personally I prefer composition of funtionality through inheritance. This can be seen in the class diagram for the serialization library. And your example makes me more convinced than ever that this is basically right. Now the only loose end it investigate why save/load object is required.

I found a lead to that. When the archive calls archive::save, a chain of 'invoke' calls is done to select the right serialization method for type T. When I pass std::vector<double> as T, it encounters a branch in oserializer.hpp:245, in struct 'save_conditional::invoke' .

<code> // adds class information to the archive. This includes // serialization level and class version struct save_conditional { static void invoke(Archive &ar, const T &t){ //if(0 == (ar.get_flags() & no_tracking)) save_standard::invoke(ar, t); //else // save_only::invoke(ar, t); } }; </code>

The alternative branch is commented out, but let that be the path that needs to be taken by my code. Current code calls save_standard, which calls directly ' ar.save_object'. save_object in turn depends on basic_oarchive to do its job. If save_only was taken, the 'non-basic_oarchive' serialization code path would have been taken.

For some reason (I've spent an hour staring at it), my unit test did not discover this flaw in my code and does as if serialisation of the std::vector was fine. I'm overlooking something stupid clearly...

Anyway, as it seems now, the code path taken by archive::save/load mandates derivation of basic_?archive. Maybe I should have called another 'selection' function ? like archive::save_only directly ?

I tried using this instead of archive::save() : <code> boost::archive::detail::load_non_pointer_type::load_only::invoke(*this,t); </code>

But the compiler (gcc 4.3.3-5ubuntu4) refuses to compile this when the nvt types are in use. Basically, this is not fine: <code> int count; ar >> BOOST_SERIALIZATION_NVP(count); </code> While this is: <code> int count; nvp<int> n = BOOST_SERIALIZATION_NVP(count); ar >> n </code>

You can try this out in plain user code. Beats me. The errors I get for the first case are in the line of:

usr/include/boost/serialization/access.hpp: In static member function static void boost::serialization::access::serialize(Archive&, T&, unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T = const boost::serialization::nvp<int>]: /usr/include/boost/serialization/serialization.hpp:74: instantiated from void boost::serialization::serialize(Archive&, T&, unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T = const boost::serialization::nvp<int>] /usr/include/boost/serialization/serialization.hpp:133: instantiated from void boost::serialization::serialize_adl(Archive&, T&, unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T = const boost::serialization::nvp<int>] /usr/include/boost/archive/detail/iserializer.hpp:316: instantiated from static void boost::archive::detail::load_non_pointer_type
...
...
...
load_only::invoke(Archive&, T&) [with Archive = RTT::marsh::binary_data_iarchive, T = const boost::serialization::nvp<int>] /home/kaltan/src/git/orocos-rtt/src/marsh/binary_data_archive.hpp:198: instantiated from RTT::marsh::binary_data_iarchive& RTT::marsh::binary_data_iarchive::load_a_type(T&, mpl_::false_) [with T = const boost::serialization::nvp<int>] /home/kaltan/src/git/orocos-rtt/src/marsh/binary_data_archive.hpp:131: instantiated from RTT::marsh::binary_data_iarchive& RTT::marsh::binary_data_iarchive::operator>>(T&) [with T = const boost::serialization::nvp<int>] /home/kaltan/src/git/orocos-rtt/tests/mqueue_archive_test.cpp:71: instantiated from here /usr/include/boost/serialization/access.hpp:109: error: passing const boost::serialization::nvp<int> as this argument of void boost::serialization::nvp<T>::serialize(Archive&, unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T = int] discards qualifiers

Passing to operator>> the nvt struct as a return value of make_nvt() makes gcc add a const qualifier, which gets propagated all the way...I have no idea why the other archive implementations don't suffer this.

I solved it, and came to the conclusion that things got worse. It seems that, even when only calling <code>

boost::archive::detail::load_non_pointer_type::load_only::invoke(*this,t); </code>

I need to implement more function overloads (but not the load_object one), which are present in the binary archive case, but not in my class. I needed to add 3 load_override overloads in the iarchive: <code> template<class T> void load_override(T & t, BOOST_PFTO int){ load_a_type(t, boost::mpl::bool_() ); //archive::load(* this->This(), t); }

void load_override(const boost::serialization::nvpboost::serialization::collection_size_type & t, int){ size_t x=0; * this >> x; t.value() = boost::serialization::collection_size_type(x); }

template<class T> void load_override(const boost::serialization::nvp<T> & t, int){ T x=0; * this >> x; t.value() = x; }

template<class T> Archive & operator>>(T & t){ this->load_override(t, 0); return * this; } </code>

If you try to delete any of these three, serializing an std::vector will no longer work. In some way, the basic_iarchive accomodates for this case. So I traded one overload for load_object with three overloads for load_override. Because my load_object function was empty, it's very likely that the load_overrides were not required because the marshalling was not done at all.

Someone with strong knowledge of all this needs to take my example + the existing codebase to something that does what is promised by the concept docs.

The attached file is how the class lives in my current project, in the RTT::marsh namespace. I fixed my unit tests, and the class in attachment now works 100% for: * marshalling primitive types (non pointer) * marshalling structs/classes similar to std::vector, but which contain non-pointer data.

It's not looking how I had imagined/hoped for, but it does the trick for me now. I'm happy to stick on this thread, but I'm not going to be able to solve this or make this an example that meets your expectations. It also looks that this thread no longer belongs on boost-users but on the boost list.

To end with a positive note: I'm very happy how the separation between serialization and archiving has been (or has intended to be) done. It allows me to tap in a real-time archiver and re-using all serialization code. That's so cool. I checked the now working implementation again (stepped through the debugger as well) and it truely works for my test set.

Peter

...
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Robert Ramey

4:55 p.m.

New subject: [Boost-users] Usingboost::serializationinreal-timewithoutallocating memory

Peter Soetens wrote:

...

Someone with strong knowledge of all this needs to take my example + the existing codebase to something that does what is promised by the concept docs.

How about posting this information in the form of a TRAC item. I'll look at it when I have the time.

...

The attached file is how the class lives in my current project, in the RTT::marsh namespace. I fixed my unit tests, and the class in attachment now works 100% for: * marshalling primitive types (non pointer) * marshalling structs/classes similar to std::vector, but which contain non-pointer data.

It's not looking how I had imagined/hoped for, but it does the trick for me now. I'm happy to stick on this thread, but I'm not going to be able to solve this or make this an example that meets your expectations. It also looks that this thread no longer belongs on boost-users but on the boost list.

...

To end with a positive note: I'm very happy how the separation between serialization and archiving has been (or has intended to be) done. It allows me to tap in a real-time archiver and re-using all serialization code. That's so cool. I checked the now working implementation again (stepped through the debugger as well) and it truely works for my test set.

Good to hear. Sounds like we're batting at least .500 Robert Ramey

Robert Ramey

18 Sep 18 Sep

4:17 p.m.

New subject: Using boost::serialization in real-timewithoutallocating memory

Peter Soetens wrote:

...

On Thu, Sep 17, 2009 at 18:33, Robert Ramey wrote:

...
Peter Soetens wrote:

...
I might do what you suggest or look for another serialization solution.

Given the limited scope of what you want to do, it wouldn't be THAT hard to make your own archive. Take a look at "trivial_archive" in the documentation and build from that.

Encouraged by your and Stefan's suggestions, I've spent another day at looking what the possibilities are.

1. I first created my own binary_?archive classes which inherited from common_?archive. This resulted in pulling in the library with the 'behind the scenes type tracking'. It seems, inheriting from common is not an option for me if I want 'zero-allocation' serialization. Correct ? (I also tested the demo_fast_archive.cpp extended with no_header, but that one also allocated).

That includes too much - I guess you've verified that.

...

2. I then tried to work further on Stefan's code snippet. It works, but indeed only for 'primitive_type's. I *think* I need functionality upto the 'object_serializable' if I want std::vector<T> as well. Since Stefan's operator& consumes every T, I'll need one operator& for each primitive type, and then an operator& that tries to save/load using free functions of the not primitive type.

3. It looks that I'll have to implement the Archive concept from scratch or at least based on Stefan's code and the load_binary/save_binary code from binary_?archive.

Am I on the right track ?

Sounds like it to me. You might have to borrow some other pieces from other archive implementaions. Robert Ramey

...

Peter

Peter Soetens

17 Sep 17 Sep

1:42 p.m.

On Thu, Sep 17, 2009 at 15:10, Peter Soetens wrote:

...

Hi,

I'm trying to find out if boost::serialization can be used in real-time applications to stream data into a fifo to another process. It is mandatory that no memory allocations happen during the serialization. I tested this with a std::vector<double> of 10 elements in combination with the boost::iostreams library.

My assumptions about the cause of the memory allocation were wrong. I could trace it to this point: #0 0x00007fdcbf4ed9a0 in operator new () from /usr/lib/libstdc++.so.6 #1 0x00007fdcbf76c984 in std::_Rb_treeboost::archive::detail::basic_oarchive_impl::cobject_type, std: :lessboost::archive::detail::basic_oarchive_impl::cobject_type, std::allocatorboost::archive::detail::basic_oarchive_impl::cobject_type

...

::_M_insert_ (this=0x94b3b8, __x=0x0, __p=0x94b3c0, __v=@0x7fffc7bd0620) at /usr/include/c++/4.3/ext/new_allocator.h:92 #2 0x00007fdcbf76c734 in boost::archive::detail::basic_oarchive::save_object (this=0x7fffc7bd0830, x=0x7fffc7bd0860, bos=@0x611550) at /usr/include/c++/4.3/bits/stl_tree.h:1148 #3 0x000000000040a5b5 in boost::archive::detail::save_non_pointer_type >::save_standard::invoke (ar=@0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/detail/oserializer.hpp:231 #4 0x000000000040a5d4 in boost::archive::detail::save_non_pointer_type ::save_conditional::invoke (ar=@0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/detail/oserializer.hpp:245 #5 0x000000000040a5f3 in boost::archive::detail::save_non_pointer_type >::invoke (ar=@0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/detail/oserializer.hpp:294 #6 0x000000000040a612 in boost::archive::save > (ar=@0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/detail/oserializer.hpp:506 #7 0x000000000040a63c in boost::archive::detail::common_oarchiveboost::archive::binary_oarchive::save_override const> (this=0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/detail/common_oarchive.hpp:64 #8 0x000000000040a661 in boost::archive::basic_binary_oarchiveboost::archive::binary_oarchive::save_override > (this=0x7fffc7bd0830, t=@0x7fffc7bd0860, version=0) at /usr/include/boost/archive/basic_binary_oarchive.hpp:63 #9 0x000000000040a689 in boost::archive::binary_oarchive_impl::save_override > (this=0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/binary_oarchive_impl.hpp:45 #10 0x000000000040a6b6 in boost::archive::detail::interface_oarchiveboost::archive::binary_oarchive::operator<< > (this=0x7fffc7bd0830, t=@0x7fffc7bd0860) at /usr/include/boost/archive/detail/interface_oarchive.hpp:64 #11 0x0000000000404a65 in main (argc=1, argv=0x7fffc7bd0f08) at serialize-rt-init.cpp:17

Which looks like std::map is being used to store class data (or so). It looks like it's going to be much harder to work around this than I assumed ? Peter

...

The aim is that all memory allocations happen during construction of the archive object, while the serialisation itself causes none.

<code> #include #include #include #include

namespace io = boost::iostreams; int main(int argc, char *argv[]) { vector<double> d(10, 1.1); char sink[1000]; memset( sink, 0, 1000); io::streamio::array_sink out(sink,1000);

boost::archive::binary_oarchive oa(out); //oa << d; // should not allocate return 0; } </code>

The setup code does 10 memory allocations according to valgrind: ==12995== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 1) ==12995== malloc/free: in use at exit: 0 bytes in 0 blocks. ==12995== malloc/free: 10 allocs, 10 frees, 913 bytes allocated. ==12995== For counts of detected errors, rerun with: -v ==12995== All heap blocks were freed -- no leaks are possible.

If we uncomment 'oa << d' we get 2 more (unwanted) allocations: ==13010== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 1) ==13010== malloc/free: in use at exit: 0 bytes in 0 blocks. ==13010== malloc/free: 12 allocs, 12 frees, 1,001 bytes allocated. ==13010== For counts of detected errors, rerun with: -v ==13010== All heap blocks were freed -- no leaks are possible

I'm guessing that the 2 allocations in the serialisation path come from a temporay std::string object, when writing the 'serialization::archive' string into the archive. Wouldn't it be possible to rewrite this library code as such that there are no allocations / strings created ? A similar pattern is observed for deserializing: 2 allocs in the serialisation path.

Peter

PS: I tried to find out who defines the macro BOOST_ARCHIVE_SIGNATURE() but I couldn't find the definition/#define !?

5534

Age (days ago)

5539

Last active (days ago)

List overview

Download

25 comments

3 participants

participants (3)

Peter Soetens
Robert Ramey
Stefan Strasser