[boost::serialization] want access to binary data sans bookkeeping data
Hi, I am wanting to serialize some C++ data structures into a binary stream, and I'm wanting to send that binary stream to an embedded python interpreter, where it can unpack the data and make use of it. So, this doesn't seem like a typical use of the serialization library, but I was wondering if it can still work. My problem is that it appears (from what I can tell) that there are various administrative bits inserted by the serialization library into the binary stream, meant for the "unserializer" to make use of in some way (e.g. a header, version information, etc.) Of course, this would mess up things for my python reader. I'd like to, as efficiently as possible, serialize, say, a 3d vector into a binary stream of 3 floats, and then send the raw binary data as an argument to a python function, and have the python function unpack this into 3 floats. This just a simple example though, but as long as the sender (C++) and the receiver (Python) both know the format, I could send arbitrary information this way, as long as there's no other bookkeeping data in the stream. Is there any way to gain access to the raw binary data, after I've filled things with my binary_oarchive, so I can send it to my python function? Or is this just not really a good use of serialization? Thanks!
Brian Stempel wrote:
I am wanting to serialize some C++ data structures into a binary stream, and I'm wanting to send that binary stream to an embedded python interpreter, where it can unpack the data and make use of it.
I have exactly the same problem: I need several kind of my own defined streamer. One is to write binary data 1:1, as Brian would like to do; my other idea is: Use the operator& syntax to stream either into a file, or into an SQL database, depending on the streamer (in my idea, the SQL-streamer would be similar to the existing XML-archive, but the XML-tags would here name tables and column names). Now I started by reading the "Serialization Archive Class Reference" on http://boost.org/libs/serialization/doc/archive_reference.html, which describes how to implement my own archive. Unfortunately, if I copy the code from the "simplest possible output archive" there into a header file, then write a code file that does nothing except including that header, the result is not compilable (GNU g++ 4.1.2). First problem: friend class boost::archive::load_access is undefined, so I have to add a forward declaration: namespace boost { namespace archive { class load_access; } } Second problem (see attached test.cpp):
g++ test.cpp /usr/include/boost/archive/detail/interface_iarchive.hpp: In member function ‘const boost::archive::detail::basic_pointer_iserializer* boost::archive::detail::interface_iarchive<Archive>::register_type(T*)’: /usr/include/boost/archive/detail/interface_iarchive.hpp:54: error: ‘instantiate_pointer_iserializer’ is not a member of ‘boost::archive::detail’ /usr/include/boost/archive/detail/interface_iarchive.hpp:58: error: invalid use of undefined type ‘const struct boost::archive::detail::basic_pointer_iserializer’ /usr/include/boost/archive/detail/basic_iarchive.hpp:41: error: forward declaration of ‘const struct boost::archive::detail::basic_pointer_iserializer’
What's missing? Thank you Regards Marc
I believe that this has been addressed in boost 1.34. Robert Ramey Marc Wäckerlin wrote:
Brian Stempel wrote:
I am wanting to serialize some C++ data structures into a binary stream, and I'm wanting to send that binary stream to an embedded python interpreter, where it can unpack the data and make use of it.
I have exactly the same problem: I need several kind of my own defined streamer. One is to write binary data 1:1, as Brian would like to do; my other idea is: Use the operator& syntax to stream either into a file, or into an SQL database, depending on the streamer (in my idea, the SQL-streamer would be similar to the existing XML-archive, but the XML-tags would here name tables and column names).
Now I started by reading the "Serialization Archive Class Reference" on http://boost.org/libs/serialization/doc/archive_reference.html, which describes how to implement my own archive.
Unfortunately, if I copy the code from the "simplest possible output archive" there into a header file, then write a code file that does nothing except including that header, the result is not compilable (GNU g++ 4.1.2).
First problem:
friend class boost::archive::load_access is undefined, so I have to add a forward declaration: namespace boost { namespace archive { class load_access; } }
Second problem (see attached test.cpp):
g++ test.cpp /usr/include/boost/archive/detail/interface_iarchive.hpp: In member function 'const boost::archive::detail::basic_pointer_iserializer* boost::archive::detail::interface_iarchive<Archive>::register_type(T*)': /usr/include/boost/archive/detail/interface_iarchive.hpp:54: error: 'instantiate_pointer_iserializer' is not a member of 'boost::archive::detail' /usr/include/boost/archive/detail/interface_iarchive.hpp:58: error: invalid use of undefined type 'const struct boost::archive::detail::basic_pointer_iserializer' /usr/include/boost/archive/detail/basic_iarchive.hpp:41: error: forward declaration of 'const struct boost::archive::detail::basic_pointer_iserializer'
What's missing?
Thank you Regards Marc
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Marc Wäckerlin wrote:
Brian Stempel wrote:
Now I started by reading the "Serialization Archive Class Reference" on http://boost.org/libs/serialization/doc/archive_reference.html, which describes how to implement my own archive.
Unfortunately, if I copy the code from the "simplest possible output archive" there into a header file, then write a code file that does nothing except including that header, the result is not compilable (GNU g++ 4.1.2).
First problem:
friend class boost::archive::load_access is undefined, so I have to add a forward declaration: namespace boost { namespace archive { class load_access; } }
Hmm - I looked at this again and didn't see the need for the above. In fact, when I compiled with my other compilers - none complained. So, though its harmless to add - I'm not sure it should be in the example.
Second problem (see attached test.cpp):
In compiling your/my example from the document with VC 7.1 - though not with the other compilers at hand, I did get an error - which was a surprise to me. However, its a different error than the one you get. Also none of my other compilers had a problem with this. So my real answer to your question "What's missing?" is "I don't know". Both your error message from gcc 4.1 and mine from VC 7.1 - which are different - can't be explained by me. So I would say we'll have to call this "under investigation" Adding more "#includes" might seem to fix the problem. But the point of the example is to show what a minimal example is - so I'm really curious as to what is going on. You might try replacing your versions of the attached files with the attached one and let me know what happens. Robert Ramey
g++ test.cpp /usr/include/boost/archive/detail/interface_iarchive.hpp: In member function 'const boost::archive::detail::basic_pointer_iserializer* boost::archive::detail::interface_iarchive<Archive>::register_type(T*)': /usr/include/boost/archive/detail/interface_iarchive.hpp:54: error: 'instantiate_pointer_iserializer' is not a member of 'boost::archive::detail' /usr/include/boost/archive/detail/interface_iarchive.hpp:58: error: invalid use of undefined type 'const struct boost::archive::detail::basic_pointer_iserializer' /usr/include/boost/archive/detail/basic_iarchive.hpp:41: error: forward declaration of 'const struct boost::archive::detail::basic_pointer_iserializer'
What's missing?
Thank you Regards Marc
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
begin 666 interface_iarchive.hpp
M(VEF;F1E9B!"3T]35%]!4D-(259%7T1%5$%)3%])3E1%4D9!0T5?24%20TA)
M5D5?2%!0#0HC9&5F:6YE($)/3U-47T%20TA)5D5?1$5404E,7TE.5$521D%#
M15])05)#2$E615](4% -"@T*+R\@35,@8V]M<&%T:6)L92!C;VUP:6QE
I got it. Please see my attachment.
This BinaryWriter and BinaryReader does store all data in plain binary
without any additional information (version, etc.). It runs in our project,
so you'll have to remove the LOG_...-macros and maybe to rename some parts.
In case of any problems in getting it to run, just ask here.
Changes:
The forward declaration for the friends (see previous mail) is no more
necessary, when you include iserializer.hpp
The following is missing in the example:
1. #include
I'm happy to see you've got things working to your taste. My real interest is in trivial_archive which is should compile as is - and in fact does with the compilers that I currently have available. Marc Wäckerlin wrote:
The following is missing in the example:
1. #include
I'm really interested in knowing what part of this "#include" is making things easier with your compiler. I'm attaching updated versions of files I would like to seen tested with your compiler. I would hope that these changes would make including "iserializer" unnecessary.
2. Overwriting "template<class T> void save_override(const T& t, int)" is mandatory, not optional!
Mandatory to make it compile or make it work for your application?
Replacing the default implementations in common_?archive<...>
is of course legal. But I can't believe that replacing them with no-ops
is going to work in general. The skipped information (class_id etc.)
is only there because it was deemed indispensible to make the system
work.
Robert Ramey
begin 666 common_iarchive.hpp
M(VEF;F1E9B!"3T]35%]!4D-(259%7T1%5$%)3%]#3TU-3TY?24%20TA)5D5?
M2%!0#0HC9&5F:6YE($)/3U-47T%20TA)5D5?1$5404E,7T-/34U/3E])05)#
M2$E615](4% -"@T*+R\@35,@8V]M<&%T:6)L92!C;VUP:6QE
I attached the latest, standalone-compilable example. Compiles with: g++ serializer.cxx -lboost_serialization Robert Ramey wrote:
Marc Wäckerlin wrote:
The following is missing in the example: 1. #include
I'm really interested in knowing what part of this "#include" is making things easier with your compiler. I'm attaching updated versions of files I would like to seen tested with your compiler.
Then I'm not very lucky: In file included from serializer.cxx:4: /usr/include/boost/archive/detail/common_iarchive.hpp:22:53: error: boost/archive/detail/register_archive.hpp: Datei oder Verzeichnis nicht gefunden I'm using boost-1.33.1-42 from an RPM-SuSE archive, and I am not very keen on compiling it myself (because I want it in my RPM database, but also because of boost's build system - I refuse to use "jam").
2. Overwriting "template<class T> void save_override(const T& t, int)" is mandatory, not optional!
Mandatory to make it compile or make it work for your application?
Mandatory to compile. Then, as you see in my example (last Mail), I overwrite all that produces additional information in order to filter it out. That's then for my application. Regards Marc
Marc Wäckerlin wrote:
I attached the latest, standalone-compilable example. Compiles with: g++ serializer.cxx -lboost_serialization
Robert Ramey wrote:
Marc Wäckerlin wrote:
The following is missing in the example: 1. #include
I'm really interested in knowing what part of this "#include" is making things easier with your compiler. I'm attaching updated versions of files I would like to seen tested with your compiler. Then I'm not very lucky:
In file included from serializer.cxx:4: /usr/include/boost/archive/detail/common_iarchive.hpp:22:53: error: boost/archive/detail/register_archive.hpp: Datei oder Verzeichnis nicht gefunden
OK - thanks for trying. I'm working with code from HEAD (1.35) which is sufficiently different that we can't just "mix and match".
2. Overwriting "template<class T> void save_override(const T& t, int)" is mandatory, not optional!
Mandatory to make it compile or make it work for your application?
Mandatory to compile.
OK - a) BOOST_SERIALIZATION_SPLIT_FREE(...) can only be used outside of any namespace. I've added this to the documentation Note that split_free.hpp should also be included. b) I've investigated 2) above and found that there there is an error common-?archive.hpp which only appears for the most recent compilers. I've now fixed it on my machine and tested with various compilers. Now both your example and the "trivial_iarchive" from the documentation compile without errors. Thanks for your help in spotting this. Robert Ramey
The documentation "Serialization Archive Class Reference" on http://boost.org/libs/serialization/doc/archive_reference.html is wrong, or at lease very incomplete: Brian Stempel wrote:
I am wanting to serialize some C++ data structures into a binary stream, and I'm wanting to send that binary stream to an embedded python interpreter, where it can unpack the data and make use of it.
I have exactly the same problem: I need several kind of my own defined streamer. One is to write binary data 1:1, as Brian would like to do; my other idea is: Use the operator& syntax to stream either into a file, or into an SQL database, depending on the streamer (in my idea, the SQL-streamer would be similar to the existing XML-archive, but the XML-tags would here name tables and column names). Now I started by reading the "Serialization Archive Class Reference" on http://boost.org/libs/serialization/doc/archive_reference.html, which describes how to implement my own archive. Unfortunately, if I copy the code from the "simplest possible output archive" there into a header file, then write a code file that does nothing except including that header, the result is not compilable (GNU g++ 4.1.2). First problem (solved): friend class boost::archive::load_access is undefined, so I have to add a forward declaration: namespace boost { namespace archive { class load_access; } } Second problem (solved): I need to include boost/archive/detail/iserializer.hpp, otherwise I get: /usr/include/boost/archive/detail/interface_iarchive.hpp: In member function ‘const boost::archive::detail::basic_pointer_iserializer* boost::archive::detail::interface_iarchive<Archive>::register_type(T*)’: /usr/include/boost/archive/detail/interface_iarchive.hpp:54: error: ‘instantiate_pointer_iserializer’ is not a member of ‘boost::archive::detail’ /usr/include/boost/archive/detail/interface_iarchive.hpp:58: error: invalid use of undefined type ‘const struct boost::archive::detail::basic_pointer_iserializer’ /usr/include/boost/archive/detail/basic_iarchive.hpp:41: error: forward declaration of ‘const struct boost::archive::detail::basic_pointer_iserializer’ Then I can compile if the "main()" is empty. Third problem (unsolved): When I try to instantiate the archive abd store something in the archive, then it does not compile at all. See the attached files test.cpp and errors. Try: g++ test.cpp What's missing? One point is: There's no default constructor! - What is meant by the "int flags" parameter? - What else is missing? - Who can provide a minimal sample that works? Thank you Regards Marc
participants (3)
-
Brian Stempel -
Marc Wäckerlin -
Robert Ramey