Johan Råde wrote:

Cliff Green wrote:

Last fall I wrote a library with portable fpclassify, isnan, signbit etc. (It is available in the vault in the Math - Numerics folder.) Several Boosters tested the library with more than 20 different OS / compiler / processor combinations.

This is something that is really needed in boost. Could you possibly: a) make a small boost library for this which would include i) code (hopefully header only?) ii) tests iii) documentation and ask for a mini-review. b) enhance the portable binary archive to take advantage of your code. c) enhance the serialization documentation to explain that portable binary archive. I realize that although each of these steps isn't all that large, taken together they do add up to a non-trivial amount of work. In order to compensate with you'll get. a) a tiny increment in one man's quest (yours) for immortality. b) a very nice addition to your resume. c) the satisfaction of knowing that your work has passed muster with what might be the pickiest group of programmers in the known universe. d) the satisfaction of knowing that your code is probably a key piece a perhaps thousands of applications. I know its not much, but its the best I can offer.

This experience taught me a lot about portable handling of floating point numbers.

Which will bring its own curse. In the course of going through even a mini-review, you'll have to spend a lot of time with people who "know" a lot more than you do. Good Luck. Robert Ramey

--Johan Råde

Robert Ramey wrote:

I realize that although each of these steps isn't all that large, taken together they do add up to a non-trivial amount of work. In order to compensate with you'll get.

a) a tiny increment in one man's quest (yours) for immortality. b) a very nice addition to your resume. c) the satisfaction of knowing that your work has passed muster with what might be the pickiest group of programmers in the known universe. d) the satisfaction of knowing that your code is probably a key piece a perhaps thousands of applications.

I know its not much, but its the best I can offer.

Maybe you could offer me a fix for the bug that makes the serialization library crash when you throw an exception in a load_construct_data function ;-) --Johan Råde

Robert Ramey wrote:

Johan Råde wrote:

...

Maybe you could offer me a fix for the bug that makes the serialization library crash when you throw an exception in a load_construct_data function ;-)

LOL - hmmm - I think I looked into this but I don't remember what I did with it. Send me a link to the original complaint and I'll check again.

Robert Ramey

The original complaint was in a private e-mail that I sent you 2 1/2 years ago. I don't think I will be able to find it. The sender was rade@maths.lth.se. If you don't have the mail, I'd be happy to send you a short example that reproduces the bug. --Johan Råde

My current copy the code includes in the file iserializer.hpp (line # 291) the following code. Doesn't this address the situation? Robert Ramey template BOOST_DLLEXPORT void pointer_iserializer::load_object_ptr( basic_iarchive & ar, void * & x, const unsigned int file_version ) const { Archive & ar_impl = boost::smart_cast_reference(ar); auto_ptr_with_deleter<T> ap(heap_allocator<T>::invoke()); if(NULL == ap.get()) boost::throw_exception(std::bad_alloc()) ; T * t = ap.get(); x = t; // catch exception during load_construct_data so that we don't // automatically delete the t which is most likely not fully // constructed BOOST_TRY { // this addresses an obscure situtation that occurs when // load_constructor de-serializes something through a pointer. ar.next_object_pointer(t); boost::serialization::load_construct_data_adl( ar_impl, t, file_version ); } BOOST_CATCH(...){ ap.release(); BOOST_RETHROW; } BOOST_CATCH_END ar_impl >> boost::serialization::make_nvp(NULL, * t); ap.release(); } Johan Råde wrote:

Robert Ramey wrote:

...

Johan Råde wrote:

...

Maybe you could offer me a fix for the bug that makes the serialization library crash when you throw an exception in a load_construct_data function ;-)

LOL - hmmm - I think I looked into this but I don't remember what I did with it. Send me a link to the original complaint and I'll check again.

Robert Ramey

The original complaint was in a private e-mail that I sent you 2 1/2 years ago. I don't think I will be able to find it. The sender was rade@maths.lth.se. If you don't have the mail, I'd be happy to send you a short example that reproduces the bug.

--Johan Råde

This is something that is really needed in boost. Could you possibly:

a) make a small boost library for this which would include i) code (hopefully header only?) ii) tests iii) documentation

and ask for a mini-review.

b) enhance the portable binary archive to take advantage of your code.

c) enhance the serialization documentation to explain that portable binary archive.

I realize I need to look at the current portable binary archive example code to better understand the design tradeoffs, but there's a number of possible approaches to consider. To me, it makes sense to have multiple portable binary archives, allowing the app to make the best choice for its need (maybe a good "compromise" binary archive could be the default for Boost.Serialization). Possible choices (some already discussed): 1. Absolute smallest space - e.g. "continuation" bits, allowing integers to be 1, 2, 3, 4, etc bytes in size, depending on if the value is (respectively) 127 or less, 32767 or less, etc. (I'm pretty sure this has been discussed before). There could even be a "bitstream" packing versus byte-oriented, but the set of users for that might be pretty small (I've written "bitstream" protocols for wireless networks before, and believe me, the compactness is significantly more than the usual binary network protocols). 2. Absolute fastest speed - XDR fits this, as it is not all that space efficient (e.g. bools take up 4 bytes, short ints take up 4 bytes, etc). But it definitely is 4 byte aligned in everything (however, I would guess that the speed differences regarding alignment are much less for modern processors versus when XDR was first defined). 3. Flexibility versus simplicity - e.g. whether object ids, tags, implicit structuring and nesting, etc are directly supported versus just providing support for fundamental types, and (relatively) simple sequences. 4. Floating point support - IEE 754 formats only versus later standards (and long double), other non-typical floating point formats, and probably a host of other possibilities for binary floating point. 5. Adherence to a commonly used (or even not that commonly used) protocol - XDR, CDR, etc. Cliff

-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Cliff Green Sent: 26 June 2007 23:58 To: boost-users@lists.boost.org Subject: Re: [Boost-users] [serialisation]Portable Binary Archives

...

Probably answered before, but I wondered whether there is a standard implementation for portable binary archives available.

I'm not quite sure what you mean by "standard implementation" - there are a number of commonly used portable binary approaches or standards. For example, XDR (IETF standard) has been around for quite a while:

http://tools.ietf.org/rfc/rfc4506.txt

as well as SDXF (not sure how much it is used):

http://tools.ietf.org/rfc/rfc3072.txt

There's CDR, used in CORBA and other libraries or frameworks where interoperability is needed:

http://en.wikipedia.org/wiki/Common_Data_Representation

I'm sure there's a gaggle of home-grown approaches (I've written some myself) as well as other industry standard approaches.

If you're talking about portable binary archives for Boost.Serialization, there's been talk, but I'm not sure what the status is - anyone have an implementation? I'd be willing to help / work with someone on it.

That's what I am looking for. We need to get the file sizes down. We could use text (XML is great for debugging but too big for final release), but again that has its overheads. We generally don't use floating point so that may help us. The link I posted does have a portable implementation which I think uses IEEE754. I just wondered if there was an official boost:serilisation version; obviously not!! <snip> James This message (including any attachments) contains confidential and/or proprietary information intended only for the addressee. Any unauthorized disclosure, copying, distribution or reliance on the contents of this information is strictly prohibited and may constitute a violation of law. If you are not the intended recipient, please notify the sender immediately by responding to this e-mail, and delete the message from your system. If you have any questions about this e-mail please notify the sender immediately.

I don't think binary archives are going to be smaller than the equivalent text archive. native binary archives are built for speed, text for portability, and xml to satisfy those who feel they need it.

I generally agree, but I wouldn't be surprised if binary archives are smaller than text archives for everything but unusual cases. It may not be a significant percentage difference, and of course without some objective test cases I'm just guessing. And of course we need to better define the specifics of a "binary archive" - I'll follow-up with a reply to your next e-mail. Cliff

"Scott McMurray" wrote:

FWIW, I'm currently using binary files for some work because the text is huge and slow.

... down to 1.2 GiB instead of the 2.1 GiB for text ... It also makes my load time drop from a few minutes to a few seconds

This has been my (somewhat limited application domain) experience, as well. Almost all of the distributed systems I've worked on that used binary message data passing did so for both processing speed and compactness of the data. Most of the system architects understood the "brittleness" and drawbacks of the design (specially when floating point values were needed), but were willing to make the tradeoff. I'd love to be able to remove at least some of the code design and portability drawbacks for applications needing to pass (or read / write) binary data. Cliff

I am hopeful that progress will soon be made on this area. This would accomplish: a) fixing noted bugs in the portable binary archive and making it pass the serialization torture tests that the other archives are subjected to. b) subjecting it (along with other archives) to profiling at least with gcc compilers. I wasn't planning on addressing floating point. If someone wants to do this I would certainly be happy to lend my constructive criticism to the effort. The main issues are: a) portable subset of NaN's etc. It seems to me this is ready to that this code has been done by John Rade? so that's would seem ok. b) portable encoding of floating point - a very attractive proposal was submitted some time ago (with code !!!) by Ralf W Grosse-Kunstleve search the mailing list for this. So feel free to get busy. Robert Ramey Pfligersdorffer, Christian wrote:

So, is there anything going on in this cause? I really need floating point types so I'd love to see some progress in that. I am also willing to do some coding if need be.

Regards,

"Scott McMurray" wrote:

...

FWIW, I'm currently using binary files for some work because the text is huge and slow.

... down to 1.2 GiB instead of the 2.1 GiB for text ... It also makes my load time drop from a few minutes to a few seconds

This has been my (somewhat limited application domain) experience, as well. Almost all of the distributed systems I've worked on that used binary message data passing did so for both processing speed and compactness of the data. Most of the system architects understood the "brittleness" and drawbacks of the design (specially when floating point values were needed), but were willing to make the tradeoff.

I'd love to be able to remove at least some of the code design and portability drawbacks for applications needing to pass (or read / write) binary data.

Cliff _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Hughes, James wrote:

Thanks for that comment on sizes Robert - will bring that to the table when we next talk about serialisation here!!!!

Just to clarify. Suppose you have an archive with a lot 4 byte integers whose values contain small numbers - 1 or two digits. The text archive would represent this as: 1 24 14 2 7 ... averaging 3 bytes for integer while the native binary archive would represent the values as 4 bytes per integer. I don't remember about floating point format off hand. So... whether the text or binary is larger or smaller would depend on the specific case. I doubt that there is a huge difference either way. Robert Ramey