August 2008 - Boost - lists.preview.boost.org

Thoughts & questions on adding serialization
by Daryle Walker 13 Aug '08

13 Aug '08

I was thinking about adding serialization to some times I've been working on in the sandbox. First I tried to recall how Mr. Ramey said serialization can be tested. I couldn't find the specific post I was thinking about, but others that were found gave me the answer. Reading other posts in that search prompted me to ask more questions. I could reduce the classes I'm working with to: //============================================= class computer; class context { public: … [View More]typedef boost::array<uint_least32_t, 4> value_type; context(); // use auto copy-ctr, copy-=, dtr void operator ()( bool ); // consumer bool operator ==() const; // equals bool operator !=() const; // not-equals value_type operator ()() const; // producer private: friend class computer; boost::uint_fast64_t length; boost::array<uint_fast32_t, 4> buffer; boost::array<bool, 512> queue; template < class Archive > void serialize( Archive &ar, const unsigned int version ); }; class computer : public convenience_methods_base<context> { // An object of type "context" is incorporated in this object // due to the base class. A mutable/const pair of non-static // member functions named "context()" gives access to the inner // context object. public: typedef context::value_type value_type; // Put various access member functions here that forward to the // internals of the "context" type, which work because of the // friend declaration. private: template < class Archive > void serialize( Archive &ar, const unsigned int version ); }; //============================================= I initially planned to have serialization functions for these two classes, the "convenience_methods_base" base class template, plus two other class templates (a base class and a support class) that "convenience_methods_base" uses. But the e-mail search I mentioned found a thread from May 2007 (on the main Boost list) the suggested that the serialization of a non-primitive should match the user's external representation of the type, and not the type's particular internal structure. So I decided to keep the serialization protocol just for the two public-facing classes, "context" and "computer." I figured that the "computer" object can be serialized like: //============================================= template < class Archive > inline void computer::serialize( Archive &ar, const unsigned int version ) { ar & boost::serialization::make_nvp("context", this->context()); } //============================================= Which leaves how "context" objects are serialized. After thinking about it for hours, I decided to just whip out something quick & dirty and refine it later. So: //============================================= template < class Archive > inline void context::serialize( Archive &ar, const unsigned int version ) { ar & BOOST_SERIALIZATION_NVP( length ) & BOOST_SERIALIZATION_NVP( buffer ) & BOOST_SERIALIZATION_NVP( queue ); } //============================================= would give a final serialization, in my test file, of: //============================================= <?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <!DOCTYPE boost_serialization> <boost_serialization signature="serialization::archive" version="5"> <test class_id="0" tracking_level="0" version="0"> <context class_id="1" tracking_level="0" version="0"> <length>1</length> <buffer class_id="2" tracking_level="0" version="0"> <elems> <count>4</count> <item>1732584193</item> <item>4023233417</item> <item>2562383102</item> <item>271733878</item> </elems> </buffer> <queue class_id="3" tracking_level="0" version="0"> <elems> <count>512</count> <item>1</item> <item>0</item>  <item>0</item> </elems> </queue> </context> </test> </boost_serialization> //============================================= Now I started refining, keeping the principle of not leaking implementation details in mind. The problem here is the array- counts, which I don't need since they'll never change. The first one I can fix by writing each element separately: //============================================= <?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <!DOCTYPE boost_serialization> <boost_serialization signature="serialization::archive" version="5"> <test class_id="0" tracking_level="0" version="0"> <context class_id="1" tracking_level="0" version="0"> <length>1</length> <buffer-A>1732584193</buffer-A> <buffer-B>4023233417</buffer-B> <buffer-C>2562383102</buffer-C> <buffer-D>271733878</buffer-D> <message-tail class_id="2" tracking_level="0" version="0"> <elems> <count>512</count> <item>1</item> <item>0</item>  <item>0</item> </elems> </message-tail> </context> </test> </boost_serialization> //============================================= I've always wanted to use something like a base-64 string encoding of the bit array, because it's cool and it'd save space. I added conversion functions to/from the bit array and a std::string, and then (de)serialized the string. I also had to separate "serialize" into "save" and "load" since conversion is complementary, not identical. So now I have: //============================================= <?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <!DOCTYPE boost_serialization> <boost_serialization signature="serialization::archive" version="5"> <test class_id="0" tracking_level="0" version="0"> <context class_id="1" tracking_level="0" version="0"> <length>1</length> <buffer-A>1732584193</buffer-A> <buffer-B>4023233417</buffer-B> <buffer-C>2562383102</buffer-C> <buffer-D>271733878</buffer-D> <message-tail>g</message-tail> </context> </test> </boost_serialization> //============================================= Then I added tests for: exactly 6 bits (i.e. one base-64 letter); a sextet (actually two) and a partial sextet together; filling a queue to capacity (actually one short of that since a full queue automatically activates a turnover); and going past capacity resulting in a new hash buffer and an empty message-tail. //============================================= <?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <!DOCTYPE boost_serialization> <boost_serialization signature="serialization::archive" version="5"> <test class_id="0" tracking_level="0" version="0"> <context class_id="1" tracking_level="0" version="0"> <length>1</length> <buffer-A>1732584193</buffer-A> <buffer-B>4023233417</buffer-B> <buffer-C>2562383102</buffer-C> <buffer-D>271733878</buffer-D> <message-tail>g</message-tail> </context> </test> </boost_serialization> <?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <!DOCTYPE boost_serialization> <boost_serialization signature="serialization::archive" version="5"> <test class_id="0" tracking_level="0" version="0"> <context class_id="1" tracking_level="0" version="0"> <length>6</length> <buffer-A>1732584193</buffer-A> <buffer-B>4023233417</buffer-B> <buffer-C>2562383102</buffer-C> <buffer-D>271733878</buffer-D> <message-tail>q</message-tail> </context> </test> </boost_serialization> <?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <!DOCTYPE boost_serialization> <boost_serialization signature="serialization::archive" version="5"> <test class_id="0" tracking_level="0" version="0"> <context class_id="1" tracking_level="0" version="0"> <length>14</length> <buffer-A>1732584193</buffer-A> <buffer-B>4023233417</buffer-B> <buffer-C>2562383102</buffer-C> <buffer-D>271733878</buffer-D> <message-tail>qQg</message-tail> </context> </test> </boost_serialization> <?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <!DOCTYPE boost_serialization> <boost_serialization signature="serialization::archive" version="5"> <test class_id="0" tracking_level="0" version="0"> <context class_id="1" tracking_level="0" version="0"> <length>511</length> <buffer-A>1732584193</buffer-A> <buffer-B>4023233417</buffer-B> <buffer-C>2562383102</buffer-C> <buffer-D>271733878</buffer-D> <message- tail>ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789- _AAAAAAAAAAH__________g</message-tail> </context> </test> </boost_serialization> <?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <!DOCTYPE boost_serialization> <boost_serialization signature="serialization::archive" version="5"> <test class_id="0" tracking_level="0" version="0"> <context class_id="1" tracking_level="0" version="0"> <length>512</length> <buffer-A>2631642121</buffer-A> <buffer-B>80961853</buffer-B> <buffer-C>4033330630</buffer-C> <buffer-D>497373075</buffer-D> <message-tail></message-tail> </context> </test> </boost_serialization> //============================================= If you want to see the actual work, look at revision/change-set #48131 in Boost's Subversion set-up. Now to the actual questions: 1. If there's only one sub-object, base or member, that has any significant data, could someone call the "serialize" member function of that sub-object directly in the wrapping class's "serialize"? (This assumes that friendship is set up.) This would make the wrapping class look identical to the sub-object's class, right? Is this a good idea? 2. Before actually trying to serialize a string, I was worried that the string's serialization would include a length count. This would be unnecessary because the object's "length" attribute already implies the length of the string (int( ceil( double( length % 512 ) / 6.0 ) )). Here, we see that the string's length isn't explicitly included in the XML archive, so I have no worries. But what about non-XML archives? Will be string's length be directly serialized, wasting space? If so, how can I fix that? 3. Having to add std::string to support serialization makes my class header heavier. My class uses fixed-sized arrays, so is there any way that I can avoid allocating a string? For writing out, could I set up a char-array with the encoding and write that out? For reading in, can I read the string in piecemeal to a char-array just in case someone added more characters than required. My converter currently ignores illegal characters and stops when enough legal characters have been read. If what I ask is possible, would the reading routine have to seek to the end of the entry so further serialization isn't messed up? -- Daryle Walker Mac, Internet, and Video Game Junkie darylew AT hotmail DOT com [View Less]

1 0

[1.36.0] Getting Started changes (ticket #1744 plus other fixes)
by Beman Dawes 13 Aug '08

13 Aug '08

Before we can ship 1.36.0, some "Getting Started" documentation changes are needed. These changes are the only thing preventing 1.36 from shipping. Enough said. * Section 1, Get Boost, both Windows and Unix Variants: need to begin with a suggestion that users first check for the availability of platform specific installers, linked to the correct URL. [Rationale: the platform specific installers are the preferred way for users to install Boost if their platform is one of those supported.] … [View More]

6 13

Re: [boost] boost::math::tools and boost::distributions
by Jesse Perla 13 Aug '08

13 Aug '08

> > >Docs here: > > > http://www.boost.org/doc/libs/1_35_0/libs/math/doc/sf_and_dist/html/math_to… > > > >Note that these are considered "experimental" in the sense that their > >interfaces haven't been reviewed for formal inclusion in Boost. They are > >all rigorously tested though as they're fundamental to the implementation > of > >the rest of the library. So... please do use them, and if you find any > >interface SNAFU's that can be … [View More]improved let us know! BTW the most likely > >reason for the interfaces to change is to facilitate improvements to their > >implementation and especially their efficiency. > > > >HTH, John. (sorry John, I sent my last email before getting the digest) Perfect. Since a snapshot of the interfaces exists for 1.35, then a port over to newer versions of boost is reasonable. Hopefully we can turn this into a top level project that does the baseline optimization/root finding people need before shopping for really specific tools/algorithms. -------- A couple of other questions on this beautiful distributions library - which I see as a great starting point for a lot of operations treating them as measures: * Is there any planned or existing support in the library for multi-dimensional distributions? In particular, bivariate norms with a covariance matrix passed in? Will the library design and notation support these sorts of extensions without too many problems? * Is there any way in the current library to pass in a user created discrete valued probability measure and have it as a "distribution" here? If not, I think I may write it, but as I am not a very good library programmer I would love to exploit any other work out there. I would need this to work for at least 1 and 2 dimension measures. * If we were to use a function to bundle a discrete valued function, is there a nice class/pattern out there that already combines the x-values and y-values into a single class? (I also want this later for implementing different interpolation algorithms returning functors after passing in a discrete function in the constructor). * Then I would write a specialization of this (2nd and higher dimension) for a markov measure. * Then, I want to write a general operator for unconditional expectation passing in a function object, and over a generic distribution as a measure from this library (though I will use it for the discrete measures to begin with). Specializing for different types of distributions, this could use exact analytical calculations, quadrature, or use monte-carlo methods. * Last, I want to write a conditional expectation operator that takes in a function object mapping reals^2 to reals, a distribution (would have to be 2 dimensional), the conditional (which would be a real for now, but could be generic), and what dimension the condition applies to. * Eventually, we may be able to tie together this operator with boost::lambda and boost::bind and we could up with a very elegant way to write out functions that involve conditional expectations. It may not have the performance of hand-tweaked fortran loops, but it should be a good enough place to start and infinitely preferable to matlab. The main use case I have for this is to implement easy/flexible notation for conditional distribution functions when writing out bellman equations for dynamic programming in economics. Most cases of this involve two dimensional markov measures (often approximated by Tauchen's algorithm from ARMA processes) If anyone is interested in this topic, has ideas, wants to work jointly, or plans to tell me that I am insane, please email me. My main fear is that I am a library user with a pretty good ability to think in generics, but with absolutely no chance of ever understanding what all of the crazy non-algorithmic code in these libraries does or writing it myself. -Jesse [View Less]

2 1

[TR1] std_run_random failure with VC9 SP1
by Richard Webb 13 Aug '08

13 Aug '08

The std_run_random test is failing on VC9 SP1 (http://tinyurl.com/64xsr5) seemingly because the mersenne_twister implementation doesn't like being seeded with 0: /////////////// void seed(unsigned long _X0 = default_seed) { // set initial values from specified value _RNG_ASSERT(0 < _X0, "invalid argument for mersenne_twister::seed"); /////////////// Is that a bug in SP1, or an incorrect test? Thanks, Richard Webb

4 6

Compiling program_options v1.35 on msvc9.0 fix
by evert.vandewaal＠imtech.nl 13 Aug '08

13 Aug '08

2 1

[1.36.0] Release packaging problems
by Beman Dawes 13 Aug '08

13 Aug '08

Grrr... 1.36.0 is up on SourceForge, but due to a screw-up on my part the .bz2 and .gz files don't include the generated docs. The .7z and .zip files do have the generated docs. The problem was caused by having to run some of the release process on Windows, some on Linux, and then combining the pieces by hand. That's a result of doc changes breaking the doc build on Windows. I'm going to be out of the office all day on non-Boost business, so won't get a chance to work on straightening … [View More]

1 0

Unordered Link on beta.boost.org doesn't work
by Hansi 13 Aug '08

13 Aug '08

Hi, the link under "New Libraries" "Unordered" doesn't work. Best regards Hansjörg

2 1

[trac] Settings link broken
by Maik Beckmann 13 Aug '08

13 Aug '08

Hi, If I click on the word "Settings" in {{{ * Use the >>Settings<< link at the top of this page to let us know your email address even if you don't have a Trac userid. ... }}} I get {{{ Error: Not Found No handler matched request to /trac/boost/settings }}} Regards, -- Maik

2 2

[1.35] thread problem : get_timespec and get_xtime
by Alexander Dong Back Kim 13 Aug '08

13 Aug '08

Dear all, I'm using GCC 3.4 on REHL machine. I used to use BOOST 1.32 until yesterday and upgraded to 1.35 last night and now I can't compile my project =( I'm using boost thread library and it worked really well but doesn't work at all after the upgrading. The error message from g++ is... /usr/local/include/boost-1_35/boost/thread/pthread/timespec.hpp: In function `timespec boost::detail::get_timespec(const boost::system_time&)': /usr/local/include/boost-1_35/boost/thread/pthread/… [View More]

2 5

contradictory information in FS docs?
by Noah Roberts 12 Aug '08

12 Aug '08

"Class basic_directory_iterator is an important component of the library. It provides an input iterator over the contents of a directory, with the value type being class basic_path." That seems to be correct. "The expression itr->path().leaf() == file_name, in the line commented // see below, calls the leaf() function on the path returned by calling the path() function of the directory_entry object pointed to by the iterator." But that claims the object pointed to the iterator is … [View More]

2 2