April 2004 - Boost - lists.preview.boost.org

Re:Re: Re: Re: [boost] Re: [serialization] + [boost.python serialization of python object
by Robert Ramey 20 Apr '04

20 Apr '04

Ralf W. Grosse-Kunstleve wrote: > FWIW: For the serialization of C++ arrays wrapped as Python objects (via > Python's pickle) I implemented a small "library" (it really is just one > header file) for converting integers and floating-point numbers to a > pseudo text format. In principle it works just like the conversion to > base-10 numbers, but uses base-256 instead. I.e. the result looks like a > binary format, but it is as machine-independent as a text format. The > serialized strings are smaller than regular text format, but larger than > raw binary format. > Integers are serialized like this: > NXX...X ... This is interesting. I made an example as part of the library called:demo_portable_binary or which attempted to handle issues such as endianness and size. I bailed at floating point because of the concern for handling things like conversion between 80 bit IEEE float and 64 proprietary floats. And then I was just making as sketch to inspire someone to implement XDR which I presumed had some sort of way of handling this. (I still don't know how other formats handle this) I like your method. I may steal it yet. Robert Ramey

1 0

Re: Re: Re: [boost] Re: [serialization] + [boost.python serialization of python object
by Robert Ramey 20 Apr '04

20 Apr '04

> Ralf W. Grosse-Kunstleve wrote: > Interesting. If I serialize and deserialize std::vector<double> as a text > archive but on the same machine, will I always get back exactly the same > bit patterns for the double values? The text archive uses a stream manipulator to set the precision of the output to capture all the precision in the double. It uses the numeric<limits> to determine this. So I can't say it will be exactly the same bit stream but it will be close to the original number. If you want to guarantee the exact representation you can either use the included native binary archive or serialize the data element (double) as a (non-portable) binary object. Robert Ramey

2 1

Re:Re: [boost] Re: Re: Serialization Formal Review #2
by Robert Ramey 20 Apr '04

20 Apr '04

Matthias Troyer >On Apr 17, 2004, at 8:13 AM, Robert Ramey wrote: >> Matthias Troyer wrote: >>> 2. which functions do I need to implement in my archive class to skip >>> writing the standard preamble that your archives write? Is it just >>> the init function or are there more functions? >> >> Depending on the implementation traits for each class, there may be a >> preamble written by the serialization library. This contains >> information such as version number, class id info, perhaps export key, >> etc required by the s library to reconstruct a faithful reproduction >> of the original data. >> If there is a preamble, there is no information in that can be >> discarded and still reproduce the original data. >I was wondering about the preamble written at the start of the archive, not >the per-class information. An archive preamble is entirely optional. The common_?archive base class includes the init function which all the included archives call in their constructor. There is no requirement to call this function however in the constructors of your own archives. Robert Ramey

1 0

Formal review: serialization
by Vladimir Prus 20 Apr '04

20 Apr '04

Here's my review of the library. It's based on reading docs and source for several hours, as well as using almost every previous version of the library on my project. I think the library should be accepted. I voted "no" the previous time, but I think library was considerably improved. I'd also like to thank Robert for pushing this effort forward! Detailed comments are below. Some of them are marked as "critical" -- it means I believe they should be addressed before the library is added to official distribution. ** General issues ** -- (critical) The primary issue with the library is lack of reference documentation -- exact description of semantics for every method (preferrable in C++ std style) and overview for all important classes. The library is non-trivial, and without such documentation user will have to experiment to find out that exact semantic. Better internal docs (comments) are also needed. For example, I'm looking at save_access::end_preamble and don't know what it does. Comment for basic_oarchive::end_preamble says what this method is for, but does not say when this method is called, which makes it hard to do anything. Again, implementation is non-trivial. Lack of comments make it somewhat dangerous to use the library -- if I hit a bug while Robert is on vacation, I'm not sure I'll be able to quickly fix it locally. As an illustraction, I've already had a couple of bugs each costing several hours to debug. -- (critical) While BOOST_CLASS_EXPORT is a big step forward, I think it can be further improved. Now, if I export a class I need to include some archive headers before invoking BOOST_CLASS_EXPORT. The class can be only serialized to archives which were included. This means that to really export a class, I need to include *all* archive headers. That's not good. What I suggest is - create new class 'polymorphic_archive', which will have virtual functions for saving/loading and create templated 'polymorphic_wrapper', which can be used to add this polymorphic interface to existing archive. - always register all exported classes with polymorphic archive. When serializing in a specific archive, check if class is registered with that archive. If not, serialize it via polymprphic wrapper. In fact, I'd be willing to help with polymorphic archive implementation, but some changes inside serialization library will be required. -- (critical) I suspect BOOST_CLASS_EXPORT does not play nice with templates. Say I'm implementing RPC system: class function_call_base { virtual ~function_call_base() {} } ; template<class Arg> class function_call1 : public function_call_base {}; // in some code rpc_server s; s.register("foo", &foo); function_call_base* c; while(archive >> c) (*c)(); The idea is the caller will serialize instances of function_call1 and the callee will deserialize it and call the proper function. To make this work, we have to BOOST_CLASS_EXPORT all used instantinations of function_call1. We can assume that all functions (and therefore all argument types) are passed to the 'register' function and can function can invoke BOOST_CLASS_EXPORT. Now the problem: BOOST_CLASS_EXPORT can be used only at top-level of a translation unit. There's not way 'register' and automatically export a class. The solution I'd like is: template<class T> class export_class { public: static void instantiate(); }; So that the object of this type can be created anywhere. -- One usage of the serialization library I have in mind is taking "debug dumps" for an application. After the run, a separate tool can be used to look at those dumps. The size of data can potentially be rather large (e.g. 100*640*480 or maybe more), so I'd rather not to load all data, but build some index. So the question is: how is it possible to access specific part of archive without reading it all. I understand it's not possible in general, but in some cases (e.g. BOOST_CLASS_EXPORT for everything) it should work. -- The documentation of 'make_binary_object' is very incomplete and does not tell the exact semantic. As far as I can tell, the function can be only used to load/store objects which have fixed size and will crash when saving/loading new[]-d array -- which seemed an obvious use-case for me. I would suggest that we have two functions: make_pod_wrapper -- which is templated, has one parameter -- object, and stores sizeof(T) bytes. Another name -- make_static_array_wrapper. make_dynarray_wrapper -- which can be used to store/load arrays allocated by new[] -- The library requires exact match between static types for saving and loading. For example: D* d = new D; ia << d; B* b = new B; ia >> b; does not work. Moreover, it does not work even if 'D' is BOOST_CLASS_EXPORT-ed. I'd suggest that this case should work, since it's very easy to save derived classes using pointer to derived class, without casting to base class. -- It appears that XML archive require member/variable names in all cases. I'm not sure that's reasonable -- given that the library requires exact correspondence between loading and saving order even for XML archives. It would load archive back even if I change all the names manually in XML file right? Then why requiring names? -- I have some concerns about XML archives. The point of XML, in general is that it's somewhat readable by humans, and readable by other applications written using different libraries or in different languages. However, when I look at: <c2 class_id_reference="1" object_id="_2"> <i>10</i> <j>4</j> <base object_id="_3"></base> </c2> it's not clear if any of those benefits are there. This actually corresponds to a class "C". To find out this, one has to find "class_id" attribute with value of "1" somewhere above. If the above was: <C name="c" ...> finding all "C" elements would be trivial with XPath, or even with more simple methods. As it stands now, even finding all instances of "C" is hard. There may be a number of similiar issues. If the goal is to produce XML which can be processed by other programs, then at least the class name should become element name. -- I believe that library should not change codecvt facet of the stream. There are two reasons. First, if user has already imbued some facet (say utf-8), he might have good reasons to do so (say, he believes utf-8 is the most space-efficient for his data). So the serializaton library should not silently change codecvt to ucs-2. Second, the whole locale stuff is complex, and it's not even clear if it's possible to change codecvt facet on the fly, and when the change will take effect, and so on. I believe requiring user to 'imbue' proper facet himself is better than some unexpected internal effects. -- I've tried to play with "implementation level" and created the example at http://zigzag.cs.msu.su:7813/tracking.cpp It stores a pointer to a single class with two data members and a single base class. I get the following archive content. 1 1 C 0 10 4 1 0 1 Is seems too much for a class with implementation level of "object serializable". 10 and 4 is the data itself. "1 C" is probably class name, but what other data is? ** Implementation issues ** -- I suggest that all MPL expressions in the library are refactored for better readability. For example, the expression in save_non_pointer_type::invoke is 28 lines long. I think it should be possible to extract some of the subexressions and create a typedef with good name -- looks like instantination of those expression in all cases should not cause compile errors. ** Custom archive creation ** I've tried to create a custom archive class (in fact, polymorphic_oarchive), and got some comments. - Documentation says the derived class must provide operator<< and method 'load'. It also says there common_iarchive::operator<<. I wonder why the derived class should provide operator<<. Can't base operator call 'load' directly? - Even if not, why derived operator<< should have return * This(); not plain "return *this"? After all, we're in most derived object - It's needed to add friend class boost::serialization::load_access; to class declaration. However, it's not stated where this type is declared, and in fact in declared in *internal* header boost/archive/detail/iserializer.hpp And, finally, the right syntax seems to be: friend class boost::archive::load_access; (Note the namespace!) - The example uses: boost::serialization::load(* This(), t); Again, what header is this function in? The answer is that it's in the same header as above, and the line should be: boost::archive::load(* This(), t); - The documentation should really state the minimal set of 'load'/'save' overloads which will make archive usable. For example, it's probably not necessary to provide separate overload for 'bool', right? - (critical) It should be explained how the newly-created class can be tested. As it stands, one should either do superficial tests or try to integrate the new archive into existing tests. As for latter -- I don't know how to do that, and also, I think it's not necessary to run all the tests with the new archive. For example, serialization of vector and map is not likely to be broken with the new type. What I'm looking for is a test which saves/loads all possible types and verifies results. ** Various comments ** -- The BOOST_CLASS_EXPORT macros defines an explicit instantiation of a function. gcc rejests this when BOOST_CLASS_EXPORT is used inside some namespace. I've asked on comp.std.c++: http://groups.google.com/groups?q=Explicit+instantiation&hl=en&lr=&ie=UTF-8… 40zigzag.lvk.cs.msu.su&rnum=3 and got a reply directing me to: http://std.dkuug.dk/jtc1/sc22/wg21/docs/cwg_defects.html#275 Which means gcc is mostly right. So, the macro should either be changed somehow, or at least the restriction should bes documented -- using BOOST_CLASS_EXPORT outside of any namespaces is OK. -- Docs and code use "const unsigned long int version" everywhere. Is that really necessary, as opposed to "unsigned version"? -- The library saves char* as arrays of integers. The rationale section says: input of char * and wchar_t * could be altered to create a buffer overrun and potential security problem. I don't understand this point at all. How storing the elements as integer values helps with the buffer overrun? Moreover, library stores char* internally (e.g. for archive signature). What if that string is modified? Will that cause buffer overrun? -- The following program (yes, of two lines), causes a compile error: #include <boost/archive/text_oarchive.hpp> #include <boost/serialization/export.hpp> It's because export.hpp uses 'instantiate_pointer_iserializer' but does not include the header where that class is defined. 'text_oarchive.hpp' does not include it either. The error goes away if I include #include <boost/archive/text_iarchive.hpp> but a better diagnostic is desirable. - Volodya

1 0

RE: [boost] Re: Date-Time
by Powell, Gary 20 Apr '04

20 Apr '04

Rob Stewart wrote: >From: David Abrahams <dave(a)boost-consulting.com> >> "Jeff Garland" <jeff(a)crystalclearsoftware.com> writes: >> > On Wed, 14 Apr 2004 13:04:34 +0100, Val Samko wrote >> >> >> >> Does that mean that add_month won't be implemented? What if we call >> >> it inc_month, instead of add_month? inc_month (Increase Month) >> >> in no way assumes that the day of the month will stay the same, >> >> it will just increase the month. >> > >> > Yes, perhaps a different name would help eliminate any confusion that we are >> > doing 'math' here. >> >> I think readability is more important. Nobody but pedants have >> problems with string + string. Likewise I think: >> >> d + 2*months >> >> or whatever is better than the functional-looking alternatives. >If there was universal acceptance of what it means to add two >months to a date, I might agree. Given the imprecise notion of >what it means to add two months, however, I disagree. >"next_month" or "increment_month" versus "add_month" better >distinguish the imprecision. Here I totally agree with Rob, adding months is a non-intuitive operation and calling out the precision helps the readability of the program. It makes perfect sense to do d + (30 * days) but not with Months. Yours, -Gary-

6 17

Re: [thread] again
by Christophe Meessen 20 Apr '04

20 Apr '04

Hello, regarding thread local storage we have come up with a different perspective of the problem. First we where not very happy with boost threads because it requires a copy of the functor object. This does not allow to put non copyable object as member variables of the functor object. Knowing that mutex are none copyable objects, for good reasons, one understand that this API is far too constraining. I guess such design choices was driven by the requirement that the functor object lifetime is tighlty linked to the thread lifetime. The join function is also too limited synchronization mechanism. And finally we where also not happy with the thread local storage approach. We seeked a design which was simple and could provide a satisfying solution for all these aspects and we came up with the following design. We chose the start&run model used in many thread API because it allows to initialize the thread object before it is started and collect result after the thread terminated. The thread object is the obvious place to use as thread local storage. This is because stored data is explicitely defined and its use is statically checked. It is also automatically destroyed with the thread object is destroyed. Thread objects are dynamically allocated and referenced through intrusive pointer. Thus thread objects can have none copyable member variables and getting out of scope doesn't destroy the thread object. The thread object life time is controlled through an intrusive smart pointer. When the internal thread main function start it instantiate a smart pointer on itself as local variable. Then it calls the virtual run by changing the 'this' context to the objects context. In the run method users have access to the thread object member variables and methods. When the thread main function terminate normally or because of an exception, the smart pointer is destroyed. If there is no other reference to the thread object, it is then deleted and all its local storage information with it. In the thread local storage managed by the system/pthread we only store a pointer to the thread object. This is to allow to get access to the thread object from anywhere in the code. This is equivalent of the this variable for objects but for threads. Thus the thread class has a static self() method returning a reference on the current thread object. For dynamically storage, one can add a map to the thread object. The user can pick the key and value types which are the most appropriate for his needs. The map will be automatically destroyed when the thread object is destroyed. A default thread object will be provided to support classical C function or functor calls. But the basic underlying model is to use a thread object. It also to be noted that boost thread implementation would start the thread before the boost thread object instantiation would complete. Such constrains should not be imposed to the users. There is thus a need to know when a thread has started and when it has terminated. When the start call completes, it doesn't mean the thread has already started. This is reflected by a state variable that would also specify if the thread terminated normally or was aborted because of an exception or cancelled (which should also use the exception model). Regarding synhcronization we have also made a slight move away from boost API which was aslo the best seen so far. We make a distinction between locks and latch. A lock has no method. When it is constructed it immediately attempts to lock the mutex, and the destructor release the mutex. Thus when a lock is instantiated, the mutex can be assumed to be locked by the thread who performed the instantiation on its call stack. When a latch is instantiated, the mutex is not locked. The following methods lock, unlock, try_lock and timed_lock are used to control the referenced mutex. When a latch variable is destroyed it automatically unlock the mutex if it was currently locking it. We believe there was a possible confusion in the initial lock state of boost locks. It is not orthogonal. We also beleived there was a performance penalty by not providing atomic lock object. The lock and latch are encapsulated classes of mutexes. And we would by default support two kinds of mutexes, classical pthread mutex and spin lock mutexes. Spin lock mutexes would support locks and not latch locks. spin locks would be used to serialize access to shared resources like smart pointer counters, monitoring variables etc. We also droped the "scoped_" in front of the lock class name because it is implicit, standard and shortens the name. -- Bien cordialement, Ch. Meessen

4 3

RE: [boost] date-generators (was Date-Time)
by Hurd, Matthew 20 Apr '04

20 Apr '04

> On Behalf Of Jeff Garland > Subject: Re: [boost] date-generators (was Date-Time) > > On Mon, 19 Apr 2004 14:22:53 -0400 (EDT), Rob Stewart wrote > > > > That does make sense, but so does the "date wrapper" idea. The > > latter provides a convenient means to use operator overloading > > meaningfully while making the behavior of those operators > > specific to the wrapper type. I think the date wrapper approach > > will prove easier to use, but more cumbersome to implement. > > I have no idea what the 'date wrapper approach' is... > > Jeff Some further quick thoughts from my dark past... Most places I've been, typically finance related, have had dates, date expressions, date generators and calendars as concepts. Date expressions give you the concept of the last day of the month. They can give you the third Wednesday of the month. I'm used to just strings like: 3wed+2b (3rd Wednesday + 2 business days), -1Sun+1b (Last Sunday + next business day), -1d (last day of the month). Strikes me that date expressions would look nice as expression templates. Extend this with date generators that can generate such dates for month patterns (every 3 months, explict months) within the context of a calendar. Calendars with holidays and being able to combine them for transactions is important. You can then refer to the second business day after the third Wednesday and such things. A typical use of a calendar is to have calendars for business jurisdictions and be able to union them. The usual implementation I've used is a bit field of days of the year for however many years, though a holiday list and look up might make more sense. A default "Western" calendar only has Sat,Sun as non-business days. Date expressions can get interesting when you're trying to represent various business rules, e.g. -Sun+2b unless it is less than 3 days before the end of the month then it is -Sun-1b, which can lead you down the path of extending the syntax to include comparisons, conditional representation and the notion of "instance". E.g (-sun+2b) <= -3d ? (-sun+2b) : (-sun-1b). Now that's fun :-) Once place I worked had easter, the lunar calendar and the like to do some further unusual stuff. Adding something to a date could be a date expression: 1. date_add( date(2004,2,29), "+3M : -d") // 31 May 2004 2. date_add( date(2004,2,29), "+1M : -0b") // 29th Mar if a business day, previous b day if not one Hmmm, I forget how that bit used to work, but you get the idea... Generate a date or dates from some specification, the last day of the month is a simple specification. $0.005 Matt Hurd. IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

3 3

Re: [boost] Re: [thread] again
by Christophe Meessen 20 Apr '04

20 Apr '04

reHello, Unfortunately I don't have the time to reverse ingneer the boost_dev thread code. So maybye Michael Glassford can help me and answer the following questions. The most fundamental problem I found in boost thread is the functor copy. So is this still present in the new thread API ? Will the start&run thread API model be available in the new boost thread package ? If not, why ? -- Bien cordialement, Ch. Meessen

2 1

Thoughts on Unicode and Strings
by Reece Dunn 20 Apr '04

20 Apr '04

Here are my thoughts on Unicode strings, based partially on the current discussions of the topic. As I understand it, the problem with strings (standard character and Unicode strings) can be broken down into several stages: [1] Storage And Representation This is how the underlying string is stored (allocation and memory mapping policy) and how it is represented (which is governed by locale, but at this stage, it is best to know the type: UTF-8, UTF-16, UTF-32, etc.) The storage can easily be represented as a container type, and so we have: template < typename CharT, template< typename T, class A > class Container = std::vector, class AllocT = std::allocator< CharT > > class string_storage: public Container< CharT, AllocT > { }; Here, I have chosen std::vector as the standard storage policy, as this reflects the current storage policies; thus, basic_string< CharT, Traits > would therefore be based on string_storage< CharT >. It would be easy, then, to select other representations like a reference counted storage (a variant of std::auto_ptr< std::vector >) and even an SGI-like rope! (Although, this would mean that a new std::roped_vector class would need to be implemented: does such a thing already exist in Boost?) [2] Basic Type And Iteration The basic representation is more complex, because now we are dealing with character boundaries (when dealing with UTF-8 and UTF-16 views of a Unicode string). At this stage, combining characters and marks should not be concerned with, only complete characters. The Unicode string should provide at least 3 types of iterator, regardless of the internal representation (NOTE: as such, they will be implementation dependant on how the string is represented): * UTF-8 -- provides access to the UTF-8 representation of the string; * UTF-16 -- provides access to the UTF-16 representation of the string; * UTF-32 -- provides access to the Unicode character type. Therefore, no matter what the representation, it should be possible to use the UTF-32 iterator variant and "see" the string in native Unicode; this should, therefore, be the standard iterator and the others should be used when converting between formats. NOTE: I am not well versed in how Unicode is represented, so I do not know how feasible it is to implement backwards traversal, but I do know that it would probably be wise to know the position of the last good end of a Unicode character (e.g. when dealing with multi-character UTF-8 and UTF-16 representations). As a side note, it should be feasible to provide specialist wrappers around existing Unicode libraries (like Win32 (CharNext, etc.), ICU and libiconv?), so I would suggest having something akin to char_traits in basic_string. RATIONALE: Why standardize on the UTF-32 view? Because UTF-8 and UTF-16 are multi-character encodings of UTF-32 (not considering combining marks at this stage), whereas UTF-32 is a single character encoding. [3] Algorithms, Locales, etc. These are build upon the UTF-32 view of the Unicode string, like the string algorithms in the Boost library. Therefore, instead of str.find( unicode_string( "World" )), you would have find( str, unicode_string( "World" )). I would also suggest that there be another iterator that operates on std::pair< unicode_string::iterator, unicode_string::iterator > to group combining marks, etc. Thus, there would also be a function unicode_string::utf32_t combine ( std::pair< unicode_string::iterator, unicode_string::iterator > & ucr ) that will map the range into a single code point. You could therefore have a combined_iterator that will provide access to the sequence of combined characters. NOTE: If ucr.first == ucr.second, then combine( ucr ) = *( ucr.first ). Regards, Reece _________________________________________________________________ Express yourself with cool new emoticons http://www.msn.co.uk/specials/myemo

5 7

RE: [boost] date-generators (was Date-Time)
by Hurd, Matthew 20 Apr '04

20 Apr '04

> Jeff Garland > > On Mon, 19 Apr 2004 19:14:12 -0400, Hurd, Matthew wrote > > Strikes me > > that date expressions would look nice as expression templates. > > Sure but the real requirement is that they be strings because > these things get > read in from databases and files in real apps -- they don't tend to be > hardcoded in code. Yep. A little spirit grammar would do the job though a little slowly. Once we can have strings for compile time template programming and a suitable parser things might be different. The distinction will blurr. I remember some banter quite a while a go about using char arrays for meta programming. Would it be possible to use boost pp to process a convenient macro string into a char array form for further meta-processing? In this vein, are string and floating point types as template parameter manipulation thingos (thingos: ISO C+ 5.2.1.3*) up for inclusion in C++0X? > Which is why I'm starting to see all these logical constructs > as just another > form of date generation type that has a set of rules and > combines 2 elements > (a date and month count in this case) to get another date. I guess: single_date_generator :: date x adjustment x result spec -> date where adjustment is the month count. Regards, Matt Hurd. ------------------------------- *don't look that up, I'm joking IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

1 0