April 2004 - Boost - lists.preview.boost.org

Re:Re: Re:[boost] Re: Serialization Formal Review #2
by Robert Ramey 26 Apr '04

26 Apr '04

Peter Dimov wrote: > FWIW, in my own library I do, indeed, have a special case code for > serializing an unmodified shared_ptr, for two reasons. First, this shows > that the library is flexible enough. Second, even if we modify > boost::shared_ptr to be boost::serialization-friendly, we won't be able to > patch std::tr1::shared_ptr. The main problem I had was that certain member variable weren't accessible. The STL collections don't have that issue so there was no problem. > However, should you decide to use your current scheme, I'll happily insert > whatever friend declarations are necessary in shared_count. > I'm away from my computer right now, so I can't paste my read/write > functions for shared_ptr, but the general idea is that they keep a > map< shared_ptr<void>, int > on writing, mapping every shared_ptr > ownership group to a pointer id, and a corresponding map< int, > shared_ptr<void> > on reading, for the reverse transformation. > The main problem for implementing a non-intrusive serialize() for > shared_ptr is that these pointer maps need to be held in the archive, > which requires either archive modification, or a general "extra state" > support in all archives. That's pretty much the same as what the serialization system does - albeit in perhaps in a more generic way. Its not clear to me how enhanced library code could be made to work work without granting access to shared_count. Of course, if access is granted to shared_count, then everything works already. Robert Ramey

1 0

ReL[boost] Re: Serialization Formal Review #2
by Robert Ramey 26 Apr '04

26 Apr '04

David Abrahams wrote: >"Peter Dimov" <pdimov(a)mmltd.net> writes: >>> http://www.boost.org/libs/smart_ptr/intrusive_ptr.html#Introduction >>> >>> However, the situation is different there as the called function >>> only accepts exactly one parameter. >> >> It doesn't matter. ADL should find the overload, provided that >> 'serialize' >> is not qualified at the point of the call. > >> In my own library I use very similar read/write overloads that are, >> indeed, placed in the namespace of the corresponding class and found by >> ADL. Yes, it works with VC 7.1. >The recommended "overload in the library namespace" procedure was a >big concern for me, but I didn't post anything because I couldn't >remember why it's an issue. I have in my mind, left over from my work >with Peter on this problem, a rule that in the current language you >shouldn't do that, and should rely instead on ADL. I don't know why >I think that anymore, but I *can* say that Peter should be listened >to in this area; he's more aware of the issues than almost anyone. OK, I've looked at code and found that found that my calls to "serialize" are qualified by boost::serialization. So this can be fixed. It seems that users of older compilers will have to overload in the library namespace but that it will not be necessary for users that are only concerned with newer compilers. So its not a problem for the library. Except... What about when I've overloaded from with in the library - such as when default serialization traits are overloaded. Should this namespace specification be #ifdef/#endif depending on the compiler config? Should this be dependent on BOOST_NO_ARGUMENT_DEPENDENT_LOOKUP? Robert Ramey

1 0

Re:[boost] Re: Serialization Formal Review #2
by Robert Ramey 26 Apr '04

26 Apr '04

Andreas Huber wrote: > Ok, but IIUC this undermines the original promise that a class > implementing serialize or safe/load can be serialized to *any* archive, > even future ones without changing its implementation. I see that I > wouldn't need to change the serialization *code* but I'd need to add > an include to the .cpp file, right? Correct. But I don't see that as undermining the original promise. You will have to re-compile for future archives, but code will not have to be changed. The polymorphic archive (in progress) will lift even that restriction. > > >> 9. The library registers all kinds of stuff with static instances. > >> This precludes its use outside main(). > > > Hmmm - that would be news to me. > Your library cannot be used to save/load anything before main() is entered > nor can it be used to save/load anything after main() has been left. Correct. It never occurred to me to mention this. Robert Ramey

1 0

Re:[boost] Re: Formal review: serialization
by Robert Ramey 26 Apr '04

26 Apr '04

Jeff F wrote: > template<class Archive> > inline void serialize( > Archive& ar > , boost::filesystem::path& t > , unsigned int version ) > { > std::string lPathString; > > if( is_saving(ar) ) > lPathString = t.native_file_string(); > > ar & make_nvp( "PathString", lPathString ); > > if( is_loading(ar) ) > t = boost::filesystem::path( lPathString, > boost::filesystem::native ); > } The "real" solution here is to implement serialization for "native_file_string() So that things look like template<class Archive> void serialize( Archive &ar, boost::filesystem::native_files_sting_type & t, unsigned int version ){ ... ?; } template<class Archive> void serialize( Archive &ar, boost::filesystem::path & t, unsigned int version ){ ar & make_nvp("PathString", t); } Robert Ramey

1 0

Re:[boost] Re:Re:Re:Formal review: serialization
by Robert Ramey 26 Apr '04

26 Apr '04

Vladimir Prus wrote: >Hmm... gotta look into this. What about my second question: >>> 2. For random access I need to make sure that all saved objects have >>> export key. How do I do that? Not necessary out-of-the box but where >>> I can plug the check? Actually, its not necessary that ALL saved objects have export keys. Only those objects that are serialized through polymorphic pointers need to be exported or registered. An attempt to serialize a polymorphic pointer that has neither been exported nor explicitly registered will trigger a run-time exception "unregistered_class". If you wanted to enforce the idea that all types should be exported I believe it would be possible to something like the following in your archive class: a) override implementation of save(class_id_type & t) and save(class_id_optional_type) so that they set a flag. b) override implementation of save(class_name & t) it sets a flag c) override end_preamble so that it checks for an undesired set of flags. I never considered this before so you'll have to check the above yourself. >>> one for dynamic arrays with element-wise save >> >> I would envision one using >> >> ar << element_count >> For(i = 0; I < element_count; i++) >> ar << element[i] >> >> I'm not sure that its worth adding and documenting such an obvious and >> trivial thing as a separate wrapper. >But on loading you need to add 'new[]' call. So load and save become >non-symmetric, and you need to split serialize, which is rather >inconvenient. Actually, I forget the original question. But in any case, I'm wondering about trying to anticipate every application. OTOH, having considered again dynamic_array I'm now more inclined to include it. I'll take a look a the other ones. These would be described in the document under "serialization wrappers". > > There are cases when an object id is > > assigned but not written to the archive I don't see these as having much > > utility outside of serialization. > So, it's not easily possible to plug a different algorithm? In a general sense - NO. Any of the "bookkeeping" types in common_?archive can be overloaded to taste. Overloading these in a special way was how XML was implemented. I'm sure that these could be used for creating other special purpose archives but I really can't know for any particular case until one tries to implement them. >>> Ah, I've missed that. Do I need to provide both 'type' and 'value'? >>> Can't serialization library work with just one? > >> Could be. I just provided both so that I could interoperate with mpl >> without having to think about each specific case. > But for user this can be inconvenient. Actually I was thinking most users would use the macros. If one fails to include one that's used it should fail compilation so I don't see it's a show stopper. Actually I think it's easier just to always include both rather than trying to document/remember which ones use which. > It it possible that you make BOOST_EXPORT always > register classes with the polymoprhic archive. > and use polymoprhic archive as fallback for serializing classes? I'm still fine tuning this (smart_cast anyone?). But basically things work like this. a) Just as now, one could pre-compile a serialization instantiating code for particular archives. In this case one of the archives would be the "polymorphic_?archive" class which is nothing but a bunch of virtual function declarations. b) when the application is built, a specific archive implemenatation is selected. E.G polymorphic_text_?archive. The is archive inherits the implementation from text_?archive and the interface from polymorphic_?archive. c) the application will link with the pre-compiled classes. I'm not sure this answers your question - but there it is. Thank god no ones asked about DLL implementations yet. Robert Ramey

1 0

Re:[boost] Re: Serialization Formal Review #2
by Robert Ramey 26 Apr '04

26 Apr '04

Message: 11 Andreas Huber wrote: > 1. On compilers supporting ADL, it seems the user-supplied serialization > function for non-intrusive serialization > template<class Archive> > void serialize(Archive & ar, gps_position & g, const unsigned int > version); > could also be put into the namespace of the type that it serializes. If > this is correct it should be mentioned in the tutorial & reference. If > not, there should be a rationale why not. I did try this in my environment VC 7.1 and it failed to work. Does anyone want to assert that it should or shouldn't work? > 2. save/load naming: All functions that have to do with reading from an archive are named load*, > all functions that have to do with writing to an archive are named save*. > To me (non-native speaker) these names are generally associated with > persistence only. Since the library is more general (one could use it for > network communication, etc.) load/save are a bit missleading. I'd rather > see read/write or other names that more clearly communicate what the > functions are doing. I used save/load consistently specifically to distinguish from read/write which seemed to me too suggestive of persistence, files and streams. > 3. archive naming: > The word archive also seems to be associated with persistence. I think > serializer would much better describe the functionality. Another option I > could live with is serialization_stream (I slightly disagree with the > assertion in archives.html that an archive as modelled by this library is > not a stream. It might not be what C++ folks traditionally understand a > stream is but to me it definitely is a more general stream, i.e. a stream > of objects/things/whatever). Archive is perhaps a little too suggestive. I borrowed from MFC. Making changes would have a huge ripple effect through code, file names, namepace names and documentation. I don't find any proposals for alternate naming sufficiently compelling to justify this. > 4. Object tracking 1: Seeming contradiction > I've had difficulty determining how exactly object tracking works. I > believe the associated docs are slightly contradictory: > - Under exceptions.html#pointer_conflict I read that the pointer_conflict > exception is thrown when we save a class member through a pointer first > and then "normally" (i.e. through a reference). Correct. > - traits.html#tracking says "... That is[,] addresses of serialized > objects are tracked if and only if an object of the same type is anywhere > in the program serialized through a pointer." Correct. > The second sentence seems to contradict the first one in that it does not > specify that the sequence of saving is important I see no conflict here > (I assume that pointer_conflict exception is not thrown if we save > normally first and then through a pointer). Correct > Moreover, I don't see how the library could possibly > achieve the "anywhere in the program" bit. It is tricky and it does work. I works by instantitiating a static variable. Static variables are constructed before main() is invoked. These static variables register themselves in global table when constructed. So even though a template is used "anywhere in the program" the entry in the global table exists before execution starts. I'm reluctant to include such information in the manual. > 5. Object tracking 2: It should not be possible to serialize pointers > referencing objects of types that have track_never tracking type. LOL - tell that to library users. There are lots of things that users are convinced they have to do that I don't think are a good idea. As a library writer I'm not in position to enforce my ideas how users should use the library. On this particular point I would be very suspicious about such a usage but as practical matter I just have to include a warning and move on. special.html#objecttracking says: <quote> By definition, data types designated primitive by Implementation Level class serialization trait are never tracked. If it is desired to track a shared primitive object through a pointer (e.g. a long used as a reference count), It should be wrapped in a class/struct so that it is an identifiable type. The alternative of changing the implementation level of a long would affect all longs serialized in the whole program - probably not what one would intend. </quote> > I think serializing untracked pointers almost never makes sense. If it > does anyway the pointed-to object can always be saved by dereferencing the > pointer. Such an error should probably be signalled with an exception. I don't understand this. > 6. Object tracking 3: One should be able to activate tracking on the fly > Rarely it makes sense to save even primitive types in tracked mode (e.g. > the example quoted in 6.). That's why I think setting the tracking type > per type is not flexible enough. It should be possible to override the > default somehow. I could imagine to save such pointers as follows: > class T > { > ... > int i; > int * pI; // always points to i > }; > template<class Archive > > void T::save(Archive &ar) const > { > ar << track( i ); // save the int here, keep track of the pointer > ar << track( pI ); // only save a reference to the int > } > Since it is an error to save an untracked type by pointer even the second > line needs the track( ) call. A main function of using a trait is to avoid instantiation of code that won't ever be used. If I understand your proposal we would have to instantiate tracking code even if it isn't used. But my main objection thing about this proposal is that it introduces a runtime modality into the type traits. This sound like it would very hard to debug and verify. I see lots of difficulties and very little advantage to trying to add this. > 7. I think the enum tracking_type should rather be named tracking_mode as > the _type suffix suggests something else than an enum. I don't think the change is sufficiently compelling to justify the hassle associated with this. > 8. I don't understand how function instantiation as described under > special.html#instantiation works. The description seems to indicate that > all possible archive classes need to be included in the files of all types > that are exported. Is that true? True. All archives that might be used have to be included. This is an artifact of using templates rather than virtual functions. Templates regenerate the code for each combination of serializable type and archive. This results in faster code - but more of it. There were strong objections raised in the first review about the fact it used virtual functions rather then templates - so it was changed to templates here. > 9. The library registers all kinds of stuff with static instances. This > precludes its use outside main(). Hmmm - that would be news to me. Robert Ramey

6 7

Re: [boost] [serialization] not so formal review
by Robert Ramey 26 Apr '04

26 Apr '04

Gennadiy Rozental wrote > Unfortunately I was not able to look with proper attention onto your > submission during review. One thing I did though: all the major issues I > mentioned in my original review looks resolved. I am sure will look into > more details later, but for now it is enough for me to cast YES vote. > This library cover very important problem domain in software development > and I think we all have now solid solution. Thanks and congratulation to > Robert for being able to bring it to us. I was waiting for a comment from you. Your incredibly insightful, exhaustive and focused review of the previous version was the main driving force behind this version. Looking back, it's still hard to for me to believe that in the six hours you said you spent anyone could really understand the implementation to the depth you did. Basically, all I did was to implement all your suggestions one at a time. Early on I switched over to your test library. This permitted me to add features one at a time and test at each stage. It did result in a lot of code being rewritten several times. But I think in system like this one can't really look far enough into the future implementation to avoid this. So it's not too much to say that if this library lives up to the hopes and expectations I have for it, it will be due to a collision between your previous review and my personality flaw that inspires me to throw myself at lost causes. I was going to suggest you might separate some time to review it. On the other hand, maybe I should quite while I'm ahead. Robert Ramey

1 0

Formal review: serialization
by Jeff Flinn 26 Apr '04

26 Apr '04

I was not involved in the original review. > Do you think the library should be accepted as a Boost library? Yes. > What is your evaluation of the design? The design, from a usage standpoint is excellent. Coming from the MFC world, boost::serialization far surpasses MFC's in usability and flexibility. Particularly notable are: no need for a common base class, a single serialize method used both for input/output. The only problem I see is that programming errors in some cases can be difficult track down, but no more so than with any template intensive library. While the current methods for Free Functions and Splitting save/load methods are certainly understandable and work. It would be nice to be able (partial)specialize the "serialize" function base on its IO type. It would also be nice to utilize enable_if to generate different code when the serialize function is instantiated for input/output archives. This would allow compile time constructs similar to mfc's runtime Ar.IsLoad(). > What is your evaluation of the implementation? I've been using the serialization library since version 13. All previous version bugs/issues have been cleared up with version 18. I've been able to remove all workarounds from my code, in particular dealing with support of multiple archive types in a single translation unit including binary, text and xml. Performance is more than adequate for my purposes. My application uses binary serialization to implement drag/drop. On a 3 year old 2GHz pc, a pair of save/loads of a ~2MB including graphical updates takes less than 2 seconds. I have only briefly viewed library source code when finding workarounds in previous versions. Others here are much more qualified to render a verdict in this regard. > What is your evaluation of the documentation? The documentation allowed me to add serialization to my application without needing to look at header/implementation files. Though, I'm sure I carry some context from having worked with MFC serialization. One misleading aspect IMO is with the Non-Intrusive Version example. The example implies that you would need access to the data members. I think an example of what to do when you don't have access to the data members is needed. For example, how would one serialize a boost::filesystem::path object. namespace boost { namespace serialization { template<class Archive> void save( Archive & ar, const boost::filesystem::path & t, unsigned int version ) { std::string lPathString( t.native_file_string() ); ar << make_nvp( "PathString", lPathString ); } template<class Archive> void load( Archive & ar, boost::filesystem::path & t, unsigned int version ) { std::string lPathString; ar >> make_nvp( "PathString", lPathString ); t = boost::filesystem::path( lPathString, boost::filesystem::native ); } } } BOOST_SERIALIZATION_SPLIT_FREE( boost::filesystem::path ) I'm sure this could be improved upon, but without it, there is a danger that many will think - "can't be used for my application". Seeing an examples like this makes it a little easier to remember that the template functions need to be defined in the boost::serialization namespace. > What is your evaluation of the potential usefulness of the library? I've always considered serialization to be a major element missing from C++. Nearly every application uses some method of custom serialization today. I've saved 100's of hours already in using boost::serialization rather than spinning custom file formats or implementing connections to DBMS systems to store/retrieve data. > Did you try to use the library? With what compiler? Did you have any problems? Yes, with VC71. As stated above, all problems with versions previous to version 18 have been corrected. One, recent problem is an access violation when loading a saved shared_ptr(NULL). > How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? Applied in an actual development project over the last several months. > Are you knowledgeable about the problem domain? I've some experience with MFC serialization. Thanks for a great library! ----------------- Jeff Flinn Applied Dynamics, International

2 3

[Boost-bugs] [ boost-Support Requests-942349 ] Problem compiling :/
by SourceForge.net 26 Apr '04

26 Apr '04

Support Requests item #942349, was opened at 2004-04-26 07:19 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=207586&aid=942349&group_i… Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Nobody/Anonymous (nobody) Assigned to: Nobody/Anonymous (nobody) Summary: Problem compiling :/ Initial Comment: Hi :) I've some problems compiling code that use \boost\iterator\transform_iterator.hpp (under VC++ 7 - VS2K2) : transform_iterator.hpp(45) : error C2825 : 'UnaryFunction::result_type': cannot form a qualified name transform_iterator.hpp(45) : error C2039: 'result_type' : is not a member of 'operator``global namespace''' transform_iterator.hpp(45) : fatal error C1507 : previous user errors and subsequent error recovery halt further compilation Thanks in advance :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=207586&aid=942349&group_i… ------------------------------------------------------- This SF.net email is sponsored by: The Robotic Monkeys at ThinkGeek For a limited time only, get FREE Ground shipping on all orders of $35 or more. Hurry up and shop folks, this offer expires April 30th! http://www.thinkgeek.com/freeshipping/?cpg=12297 _______________________________________________ Boost-bugs mailing list Boost-bugs(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/boost-bugs

1 0

Re:[boost] Re:Re:Formal review: serialization
by Robert Ramey 26 Apr '04

26 Apr '04

Vladimir Prus wrote: > > Robert Ramey wrote: > > I guess I have two questions: > 1. Won't serialization fail in some way if I just seek the stream to the > position found in index and try reading. > 2. For random access I need to make sure that all saved objects have > export > key. How do I do that? Not necessary out-of-the box but where I can plug > the check? > I believe the original question referred the ability to "browse" the archive for purposes of debugging etc. My view would be that the best way to do this would be to create and xml_archve. Your use case suggested a 30 data would probably would end up as a 90MB XML datafile. The reason for thinking in XML would be that I would hope that here exists "XML Browser for large files" which would be suitable for this case. If such a thing doesn't exist - I would prefer spending time making this general purpose tool - (and maybe a commercial product) to making a custom archive of narrow purpose. Random access into an archive would require some thought. First, it's not clear how it would be used. In general archives are nested structures, one could de-serialize an inner piece - but to where? The original data structure was embedded in something else which is not now there. So this can really only consider in the context of a specific application. Here are a couple of plausible scenarios. a) A general purpose archive browser - this could browse the archive in a random manner but wouldn't actually de-serialize the data. I don't see any real problem here. One would need to create an index either as a side effect of archive create or with a second pass over the final archive. b) using serialization for a data-state logging. Log(Archive ar, statedata){ // save seek point to index // append to archive ar << statedata; } Recover(Archive ar, statedata &, seekpoint) // set stream seekpoint ar >> statedata } I could envision something like this being made to work. So I would say that generally, serialization would be expected to be a serial operation. (surprise!). On the other hand, in certain special situations it might be possible/convenient to provide for some random access but I would expect that to be application specific. > >> The reason why I think the other is important, it that's it's actually > >> saving/loading support for plain C++ array -- which is rather basic > >> thing. > >> > > > > Hmmm - the library already implements serialization of plane C++ arrays > by > > serializing each element. > > For *fixed-size* arrays. But not for dynamically allocated arrays. BTW, it > seems we need two wrappers for completeness: > one for dynamic arrays with element-wise save I would envision one using ar << element_count For(i = 0; I < element_count; i++) ar << element[i] I'm not sure that its worth adding and documenting such an obvious and trivial thing as a separate wrapper. > and another for dynamic arrays with binary save. I see the binary object as filling that role. Ar << make_binary_object(dynamic_array_address, element_size * element_count); I once did propose a manual section name "Serialization Wrappers" which would have a couple of examples and highlight the fact that nvp and binary_object are instances of this concept. The idea was received unenthusiastically at the time but now I'm becoming convinced It's more effective to explain and illustrate the concept rather than try to anticipate all the wrappers that users might need. Actually, I think having such a section would have avoided the confusion surrounding the intended usage and purpose of binary_object. > I agree. I actually have a *crazy* but cute idea that one can use file > offset for object id. How object ids are assigned and can I customize that > process? That would keep overhead at absolute minimum. Object_id are assigned sequentially starting with 0. They are only used for classes which require tracking (e.g. when instances are serialized through pointers) they are used as indices into a vector so using these indices keeps overhead to a minimum. There are cases when an object id is assigned but not written to the archive I don't see these as having much utility outside of serialization. > Ah, I've missed that. Do I need to provide both 'type' and 'value'? Can't > serialization library work with just one? Could be. I just provided both so that I could interoperate with mpl without having to think about each specific case. > >> It also necessary to provide overload for char*. Does not count as > >> primitive > >> type? > > > > Default implementation for char * will work as it does for other > pointers. > > Eh... I though that serialization of pointers to builtin types is just not > allowed. > Default implementation level for pointers to primitives is set to not-serializable. This is resetable as it is for any other type. I doubt that on would want to do it for something like int or double as I would expect a lot of unintended side effects. One might want to use STRONG_TYPEDEF to define another int or double type and reset the implementation level of that - but that going off on a tangent. > Actually, my archive initially have only one (non-templated) 'save' for > unsigned. I got compile error until I've declared 'save' for const char*. > I'm not sure why. Its probably because of the above (pointer to a primitive type). Attempts to serialize instances of types whose implementation level is set to not-serializable will result in compile time assertions. (these assertions are sometimes the reason for the deep nested mpl). > > I came to conclude it presented a big security risk. The problem is the > > following: > > > later: > > > > char str[MAX_STRING_SIZE] > > > > ar >> static_cast<char *>(str); // to avoid str being treated as an > array > > > > suppose the text archive gets corrupted to: > > > > 3000 abc............ > > > > The archive will load with a buffer overrun - a security risk. > > Right. I think this problem can be addressed with a wrapper for dynamic > array > > char* str(0); > ar >> make_dynarray_wrapper(str) > > so that the library allocates the string itself. How does it know what size to make the array? Maybe you meen Ar >> make_dynarray_wrapper(str, number of elements * element size) Of course for char arrays one could use Ar << binary_object(str, element_count * element_size) And it would still be portable. > >> Can I define only one 'load' for unsigned int? > > > > As oppose to ? > > As opposed to overloads for all builtin types. For polymorphic_archive we > can't have templated function which 'fallbacks' anywhere. We need to have > a > closed set of virtual functions and I wonder what's the minimal set. > > the rest. BTW this provided a huge benefit. In the original version of > > last year I got into a never-ending battle to specify virtual functions > > which was > > dependent on the compiler - long long, etc. It was hopeless - moving to > > templates solved that. > > :-( I guess we're back to those problems. > I have recently got the polymorphic array working on my machine. Only a couple really small changes to the library code were required. For the list of primitive types I included all portable C++ primitive types. (that is no long long, __int64 etc.) I'm not sure how much interest the polymorphic archive will engender and it's a little hard to understand until you've actually spent enough time with the library to appreciate the limitations of templated code in certain scenarios. So although its clean (and clever) its not going to be easy to explain. Robert Ramey

3 2