"Robert Ramey" <ramey@rrsd.com> writes:

David Abrahams wrote:

...

1. The menu control doesn't seem to keep the browser's displayed URL in synch with the location being browsed (FireFox on Windows XP). Wasn't that a major coup of Jonathan's menu control?

The table of contents doesn't use Jonathons's control.

Aw, shucks.

I looked into it but it required that I mess around with allt the html (IIRC) and I was thinking that an eventual conversion to boost book would take care of this as a side effect. I've checked back on boost book from time to time and it doesn't seem to include this "for free".

Not yet.

In fact, a cursory examination of boost book makes it look to me that could be another time consumer to set up and get running on my windows/xp developement system.

It's fast to get going if you just use Rene's installer. Even so, there are many of us who'd be happy to hold your hand.

Also it seems that I have to install and keep up to date another bunch of stuff that I really don't have the time to keep up with (DOxygen?).

Nope; if you want to use doxygen you have to use an old version and not update it ;-)

It may not be as bad as I think its going to be - but why risk it as I had/have other stuff that consumes my attention.

Anyway, this has nothing to do with #1, really.

...

2. Deserialization of derived classes through base class pointers is an incredibly important subject -- even for many first-time users -- but the instructions for handling it are buried deep in Reference > Serializable Concept > Pointers > Pointers to Objects of Derived Classes

(http://www.boost.org/libs/serialization/doc/serialization.html#derivedpointe...)

This material _needs_ to be in the tutorial! In fact, the material in the reference section is narrative and tutorial in nature, which seems inappropriate for a reference manual. Maybe you should just move it?

Maybe. I'll consider it. But importance isn't really a good reason for including it in the tutorial. My decision of what to include in the tutorial is based on my own experience in trying to use other people's software. I need something I can skim to see if the thing is useful for what I want to do. I'm not really interested in understanding the intricacies of the package at this point. That's for later. I really want to know if it does enough in the way I need to justify investing more time in. Understanding of the issues related to derived pointers is a lot deeper than I think anyone want's to go at this point.

a. You might be skimming this introduction to see if it handles base/derived class serialization correctly, especially if you've been disappointed by other serialization frameworks. b. I've seen this phenomenon before. In considering the Boost.Build docs, even after a simple Hello, World tutorial intro, it's clear that users need to be guided onward with more tutorial-style writing. In other words, users need a tutorial, extended user guide, and formal/technical reference. I think the same applies to Boost.Python. Neither of these follow through optimally on the need for a user guide, yet. On a smaller scale, I think we did pretty well with the parameter library, even though the tutorial isn't really divided from the user guide.

It might be tempting to tweak the tutorial example to demostrate serialization through a base class pointer - but then one has to explain registration or export an instantiation of code not explicitly referred to.

Not necessarily. In the tutorial section you can present it as a "magic boilerplate incantation" that will be explained later.

All this make me quite skeptical of the idea.

Proof is in the pudding. My clients who have tried to use serialization were unable to get it right, because they never found that part of the doc, or if they saw it, they didn't recognize its importance. Serializing containers of pointers to polymorphic objects is a pretty basic need. You haven't had this question from users before?

Of course if I were writing a tutorial with David Abrahams in mind as the audience it would be quite different - actually I don't think we'd even need it!

?? I would *definitely* have needed it.

...

4. The documentation says that you can write your freestanding serialize() function in namespace boost::serialization, with the strong implication that it will work even on compilers that support ADL. But it won't work unless boost::serialization is an associated namespace of one of the arguments, as demonstrated by the following program:

namespace me { class X {}; }

namespace boost { namespace serialization {

template <class T> int call_serialize(T const& x) { serialize(x); return 0; }

void serialize(me::X);

}}

int y = boost::serialization::call_serialize(me::X());

In case it wasn't 100% obvious, the above fails to compile on a conforming compiler.

...

As far as I can tell, there's no requirement that any of the arguments to serialize have boost::serialization as an associated namespace.

There isn't. And I don't think it's necessary. serialize(me::X) is only called from within the namespace boost::serialization never from anywhere out side this namespace. Hence, the serialize function is found according to the rules of ordinary lookup.

No, not in the example above it isn't, because serialize follows the definition of call_serialize. The set of candidates that can be found by ordinary lookup is fixed at the template's point of definition. You have introduced yet another header order dependency here. You don't need to experience the wailing and gnashing of teeth associated with that problem again, do you?

Note that all the stl serializations (eg. boost/serialization/list.hpp ) all work on all platforms regardless of the namespace that the templated arguments are found it.

I don't see what that has to do with anything.

If one's compiler supports ADL, then he can use a free function in namespaces associated with the type being serialized. But it's not a requirement

I beg to differ. And the docs make it sound as though putting the functions in boost::serialization is the more portable of the two options.

I concede I've struggled with two-phase lookup and ADL so I'm willing to be shown to be wrong about this.

The example above demonstrates.

...

5. Archive Concept Requirements are specified in a completely new, unprecedented way.

Well that was certainly not my intent. I studied the SGI documentation and the explanation of why it was written the way it was and tried to conform to it in substance if not exactly in form. I also looked at the documentation for the new iterators to understand how to do this.

...

can see why you want to write them this way, but the result is that it isn't completely clear which elements of the interface are required and which are optional. For example, there's a colon after the class name. Does the class need to be derived from something?

The intention is to describe the requirements that an archive class must fulfill to be used with the serialization class that conform to their requirements. It describes what an archive has to be able to do in order to work. This is to be independent of any particular implementation of the archive concept and apply to any one.

Yes, I understand what the intent is. Actually, there is precedent for what you're trying to do, sorta. The Indiana proposal for concepts in C++0x (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2005/n1849.pdf) uses something called a "pseudo-signature" that looks very much like what you've written. However, there are special rules for reading and understanding pseudo-signatures. One way to make your specification formally complete would be to make sure that, when expressed according to the rules in that proposal, it says what you mean it to say. Then you can reference the proposal in the docs in case someone wants to understand the requirements on a formal level.

...

What about the names of function and member template parameters? I know the answer to that one, but a novice might not.

Hmm, I'm extremely doubtful that a novice understand documentation which describes library templates in a formal way. I suppose it depends who is considered an novice.

Maybe. It's certainly not complete enough for someone like me to be sure of what it really means.

...

What about the default arguments to member functions? Is it okay to write overloads?

As I read this, I don't understand the question but I'll review the relevant section.

I will concede I struggled with formal documentation. I never really understood the nomenclature and format of "formal library documentation" in any detailed way. The SGI website has been very helpful to me in this regard.

:) I'm using it on some slides now and discovering just how many holes, anachronisms, and inaccuracies there are in it. Still, the basic approach is a good one.

The boost page on how to write documentation by William Kempf has also been helpful in various ways.

I resolved to make the reference part of the documentation conform to the "formal" standard and used the references I described above. I found that I did have most of the required information in the documentation but that it was sort of jumbled about so I moved it about best I could to conform to the the "formal" standard. Still there wasn't a good place for some things so they ended up in places like "Special Considerations". layout

I don't actually think Bill Kempf's format results in particularly good docs. I used that approach for Boost.Python's reference and now I wish I hadn't. I think the Parameter library (admittedly much smaller) is much better.

...

6. In

http://www.boost.org/libs/serialization/doc/archive_reference.html#implement... it says:

All input archives should be derived from the following template:

This would be incorrect.

This section describes the archive implementations included in this library and the common features that they have. So the above should say: "All archives included in this library are derived from the following template"

...

template<class Archive> detail::common_iarchive;

but that's nowhere to be found in the archive concept requirements.

As I said, I see this implementation section as A means to fullfilling the basic requirement not as new requirements.

You said that? Where?

For example. The archive concept in no way requires that archive be implemented in terms of streams. However, all the archive classes in the library and all those derived from base classes in the library do in fact depend on streams.

The intention is that this is useful to those that might want to use these implementations as base classes. But it is not required that these classes be used. Any classes which fullfill the requirements of the Archive concept would be acceptable.

...

Which is it? Also, that "detail::" is actually nested in the boost::archive namespace, which is not at all clear from the text there.

I've put them in detail as they are features of the implementation of the archive classes included - not interfaces that any users would be expected to have to know about.

I don't care about your rationale---at least not yet. I care about the fact that the doc is unclear/confusing.

...

It's not clear to me why archives should live in a namespace other than serialization;

I can't answer the rest tonight, sorry. Gotta sleep. -- Dave Abrahams Boost Consulting www.boost-consulting.com

"Robert Ramey" <ramey@rrsd.com> writes:

.>> 4. The documentation says that you can write your freestanding

...

...

...

serialize() function in namespace boost::serialization, with the strong implication that it will work even on compilers that support ADL. But it won't work unless boost::serialization is an associated namespace of one of the arguments, as demonstrated by the following program:

namespace me { class X {}; }

namespace boost { namespace serialization {

template <class T> int call_serialize(T const& x) { serialize(x); return 0; }

void serialize(me::X);

}}

int y = boost::serialization::call_serialize(me::X());

In case it wasn't 100% obvious, the above fails to compile on a conforming compiler.

Well it wasn't obvious to me. Its still not. It just compiled on my comeau 4.3 without error.

I always thought that compiler was conforming. Oh well.

On Windows? That's normally in "microsoft bug emulation mode." Try http://www.comeaucomputing.com/tryitout (I did!) I hate having to go crawling through the standard just to find this stuff, but for you, because I like you, kid, I make a special exception: 14.6.4 Dependent name resolution [temp.dep.res] In resolving dependent names, names from the following sources are considered: — Declarations that are visible at the point of definition of the template. — Declarations from namespaces associated with the types of the function arguments both from the instantiation context (14.6.4.1) and from the definition context. Unless one of the arguments has boost::serialization as an associated namespace, only the first bullet point applies in this case. It means that any overloads provided by the user after the call_serialize function definition has been seen will not be considered for overload resolution.

Maybe no one has every had an issue with this as it is customary and natural to declare and/or define the template

template<class Archive> void serialize(Archive &ar, me::X, const unsigned int version){ ... }

in the same header where X itself is declared and/or defined.

That's irrelevant to the problem, actually.

Also the body of the serialize function typically will refer only to member data and functions.

So? The problem happens during name lookup. If the user's serialize overload is never considered, it doesn't matter what gets put in its implementation.

But the truth is I would have to delve into this a lot deeper to see what's going on here.

Well, you'd better. The documentation as of now is not adequate to know how to write an archive that works with the serialization library.

...

...

...

As far as I can tell, there's no requirement that any of the arguments to serialize have boost::serialization as an associated namespace.

There isn't. And I don't think it's necessary. serialize(me::X) is only called from within the namespace boost::serialization never from anywhere out side this namespace. Hence, the serialize function is found according to the rules of ordinary lookup.

No, not in the example above it isn't, because serialize follows the definition of call_serialize.

As above its not clear to me that anyone ever does this.

?? #include <boost/serialization/header-containing-call_serialize.hpp> #include "mytype-and-its-serialize-function.hpp" is all it takes.

...

The set of candidates that can be found by ordinary lookup is fixed at the template's point of definition.

...

You have introduced yet another header order dependency here. You don't need to experience the wailing and gnashing of teeth associated with that problem again, do you?

I'm getting used to it.

Please, don't get used to it! If your tolerance for it goes up any further nobody will be able to convince you to fix anything! ;-)

Actually this would be much worse than the one before. In my other infamous case, at least the compiler threw and #error when the rule was violated. With two-phase lookup, the compiler silently changes the program semantics when header order is changed.

It's not two-phase lookup that's the culprit, it's incorrect instructions about how to customize the serialize() functionality.

...

...

If one's compiler supports ADL, then he can use a free function in namespaces associated with the type being serialized. But it's not a requirement

I beg to differ. And the docs make it sound as though putting the functions in boost::serialization is the more portable of the two options.

Hmm - if one puts in the boost::serialization namespace it works except mayb in cases such as that above.

?? It fails unless one of the arguments to serialize has boost::serialization as an associated namespace. That's

Its not clear to me that that is not a contived example which never ^^^^ what is "that?" happens in practice.

If one puts in the namespace of the class being serialized - it depends upon ADL to function. So my view is that putting it into boost::serialization is more portable.

Even if that were true in practice, you can't recommend a practice that doesn't work on a conforming compiler as the more portable of the two, at least not without a big caveat that it relies on nonportable techniques... can you?!

In reality I suspect its not such an issue these days as most compilers seem to support ADL.

...

...

I concede I've struggled with two-phase lookup and ADL so I'm willing to be shown to be wrong about this.

...

The example above demonstrates.

Hmm - I'm not convinced yet.

The online compiler demonstrates.

...

Yes, I understand what the intent is. Actually, there is precedent for what you're trying to do, sorta. The Indiana proposal for concepts in C++0x (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2005/n1849.pdf) uses something called a "pseudo-signature" that looks very much like what you've written. However, there are special rules for reading and understanding pseudo-signatures. One way to make your specification formally complete would be to make sure that, when expressed according to the rules in that proposal, it says what you mean it to say. Then you can reference the proposal in the docs in case someone wants to understand the requirements on a formal level.

Well, I didn't know about this. But your explanation doesn't make sound very easy to understand or use.

Very well, then, if you don't like that, use the "traditional" table of valid expressions, etc.

...

...

...

What about the names of function and member template parameters?

What about about them? What is the question here?

Do I have to use the same names as you do in your example?

...

...

...

I know the answer to that one, but a novice might not.

What is the answer?

No.

...

Maybe. It's certainly not complete enough for someone like me to be sure of what it really means.

<referring to the SGI STL docs>

...

:) I'm using it on some slides now and discovering just how many holes, anachronisms, and inaccuracies there are in it. Still, the basic approach is a good one.

...

I don't actually think Bill Kempf's format results in particularly good docs. I used that approach for Boost.Python's reference and now I wish I hadn't. I think the Parameter library (admittedly much smaller) is much better.

Then someone should update those pages.

Or replace them.

I've strived to do things in the "boost" way to maximize the mental leverage to be gained from using an established pattern. If we should use them we should have another model.

Feel free to propose one.

BTW, updating these pages ain't going to be that easy. Traditionally, C/C++ programs were fairly easy to describe with an interface for each class along with notes regarding implementation semantics. Now with both class and function templates the old patterns are not enough. I look forward to seeing this improved.

Me too. <taps foot expectantly>

Another great idea would be to tie concept checking into the "formal" template documentation so that the code checks the documentation. A way too ambitious idea but fun to consider.

Already in the pipeline, believe it or not.

...

...

As I said, I see this implementation section as a means to fullfilling the basic requirement not as new requirements.

You said that? Where?

Reference/Archive Class Reference/Implementation states in its opening paragraph:

The Archive concept specifies the functions that a class must implement to in order to be used to serialize Serializable types. The library implements a family of archives appropriate for different purposes. This section describes how they have been implemented and how one can implement his own archive class.

That seems pretty clear to me that the are archives included model he Archive concept. It would be easy to add that sentence

Sorry, I can't understand anything you're trying to tell me in the whole quoted section above, so I don't know if it would help. -- Dave Abrahams Boost Consulting www.boost-consulting.com

David Abrahams wrote:

David Abrahams <dave@boost-consulting.com> writes:

...

...

Well it wasn't obvious to me. Its still not. It just compiled on my comeau 4.3 without error.

I always thought that compiler was conforming. Oh well.

On Windows? That's normally in "microsoft bug emulation mode." Try http://www.comeaucomputing.com/tryitout (I did!)

I hate having to go crawling through the standard just to find this stuff, but for you, because I like you, kid, I make a special exception:

14.6.4 Dependent name resolution [temp.dep.res]

In resolving dependent names, names from the following sources are considered:

- Declarations that are visible at the point of definition of the template.

- Declarations from namespaces associated with the types of the function arguments both from the instantiation context (14.6.4.1) and from the definition context.

To be honest, I'm still trying to parse this last sentence.

...

Unless one of the arguments has boost::serialization as an associated namespace, only the first bullet point applies in this case. It means that any overloads provided by the user after the call_serialize function definition has been seen will not be considered for overload resolution.

Have I still failed to make my point? Did you not get this message? How hard are you going to make me work for this one?

Working harder won't help. I just need more time to think about this. I don't know if you saw the documentation and code required to deal wih this issue in version 1.32 but it was very messy. I invested effort which I believe resulted in a major improvement. So far no one has come upon this in practice and I'm curious to know why that might be. Robert Ramey

David Abrahams <dave@boost-consulting.com> writes:

"Robert Ramey" <ramey@rrsd.com> writes:

...

Working harder won't help. I just need more time to think about this. I don't know if you saw the documentation and code required to deal wih this issue in version 1.32 but it was very messy. I invested effort which I believe resulted in a major improvement. So far no one has come upon this in practice and I'm curious to know why that might be.

Because they're all using broken compilers, and I know the standard better than most people. I was bit by not respecting 2-phase lookup in Boost.Python years ago and the lesson stuck with me, so I'm quite aware of what works on a conforming compiler and what doesn't.

So have you had time to consider the standard text and decide if I'm just inventing my interpretation of 14.6.4 Dependent name resolution [temp.dep.res] or if some adjustment in your approach is warranted? -- Dave Abrahams Boost Consulting www.boost-consulting.com

"Robert Ramey" <ramey@rrsd.com> writes:

Honestly, with time I have available, I've been resolving a couple of obscure but problematic issues re RC_1_33_0.

Just to summarize:

a) I've agreed to enhance the tutorial section on pointers so that the application of serialization of derived pointers isn't overlooked by the casual reader.

b) I've agreed to alter the "Archive Concept" section i) to eliminate the implementation of of the functions. ii) to put it in a tabular format without the "class ... " iii) consider whether it would better or worse to make to almost identical sections for loading archives and saving archives or just one section with notes for the operators/functions which don't apply.

If they have commonalities it may make sense to make them both refinements of the same base concept, "Archive." That's how you factor out commonality in concept definitions. It also makes a lot of sense because of your universal "&" operator, so users can write the requirements on their serialize() functions.

iv) probably change the name from "Archive Concept" to "Archive Type". I still have reservations about this as the term Archive is used to refer to any type that supports a specified set of operations. That would seem to fit in the definition of a concept to me. I'm still mulling this over.

i-iii sound perfectly appropriate for a concept definition. In that light, I don't know why you'd want to do iv; it seems completely incongruous. The fact that iv appears in your list with the label "probably" makes me wonder if you have understood me. Let me summarize my position: * We need Input and Output Archive concept definitions: sets of requirements that a type needs need to fulfill in order to use it as an archive with the serialization library. * These concept definitions are properly grouped with the serialization library. * You should find some way to describe what it means for an input and an output archive to be compatible. Try to do that as formally as possible. You may have to rely on the fuzzy C++ notion of "equivalence," but I wouldn't blame you for that. * If you want to describe the implementation and/or interface specifics of the library-supplied archives (not a bad idea), that should not be labelled a "concept."

c) Re the last issue of which namespace non-instrusive serialization functions should be placed into.

I actually did spend a lot of time on this in the past. The issue surfaced when I first compiled with CW, gcc 4.0 and Commeau.

CW does 2-phase lookup properly, I believe.

I carefully studied the explanation in Vandevoordes and Jostuttis book, and made changes. This resulted in a much simpler "suggestion" as which namespace to use for non-intrusive serialization functions. It also resulted in all the above compilers compiling and executing all the tests without problem of any kind when.

...

So I've already spent a lot of time on this issue.

Unfortunately, it's easy to write a test that fails to uncover a problem. The optimal thing to do when you are confronted with an issue like this is to write tests that try to use the issue to break your library. If you pick the wrong declaration order or inadvertently associate the wrong namespaces, you'll miss it.

Now it seems I didn't appreciate all there is two know about two-phase lookup. So it takes some time to go back and review all the considerations that entered my mind when I "fixed" it the first time. It's not that I'm dismissing your concerns.

good.

Remind me of what its that you want to see changed. The documentation re-worded or what? If its just a question about making that documentation more precise, that would be pretty easy.

Precision isn't the issue. It's hard to imagine that this helps, since you can look at the message history as easily as I can, but let me refresh your memory: 4. The documentation says that you can write your freestanding serialize() function in namespace boost::serialization, with the strong implication that it will work even on compilers that support ADL. But it won't work unless boost::serialization is an associated namespace of one of the arguments, as demonstrated by the following program: namespace me { class X {}; } namespace boost { namespace serialization { template <class T> int call_serialize(T const& x) { serialize(x); return 0; } void serialize(me::X); }} int y = boost::serialization::call_serialize(me::X()); As far as I can tell, there's no requirement that any of the arguments to serialize have boost::serialization as an associated namespace.

If one's compiler supports ADL, then he can use a free function in namespaces associated with the type being serialized. But it's not a requirement

I beg to differ. And the docs make it sound as though putting the functions in boost::serialization is the more portable of the two options. So the recommendation that users write their freestanding serialize functions in namespace boost::serialization doesn't work in general on conforming compilers, and yet your documentation makes it should like it's the most portable approach. You need to mke sure that it isn't made out to be a portable option. The best thing you could do, IMO, is write a recommendation that works on conforming compilers, e.g. Overload serialize in the namespace of your class. Then write another, possibly uglier, recommendation that works on all supported compilers. If you have to use macros, that's fine. E.g., // somewhere in the serialization library #if BOOST_NO_ARGUMENT_DEPENDENT_LOOKUP # define BOOST_SERIALIZE(namespace) serialize #else # define BOOST_SERIALIZE(namespace) namespace::serialize #endif // user code namespace my { namespace ns { class my_class { ... }; template <class A> void serialize(A&, my_class); // declaration (unused in non-ADL case) }} // definition template <class Archive> void BOOST_SERIALIZE(my::ns)(Archive& ar, ny::ns::my_class& x) { ... } -- Dave Abrahams Boost Consulting www.boost-consulting.com

"Robert Ramey" <ramey@rrsd.com> writes:

David Abrahams wrote:

...

Overload serialize in the namespace of your class.

Then write another, possibly uglier, recommendation that works on all supported compilers. If you have to use macros, that's fine. E.g.,

// somewhere in the serialization library #if BOOST_NO_ARGUMENT_DEPENDENT_LOOKUP # define BOOST_SERIALIZE(namespace) serialize #else # define BOOST_SERIALIZE(namespace) namespace::serialize #endif

// user code namespace my { namespace ns { class my_class { ... };

template <class A> void serialize(A&, my_class); // declaration (unused in non-ADL case) }}

// definition template <class Archive> void BOOST_SERIALIZE(my::ns)(Archive& ar, ny::ns::my_class& x) { ... }

That sounds like what I did for version 1.32.

Is it equivalent to what you did, or does it just sound reminiscent?

I considered a very ugly hack. ^ "it?"

I don't think I was the only person that felt this way. I resolved to fix it in the next version - and here we are. oh well.

Surely you don't think the recommendation I'm suggesting for conforming compilers is an ugly hack? IMO it doesn't matter much how ugly the portable-to-broken-compilers workaround is, as long as it's legit for the conforming ones and doesn't induce maintenance problems. -- Dave Abrahams Boost Consulting www.boost-consulting.com

David Abrahams wrote:

David Abrahams <dave@boost-consulting.com> writes:

...

"Robert Ramey" <ramey@rrsd.com> writes:

...

That sounds like what I did for version 1.32.

Is it equivalent to what you did, or does it just sound reminiscent?

...

I considered a very ugly hack. ^ "it?"

...

I don't think I was the only person that felt this way. I resolved to fix it in the next version - and here we are. oh well.

Surely you don't think the recommendation I'm suggesting for conforming compilers is an ugly hack?

LOL - I said I considered my first incarnation in 1.32 and ugly hack. I haven't studied your suggestion in enough detail to definitively characterise it as an ugly hack. But it IS ugly.

...

IMO it doesn't matter much how ugly the portable-to-broken-compilers workaround is, as long as it's legit for the conforming ones and doesn't induce maintenance problems.

So do you have anything to say about this?

I hate to be a pest, but I've sunk so much time into getting these issues remedied that I'm unwilling to have the issue evaporate into the past without getting addressed.

Here is what I'm doing. a) I've updated the tutorial and tweaked the explanation of derived pointers to highlight the issue more. b) I'm going to remove the class declaration and function implementation from the Archive Concept part of the document. The above I will check into RC_1_33_0 Other issues that I intend to defer for the next version c) investigate the ar.template register_type<T>() vs ar.register_type(static_cast<T*>(NULL) workaround. I did spend some time coming up with this workaround and it has worked find and now its used by user code so changing this per your suggestion (assuming it would actually work) would have to be considered. It also takes a lot a time to verify that it works or doesn't on all the compilers. d) investicate the two-phase lookup issue. There is no way any such change could be made for RC_1_33_0 and the next boost release won't be for another 10? months, so I feel that I can take some time with this. e) I'm intrigued with Joaquin's suggestion regarding formal documentation of Archive and Serializable Concepts but this can and will be defered. f) I would like to revisit the Archive Implementation section at my leisure in light of this discussion. Thsi section describes how the included archve classes are implemented. There is a separate interface class which uses CRTP which conceptually dovetails with the Archive Concept as decribed above. Other issues I would like to spend time on if I had it. g) Makding a demo/test for a DLL type plugin using the polymorphic archive. This would be lightweight COM. h) Making a demo/test that generates a table of timings on all the boost platforms in order to answer questions like how much overhead does using an archive vs standard i/o. Does the compiler really optimize inline functiions to hte max, etc. So that's how it stands now. Robert Ramey

David Abrahams wrote:

The best thing you could do, IMO, is write a recommendation that works on conforming compilers, e.g.

Overload serialize in the namespace of your class.

Oh, that's news to me. I thought that would work only in compilers that implemented ADL. I understood from previous postings that this would entail building a set a macros which addressed the varying aspect of conformity. In 1.32 I had #if ... in order to place stl serializations in either the stl namespace or the boost::serialization namespace depending on whether the compiler supported ADL and two-phase lookup or neither of these things. That was the only way I could get everything to compile on all platforms. Then generated the requirement to address the issue in the documentation with a table on what the user should use. That's what I call ugly. FWIW - I'm not sure if the example you site would ever come up in practice - it certainly hasn't yet. If I understand this correctly, the problem would occur if someone writes somethng like: my_class.hpp class my_class ... template<class Archive, my_class> void serialize(Archive &ar, my_class & t, cont unsigned int version); // declaration only my_app.cpp main(... { my_class mc; ... ar & mc; } template<class Archive, my_class> void serialize(Archive &ar, my_class & t, cont unsigned int version); // definition only I'm not absolutly sure that this is the only scenario which would create problem and that's why I wanted more time to consider it. If this is the only way the two-phase problem could manifest itself, I would guess that in practice it would never be an issue. I don't see users doing this. This was what I was getting at when I asked why the problem hasn't appeared upto now. Of course I don't really know whether this is because so many compilers fail to implement two-phase lookup.

...

b) I'm going to remove the class declaration and function implementation from the Archive Concept part of the document.

Are you planning to replace it with anything? It would be pretty silly to have a section called Archive Concept with no concept description.

I meant to leave in the text below the "class" schema. Before I started I reviewed again the SGI documents and concluded that that text could be fit into the SGI form for concepts. I'm using http://www.sgi.com/tech/stl/Container.html as a "typical example". This isn't hard - its basically a question of reformatting. I hope that will be satisfactory.

...

The above I will check into RC_1_33_0

Other issues that I intend to defer for the next version

...

d) investicate the two-phase lookup issue. There is no way any such change could be made for RC_1_33_0 and the next boost release won't be for another 10? months, so I feel that I can take some time with this.

That seems extreme to me. Surely for RC_1_33_0 a documentation fix that provides instructions that are both correct for conforming compilers and consistent with the current library implementation is possible? Not much more than a small tweak in emphasis would be required.

see above. Robert Ramey

David Abrahams wrote:

How to solve this problem is not a mystery. We just had a long thread about it entitled "[range] How to extend Boost.Range?" Have you completely missed it?

I haven't followed it.

I am getting frustrated here because it seems to me that you haven't done your homework,

That's why I asked for more time to consider the issue - but you couldn't accept that.

yet, after asking me to propose solutions, you resist and call them ugly.

...

That was the only way I could get everything to compile on all platforms.

It seems to me from what you've been saying that your approach to solving the serialize( ... ) dispatching/customization problem here has been to try different configurations of the code until you could get it to pass select tests on certain platforms, without regard for standard conformance and theoretical portability.

That's only partly true. I start out with what I believe is standard conforming code. The I compile it on all my platforms and discover problems. Then try to make modifications so that it passes with all the compilers and still is standard conforming. BOOST macros play an important role here. Sometimes standard conforming code has to be suppresed and replaced with other standard conforming code - e.g ar.template register_type<my_class>() gets replaced with ar.register_type(static_cast<my_class *>(NULL)). (God I hope that's conforming.). In this case I had to confront the question of which namespace serializations of standard containers should be placed in and still have the code compile and function with all compilers. The first solution used either std or boost::serialization depending upon configuration macros. It seemed very arbitrary to me. By arranging things so that the serialize(..) template was allways called from within the boost::serialization namespace I thought I had fixed the problem by permiting the overloads to be found via ordinary lookup.

That's a well-known way to end up with code that doesn't work portably or in configurations you haven't tried (for example, hmm, let me think... when the order of headers changes).

I'm aware of this.

...

Then generated the requirement to address the issue in the documentation with a table on what the user should use. That's what I call ugly.

Huh? I didn't propose a table.

I was refering to my previous solution.

And, IMO there's nothing particularly inelegant about using a table in documentation; there's well-established precedent for it in the standard and Boost. Certainly documenting what the user should do is a must. So I don't see what you're unhappy about.

If you don't like having to tell users to do something special on nonconforming compilers, well, the answer is to either find a clean idiom that works everywhere (for this particular problem, many have tried -- I don't believe such an idiom exists)

The problem when one writes code which is to be portable accross compilers - say serialization of the std::set . Its not enough for it compile and run on my platform - it has to run on them all. So one ends up with #if ... all over the place to vary the destination namespace depending on whether or not the compiler supports ADL - that's ugly.

or stop supporting those broken compilers.

This the crux of the matter right here. From the prespective of two-phase lookup - they're all "broken" except CW which can't use the most interesting parts of the serilization library because its broken in other ways. I made one choice - you would make another.

Read the passage of the standard I quoted to you, and look carefully at the example above. Do your homework.

I'm trying.

Lots of generic libraries have a similar dispatching issue to address, e.g. Boost.Range. There's a nice, free online compiler againist which you can test your 2-phase problems.

I downloaded and installed comeau 4.3 sometime ago specifically in order to be able to test this. I still use it to compile all the library modules and tests as a double check. They all pass. Alas, they don't link due to some collision with the Boost Test library that cropped up after 1.32. Now I'm informed that we have the test compile time switches set so supress two-phase lookup (or parts of it) so I have no "conforming" compiler available. This is a hinderence to addressing problems such as this.

I've quoted the standard chapter and verse. I've given you simple example programs that demonstrate the problem. I don't know what more I can do... do I need to build a program that actually uses the serialization library before you'll believe me?!

That's what I'll have to do. FWIW its not that don't believe you're correct about the way two-phase lookup works. But I"m curious to know why things have worked so well so far. Is it because no one is using conforming compilers? Is it because common use cases don't trip on this? This is so far unexplained. The only way I'll discover this is by "doing my homework" which I haven't been able to get to. Maybe there is a real simple solution here. The relevent passage in the documentation currently states: Namespaces for Free Function Overrides For maximum portability, include any free functions templates and definitions in the namespace boost::serialization. If portability is not a concern and the compiler being used supports ADL (Argument Dependent Lookup) the free functions and templates can be in any of the following namespaces: a.. boost::serialization b.. namespace of the archive class c.. namespace of the type being serialized So if I were to change it to the following would that be satisfactory? Namespaces for Free Function Overrides If your compiler supports two-phase lookup and ADL (Argument Dependent Lookup) the free functions and templates can be in any of the following namespaces: namespace of the archive class namespace of the type being serialized If your compiler doesn't support two-phase lookup but it does support ADL free functions and templates can be in any of the following namespaces a.. boost::serialization b.. namespace of the archive class c.. namespace of the type being serialized If your compiler doesn't support ADL free functions and templates must be in the boost::serialization namespace. Of course that begs the main question I set out to answer. How to make ones code work on all boost platforms - but we can just leave that to the judgement of the library user. Robert Ramey

David Abrahams wrote:

...

that cropped up after 1.32. Now I'm informed that we have the test compile time switches set so supress two-phase lookup (or parts of it) so I have no "conforming" compiler available. This is a hinderence to addressing problems such as this.

It's hard for me to feel very sympathetic. You have the online compiler. You could find out what the switches are (--no-microsoft and/or --no-microsoft-bugs) to turn on 2-phase lookup on your local installation of comeau and do some experiments.

Do you mean that --A or --a is insufficient to enable two-phase lookup with VC as backend? This is very surprising to me. I seem to remember como catching two-phase lookup problems when I only use --A. Jonathan

David Abrahams wrote:

"Jonathan Turkanis" <technews@kangaroologic.com> writes:

...

Do you mean that --A or --a is insufficient to enable two-phase lookup with VC as backend? This is very surprising to me. I seem to remember como catching two-phase lookup problems when I only use --A.

I don't actually know. The point is it takes about 3-5 min of research to find out. Read the como docs, and if that doesn't do it, email support@comeaucomputing.com and do some experiments to find out while waiting for Greg to get back to you.

I've already spent a lot of time doing this. I was surprised because it sounded like you had some new information. Jonathan

"Robert Ramey" <ramey@rrsd.com> writes:

I would call your attention to the code in serialization.hpp - line 108 "layer 3". I believe that this code actually considered this problem. By using an argument from the boost::serialzation namespace It guarenteed that the correct overload would be found on the second phase of two phase lookup on compilers supporting ADL (all those that implement two-phase lookup). If I remember correctly, I also studied the rules that governed the conversion from boost::serialization::version_type to const unsigned integer and this information was used to make the process work. I left the note you see in the documentation. This resolved the compilation errors I was having with compilers which implement two-phase lookup so naturally I took this a confirmation that I understood the details. Unfortunately it was some time ago and and after I thought I resolved the problems generated by two-phase lookup I moved on to other problems. So I thought I understood it. Of course now I'm not so sure. I forgot the details of my solution but did remember the amount of detail I had to plow through.

Yikes! **** I owe you an abject apology. I'm very sorry for wasting all **** **** this time and energy. I should have looked at the code much **** **** sooner. That's a very clever technique. **** Provided the file_version parameter is never templatized, that trick works and it's always okay to overload in boost::serialization.

...

It's hard for me to feel very sympathetic. You have the online compiler.

I don't think its practical to use the online compiler to compile code that requires scores of files.

I've done it. Just preprocess and paste.

...

You could find out what the switches are (--no-microsoft and/or --no-microsoft-bugs) to turn on 2-phase lookup on your local installation of comeau and do some experiments.

I can and I will. I had no reason to suppose that it was turned off for boost (presumably in the toolset). I just assumed that build tools would be setup to maximize compiler conformance with the standard. Perhaps presumptuous on my part but I have to make some reasonable assumptions or I'll never get anything done.

The problem is that como still doesn't have a switch that turns off MS bug compatibility but leaves on the extensions needed to compile MS system headers. Without the ability to process Windows.h, many Boost libraries wouldn't compile.

...

That would still be a big improvement on the status quo.

OK I'll just make this alteration in that section. Halleluhuh

At this point, if you make any change I think you should add a note for those of us who are too clever by half, saying that, yes, it's really okay to overload in boost::serialization, and describing the trick used. So sorry, -- Dave Abrahams Boost Consulting www.boost-consulting.com

"Robert Ramey" <ramey@rrsd.com> writes:

David Abrahams wrote:

...

At this point, if you make any change I think you should add a note for those of us who are too clever by half, saying that, yes, it's really okay to overload in boost::serialization, and describing the trick used.

no problem with that.

I've recently checked in changes in the documentation in accordance with this discussion. I believe that they will be satisfactory.

BUT there is a problem.

I recast the section Archive Concept / Saving in the table format used by SGI. I'm very dissatisfied with this for esthetic reasons. I may be more sensitive to this than other people because I use a monitor than can display in portrait mode and that is the way I always use it. (Its in explicable to me that everyone doesn't do this!).

In principle I agree. However, I like having two portrait-sized images side-by-side :) I think the aesthetics of the presentation are mostly OK; your biggest problem is with the long code in the table cells. For the MPL book we dealt with this by inserting line breaks in the cell, e.g.: ar.template register_type< T >() That's an extreme example -- I probably would only use one newline in that one. Oh, part of the problem is that the "Expression" column is full of things that aren't valid expressions, and are therefore much longer than they should be! Okay, it looks like I'm going to have to go through the whole thing and add my comments :(

An Archive object is a sequence of bytes created from an arbitrary nested set of C++ data structures. Here we describe the Archive concept. This is a list of requirements that a class must fullfill in order for a class to be used to save and load and arbitary nest set of C++ structures. Strictly speaking, there are two Archive concepts. One for saving data and one for loading it back. In this document we use the term Archive concepts for either one. Which one we mean should be clear from the context.

That's not really acceptable. The proper thing to do is define the Archive concept of which LoadingArchive and SavingArchive are refinements (choose better names; I picked those to be consistent

Saving

Associated types

+-----------------+-------------------+-------------------------+ |boost::mpl::bool_|Archive::is_saving |boost::mpl::true if this | missing "_ "---------------^ should be "Archive"--^^^^

rewrite: boost::mpl::true_ if data can be saved to an object of type Archive.

| | |is an Archive to which | | | |data can be | | | |saved. boost::mpl::false | | | |otherwise. For an | | | |example showing how this | | | |can beused, see the | | | |implementation of | | | |split_free.hpp. |

The last sentence should be in a footnote or in the main body of the text below.

+-----------------+-------------------+-------------------------+ |boost::mpl::bool_|Archive::is_loading|boost::mpl::true if this | | | |is an Archive from which | | | |data can be | | | |loaded. boost::mpl::false| | | |otherwise. | +-----------------+-------------------+-------------------------+

All table columns need column headers. The first column doesn't make any sense. It's the name of a template. Strike it. It seems like the second column is a Valid Expression just like any other. [is_saving/is_loading are poorly chosen names for compile-time constants: they strongly imply state. I would use can_save and can_load]

Notation

Archive A type that is a model of the Archive concept. T A type that is a model of the Serializable Concept. ar Object of type Archive t Object of type T

You should use something like x instead of t. Having T and t in the same context is slightly confusing and it makes it very difficult to verbally discuss what's in the table.

Valid expressions The following expressions must be valid. \

Kill the backslash-------------------------^

+----------+---------------------+---------------------------------+ |Name |Expression |Return type | | | | |

Note: the "Name" column is a luxury. If you find it helps the aesthetics it's okay to leave it out. I suggest you also drop the word "return" from "return type"

+==========+=====================+=================================+ |Save |ar.save_binary(const |void | |Binary |void *, std::size_t) | |

Okay, that's not a valid expression. You should write something like: ar.save_binary( p, n ) and then add a Notation section that includes something like n a non-negative value of integral type p a pointer to n bytes of storage containing POD data

+----------+---------------------+---------------------------------+ |Save |ar << t |Archive & | | | | | +----------+---------------------+---------------------------------+ |Save |ar & t |Archive & | | | | | +----------+---------------------+---------------------------------+ |Register |ar.template |void | |type |register_type<T>() | | +----------+---------------------+---------------------------------+ |Library |ar.library_version() |unsigned int | |Version | | | +----------+---------------------+---------------------------------+

Expression Semantics +----------+---------------------+---------------------------------+ |Name |Expression |Semantics | +----------+---------------------+---------------------------------+ |Save |ar.save_binary(const |Appends count sequence of bytes | |Binary |void * address, |starting at address to the | | |std::size_t count) |archive. |

Again, not a valid expression. Apply the same correction used earlier here. I don't see why you're not using a single table with 4 columns including Semantics and Return type. Cross-referencing two tables is more work for the user than it should be.

+----------+---------------------+---------------------------------+ |Save |ar << t |Appends a sequence of bytes | | | |corresponding to the value of | | | |type T to the archive. | +----------+---------------------+---------------------------------+ |Save |ar & t |Appends a sequence of bytes | | | |corresponding to the value of | | | |type T to the archive. |

That description of the semantics doesn't seem detailed enough to me. Doesn't it need to call serialize(ar, t, some-version) at some point? I bet it's not enough to append any arbitrary sequence of bytes that I determine corresponds to the value of type T. Remember, this table has to tell me what I need to do in my archive to make it work with the library. Concept tables are not primarily for users of the concept's models (who should have specific documentation for each model), they're for implementors of the models. Don't use the container requirements tables as examples, because the container concepts are almost useless for generic programming. Look at the iterator requirements.

+----------+---------------------+---------------------------------+ |Register |ar.register_type<T>(T|Appends a sequential integer to | |Type |* t = NULL) |the archive. This integer becomes|

What is a "sequential integer?" The expository writing there really should go in a footnote or in the main body of the text below. These tables should be both minimal and complete.

| | |the "key" used to look up the | | | |class type when the archive is | | | |later loaded. This process is | | | |referred to as "class | | | |registration". It is only | | | |necessary to invoke this function| | | |for objects of derived classes | | | |which are serialized through a | | | |base class pointer. This is | | | |explained in detail in Special | | | |Considerations - Derived | | | |Pointers. |

Again, not a valid expression. Maybe you mean ar.register_type((T*)0) ?? The semantics column once again seems far too much geared to users of models of the concept and gives little information to the potential archive implementor.

+----------+---------------------+---------------------------------+ |Library |ar.library_version() |Returns the version number of the| |Version | |serialization library that | | | |created the archive. This number |

So, I'm implementing an archive. How am I supposed to acquire the correct version number? Take the following expository stuff out of the table.

| | |will be incremented each time the| | | |library is altered in such a way | | | |that serialization could be | | | |altered for some type. For | | | |example, suppose the type used | | | |for a count of collection members| | | |is changed. The code that loads | | | |collections might be conditioned | | | |on the library version to make | | | |sure that libraries created by | | | |previous versions of the library | | | |can still be read. | +----------+---------------------+---------------------------------+

Loading

A type that models the Archive concept is a class with the following members:

The template parameter T must correspond to a type which models the

"correspond to" or "be?" AFAICT, you mean: T must be a model of Serializable.

Serializable concept.

template<class T> iarchive & operator>>(T & t);

What is this? operator>> is a binary function. Perhaps you mean this to be a member of something, but if so, it's unclear of what. What is iarchive? Is this documentation of the Archive concept or of some specific model? If the former, is operator>> really required to return iarchive&? If the latter, it doesn't belong on a page titled "Archive Concept." I just snipped out the rest of the documentation because all of the questions I am asking in this break between quotations apply to most of it. I can't tell what you're trying to do. It seems like you don't quite have a clear understanding of what a "Concept" is yet.

I know I didn't choose the orginal form without thinking about it. Looking back I realize now I was strongly influenced by the presentation in "STL Tutorial and Reference Guide" which lays out the requirements in a form similar to the one I chose.

I haven't seen it, but it would shock me if Dave Musser did it wrong.

Aside from any errors that remain which I will be happy to fix in any case, I would like to leave the requirements in the dictionary text form as it was originally.

?? Dictionary text form??

If I move to the table form for the Archive Concept, then in the interest of consistency I would also have to do that with the section Serializable Concept which would be a much larger task. So what I would like to do is revert to the dictionary format for the saving archive concept. I've just uploaded updated document files. It contains one format for the save archive concept and the other for the load archive concept. Interested parties are invited to comment before I make a final change in one or the other.

Unless I'm missing something huge (happened once this week already), the format is the least of your problems. The content is wrong; it doesn't do what a concept description is supposed to do. -- Dave Abrahams Boost Consulting www.boost-consulting.com

I considered your comments and re-wrote the section "Archive Concept" and checked it into RC_1_33_0 last week. Robert Ramey David Abrahams wrote:

Hi Robert,

Are you going to reply to

http://article.gmane.org/gmane.comp.lib.boost.devel/131257

??

Thanks,

"Robert Ramey" <ramey@rrsd.com> writes:

The library is two parts:

a) Serialization Concept. This describes what a type has to support in order to be considered a serializable type. All the support for this concept is in the namespace "boost::serialization". Its concievable that someday someone (not me) might want to expand this to something more grandios such as Reflection concept. There is already a baby step in that direction wiht the notion of serializable traits, assignment of an exportable name, assignmetn of a printable name for the variable which the type is applied to(name-value pairs). There is to be no notion of the archive concept here.

b) Archive Concept. This describes what an type has to support in order to be used an archive. Basically these requirements boil down to the archive supporting the operations ar << and ar >> for any type which models the Serialization Concept.

You mean the "Serializeable" Concept? I thought input and output archives could be different types? Seems to me that you have a multi-type concept (look that up in the indiana concepts proposal cited earlier) at play here, Serlializeable<InputArchive, OutputArchive, Datum> with requirements, (given InputArchive ia, OutputArchive io, and Datum d), something like: +----------------+----------+-----------------------------------------------+ |Valid Expression|Semantics |Notes | +================+==========+===============================================+ |oa << d; |d2 is |I know, it's a bit more complicated than this | | |equivalent|since d may not be default constructible and | |Datum d2; |to d |instead might support inplace deserializing. | | | |Maybe there are two concepts for the two cases.| |ia >> d2; | | | +----------------+----------+-----------------------------------------------+

Thus

a) any class fullfilling the requirements of the Archive Concept, can be used to serialize any types which model the Serializable concept.

It's not a very useful concept unless it also allows you to deserialize and be assured of a sensible result.

b) in creating a new data type, it is sufficient to know that it fullfills the Serialization Concept to know that it can be serialized with any archive class.

This is THE key design goal of the serialization libary.

That's great. You have separation of concerns and orthogonality. You have turned an NxM problem into an N+M problem, just as iterators do in the STL. That doesn't mean the STL containers and algorithms should all be in separate top-level namespaces!

In fact, along with two requests for more formal documentation (one from you), one of my main motivations for investing effort in improving the documentation was to make this design much more obvious. Hmmm - looks like it failed in this regard. Oh well.

My intention has been to organize source code modules in subdirectories and matching namespaces to reflect this design. This is described in a special section of the manual.

Makes sense, but it's all part of the serialization library. Any subnamespaces should go under boost::serialization.

The maintainence of the key distinction between serialization and archive concepts - and the fact that perhaps its not as clear in the documentation as it might be has a couple of other repercussions that have manifested themselves on this list.

a) Any header from the "serialization" part can be included in a header without triggering requiremetn for the library even in the presense of auto-link.

I don't know what that means. All I know is that you've had some problems apparently related to auto-link.

This would not have been possible without a library design based on this concept. Actually supporting this was a some effort - mainly to fix places where I accidently violated this design feature.

b) My imposition of the "rule" that serialiation headers precede archive headers. This came about as an afterthought while fixeing the auto-link issue described above. Remember I always have in mind that serialization is an independent concept an archives are dependent upon seerialization. So it never occured to me that one would ever mix and archive and serialization header in another header. So I saw no issue in imposing this "rule". As it turned out, this "rule" wasn't really the the right way to fix the auto-link issue above and now I'm manage to fix it so the rule isn't necessary anymore. But I'm struck by the fact that lots of people never saw my point of view.

Which point of view?

c) I sometimes get code which overides the serialization for a specific archive. I can see where sometimes it might be interesting (eg. a special archive type for debug or checkpointing) but in the cases I've seen its just done without considering that one is now breaking the orthogonality between archves and serializations in a way that doesn't add anything other than future job security. Its not a big thing - its just slightly irksome - like sitting on a small pebble.

Sounds like, if you want to support that need while preserving orthogonality, you need an additional layer of indirection in the library design.

Clearly there's an aspect of the library that I see as fundamental that either I'm wrong about or have failed to communicate.

Probably the latter. How about making it more prominent in the docs?

Oh well.

...

if they were in boost::serialization *and* users were required to derive from one of the archive types, that would solve problem 4.

I believe that problem 4 is not a problem. Feel free to prove me wrong here.

Already done, if you had only tried to compile the code I posted ;-)

I very much appreciate you're looking at my documentation with your incredibly acute vision.

You're welcome. I had help from colleagues and clients, I admit. Sometimes it's just a matter of being willing to write down what we've noticed, and then go through the arguments that almost invariably ensue.

This is the first time anyone has really done this and find it very helpful. I have spent more time on this than it might seem from looking at it and I do want to improve it - but it ain't as easy as it looks.

It never is. Docs are hard and time-consuming to get right. It helps to encourage your critical reviewers :) -- Dave Abrahams Boost Consulting www.boost-consulting.com

"Robert Ramey" <ramey@rrsd.com> writes:

David Abrahams wrote:

...

"Robert Ramey" <ramey@rrsd.com> writes:

...

The library is two parts:

a) Serialization Concept. ... b) Archive Concept. This describes what an type has to support in order to be used an archive. Basically these requirements boil down to the archive supporting the operations ar << and ar >> for any ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ type which models the Serialization Concept.

You mean the "Serializeable" Concept? Of course, sorry

...

I thought input and output > archives could be different types?

I don't understand this.

You just claimed above that an archive has to support both << and >>. That's inconsistent with the idea that you can write an archive that only supports input, for example.

I guess one should really say there are two archive concepts. A Saving Archive concept and a Loading Archive concept.

Okay, maybe _the docs_ should really say that.

I'm not sure if that's what you're getting at.

What I'm really getting at is that concepts need to be documented rigorously, and not sort of clearly enough that you can explain the real requirements to someone in a mailing list posting after they've told you that your first explanation doesn't seem to match up with reality.

But there it is. They are noted as being different but explained in terms of just one of them in order to save typing and aid understanding through the usage of symetry and analogy.

Well, it doesn't aid in understanding.

...

Seems to me that you have a multi-type concept (look that up in the indiana concepts proposal cited earlier) at play here, ...

Well, just my opinion - this idea seems a lot more complicated to explain and understand.

If you don't like multi-type concepts, then when you describe the effects of deserialization you need to have defined some notion of "compatible archive format" that allows you to guarantee the results make sense. You can't expect to serialize with a binary oarchive and deserialize with an XML iarchive and get a sensible result.

...

...

a) any class fullfilling the requirements of the Archive Concept, can be used to serialize any types which model the Serializable concept.

It's not a very useful concept unless it also allows you to deserialize and be assured of a sensible result.

Lots of things can be formally correct but utterly useless.

Lots of things can seem logical and common sense to their author but be logically vacuous and thus utterly useless to someone who doesn't already understand what the author intends to say.

Or so it would seem.

??

I would suggest something like non-euclidean geometry. But the real question is what is the proper role and purpose of formal documents? If its to describe something in a mathematical sense you'll have one thing. The other extreme would be a cookbook. A software library is something in between. We're trying to find where it fits.

Right. But the author of a piece of software is not in a very good position to judge the comprehensibility and utility of its documentation. IMO you ought to recognize that and not be so quick to argue with someone who is reporting that the docs are hard to use and suggesting ways to improve them, especially when that person has been through that cycle many times with his own users and learnt from his mistakes.

...

That's great. You have separation of concerns and orthogonality. You have turned an NxM problem into an N+M problem, just as iterators do in the STL. That doesn't mean the STL containers and algorithms should all be in separate top-level namespaces! ^^^^^^^^^

It doesn't mean they shouldn't either. Some statistics on the serialization library

directory lines of code files ====== ======== === serialization 6600 46 archive 9400 89 src 3200 37 doc 7200 34 test 8400 105 example 2200 18 build 233 3 ==== ======= ==== total 37200 326

So? Formatting aside, I don't see your point. If all that can be grouped as boost::whatever it can also be grouped as boost::serialization::whatever.

...

Makes sense, but it's all part of the serialization library. Any subnamespaces should go under boost::serialization. ^^^^^^^^^^^^^

This is a BIG package. Its a huge job to keep things from getting mixed up. Implementing this in one namespace would have be a big problem.

I'm confused about how you could think I am suggesting "Implementing this in one namespace." What do I need to do to make myself clearer.

Now your thinking (may I presume) well use serialization::archive and serialization::serialization (or something similar).

Yes. Why are you wasting your time with all the foregoing?

I suppose this could have been done but it wouldn't have added anything other than a lot of typing. The current organization makes it clear to me that serialization (ncluding serializaton class traits) is really a sort of reflection which says something about the class without regard to how any particular archive class chooses to render it.

Are the archive classes useful without the stuff in the serialization namespace?

...

...

a) Any header from the "serialization" part can be included in a header without triggering requiremetn for the library even in the presense of auto-link.

I don't know what that means. All I know is that you've had some problems apparently related to auto-link.

suppose I have a header x.hpp which includes he auto link stuff.

class X { void f(); // declared - defined in library code void g(){ // declared and defined here. ... } };

Now I make a program which uses x.hpp

#include "x.hpp" int main(... X x; x.g(); }

Auto-link requires the library to be present and linked into the program in order to build. This occurs even though there are no calls to any code in the library. This is conflicts with common practice of declaring functions taht are not always used. Avoiding this situation required some re-organization.

Yep, you need to do the same reorganization for dynamic libraries even if you don't auto-link. Boost.Python had to do it.

...

...

But I'm struck by the fact that lots of people never saw my point of view.

Which point of view?

That including archive headers and serialization headers in the same header module breaks the orthogonality between these concepts and that breaking orthogonality suggests a design error.

What's a "header module?"

...

Sounds like, if you want to support that need while preserving orthogonality, you need an additional layer of indirection in the library design.

Its supported now - but not requried. That seems to be in line with users needs and desires.

I have no idea what this was about anymore.

...

...

I believe that problem 4 is not a problem. Feel free to prove me wrong here.

Already done, if you had only tried to compile the code I posted ;-)

I did - it compiles fine with VC 7.1, Commeau 4.3, Borland 5.64, and gcc 3.3

Well, I clearly wouldn't have posted it if I couldn't get it to fail. Clearly none of those is using 2-phase lookup. Did you look at 14.6.4 Dependent name resolution [temp.dep.res] in the standard like I suggested, or use the Comeau online compiler? -- Dave Abrahams Boost Consulting www.boost-consulting.com

On Sep 9, 2005, at 3:04 AM, Robert Ramey wrote:

David Abrahams wrote:

...

...

I did - it compiles fine with VC 7.1, Commeau 4.3, Borland 5.64, and gcc 3.3

Well, I clearly wouldn't have posted it if I couldn't get it to fail.

Well, you might have made a mistake.

Here's an example that fails on more compilers: namespace me { class X {}; } namespace boost { namespace serialization { void serialize(double); template <class T> int call_serialize(T const& x) { serialize(x); return 0; } void serialize(me::X); }} int y = boost::serialization::call_serialize(me::X()); Note the extra serialize function declared before call_serialize. The IBM xlC compiler issues helpful error messages confirming Dave's point: "test.C", line 14.24: 1540-0256 (S) A parameter of type "double" cannot be initialized with an expression of type "const me::X". "test.C", line 14.24: 1540-1205 (I) The error occurred while converting to parameter 1 of "boost::serialization::serialize(double)". "test.C", line 18.16: 1540-1298 (I) "void serialize(me::X)" needs to be declared in the containing scope to be found by name lookup. "test.C", line 12.15: 1540-0700 (I) The previous message was produced while processing "boost::serialization::call_serialize<me::X>(const X &)". "test.C", line 22.17: 1540-0700 (I) The previous message was produced while processing "y". I hope this helps you understand what is going on, Matthias

It's become clear to me that users of the serialization library are of at least two kinds. a) those who use archives included with the system. This would be analgous to a user of say a transform_iterator who no interest about how its constructed but just wants to use it. He doesn't need to know how its implemented. He just needs to know the requirements on its type arguments and the operations it supports. This is the intent of the Archive Concept section and I believe that it fullfills this role. I see no problem in refining it along the lines of this recent discussion. I might note that no one up to now seems to have had too much difficulty in using the library at this level, so although I don't mind making the changes, I don't think its going to affect things very much in the scheme of things. Of course, it might be that not that many people use the package at all and that's the reason I don't get complaints on this. Its very hard to know. b) those who wish to make thier own archive. This is different and much smaller group of users. It would be analogous to those how want to make new iterator using the facilities of the new iterator library. This requires much more than than just type requirements of some of the classes. I made a family of dataflow iterators so I know first hand about this. This is the intent of the section "Archive Implementation". Having a clearer idea of the role of this part of the documentation will permit me to improve this as well. One could say "Given a clear and unambigous Archive Concept", design of a particular model (archive class) is an implementation detail. And for most users it is. But for users wanting to make their own archive, I want to better describe how to use the included classes. Matthias Troyer wrote:

On Sep 9, 2005, at 3:04 AM, Robert Ramey wrote:

FWIW, I've tried to code a few archive types over the past year and reading the documentation alone was not sufficient. I had to look at the source codes of your archives to see how to actually write an archive. This should not be necessary.

Did you manage to make any new archive types? Robert Ramey

"Robert Ramey" <ramey@rrsd.com> writes:

It's become clear to me that users of the serialization library are of at least two kinds.

a) those who use archives included with the system.

This would be analgous to a user of say a transform_iterator who no interest about how its constructed but just wants to use it. He doesn't need to know how its implemented.

No user should be interested in how transform_iterator is constructed except as an instructive example. In the domain where concepts are useful, instructive examples are icing on the cake, but not crucial. I'm not asking for anything analogous to details about how transform_iterator is implemented, and I don't care how the library-supplied archives are implemented.

He just needs to know the requirements on its type arguments and the operations it supports. This is the intent of the Archive Concept section and I believe that it fullfills this role.

Then you should rename that section. Concepts are not for describing the commonalities of a few components provided by a library (standard container requirements notwithstanding); they're for describing the minimal _requirements_ of other library components: the ones to which models of the concepts are passed. In your case, that means the archive concepts should describe the requirements of the serialization code to which archives are passed. I know you like to think of the archive concept and the serialization library as somehow independent, but that just doesn't reflect reality: serialization makes certain assumptions about the objects passed to it as archives and those assumptions need to be spelled out if there are ever going to be any archives other than the ones you wrote.

I see no problem in refining it along the lines of this recent discussion. I might note that no one up to now seems to have had too much difficulty in using the library at this level, so although I don't mind making the changes, I don't think its going to affect things very much in the scheme of things.

What is your point? That the documentation is really okay the way it is?

Of course, it might be that not that many people use the package at all and that's the reason I don't get complaints on this. Its very hard to know.

If you don't care about making a Generic C++ Library -- in the Generic Programming sense as the discipline was developed by Stepanov and Musser and carried through into the STL and the BGL, etc. -- that's fine, although it would be a terrible missed opportunity. But then you shouldn't be using terms like "concept" in the context of Boost and C++ where it has a very specific meaning, and raises the expectations of your users that they'll be able to create new components that work with your library.

b) those who wish to make thier own archive.

This is different and much smaller group of users.

So?

It would be analogous to those how want to make new iterator using the facilities of the new iterator library. This requires much more than than just type requirements of some of the classes.

No.

I made a family of dataflow iterators so I know first hand about this.

I don't know what you mean by that, but if you tried to use the iterator library to build a new iterator and the library's requirements tables didn't tell you what you needed to know, then it was either an outright bug in the library docs or you didn't understand how to read them properly.

This is the intent of the section "Archive Implementation". Having a clearer idea of the role of this part of the documentation will permit me to improve this as well. One could say "Given a clear and unambigous Archive Concept", design of a particular model (archive class) is an implementation detail.

An implementation detail of what? I don't know what that means. If you replace "an implementation detail" by "possible," then it makes sense to me.

And for most users it is. But for users wanting to make their own archive, I want to better describe how to use the included classes.

I don't see how understanding how to use the included classes is going to help me satisfy the requirements of the serialization library when I need to build a new archive. -- Dave Abrahams Boost Consulting www.boost-consulting.com

Robert, I'm wondering what happened to these threads. Do you understand what I'm saying? Do you think I still don't understand what's going on? David Abrahams <dave@boost-consulting.com> writes:

"Robert Ramey" <ramey@rrsd.com> writes:

...

...

3. In that reference section, '&' is used repeatedly where only '>>' is appropriate. For example:

main(){ ... base *b; ar & b; }

That can only be confusing.

Hmm - I checked that section and maybe it is confusing but not in the way it seems.

the registration or export must be done on BOTH saving and loading of archives.

It doesn't matter, since saving through an unitialized pointer causes undefined behavior. That code can only work for loading.

-- Dave Abrahams Boost Consulting www.boost-consulting.com

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

David Abrahams <dave@boost-consulting.com> writes:

"Robert Ramey" <ramey@rrsd.com> writes:

...

...

3. In that reference section, '&' is used repeatedly where only '>>' is appropriate. For example:

main(){ ... base *b; ar & b; }

That can only be confusing.

Hmm - I checked that section and maybe it is confusing but not in the way it seems.

the registration or export must be done on BOTH saving and loading of archives. It just occurred to me that this might not be obvious. My usage of & rather than << or >> reflects that examples such as the above use ... to replace code which creates and archive instance "ar" which could either be in input and /or output archive. I would be reluctant to replace the & above with << as it would suggest that << would be OK when it wouldn't be in all the cases where the example is applicable.

Um, I never suggested '<<'. Only '>>' is appropriate here.

-- Dave Abrahams Boost Consulting www.boost-consulting.com

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

David Abrahams <dave@boost-consulting.com> writes:

"Robert Ramey" <ramey@rrsd.com> writes:

...

...

IMO you'd be better off simply leaving out "template" on those compilers, if you can get away with it.

ar.register_type<T>()

If I remember correctly, that's what I started out with.

Won't the CVS history tell you for sure?

...

I had to do what I did in order to get all compilers to accept it.

I understand why you need template on _real_ C++ compilers; I also know that vc6 requires that you get all explicit function args into real function args, to avoid link errors, so as long as you write

register_type(T * t = NULL);

in the declaration you can still call

ar.register_type<T>()

I've never heard of one that won't accept the above syntax no matter what you do.

Anyway, workaround hacks should be kept out of the specification of requiremeents for a concept, except as a footnote.

-- Dave Abrahams Boost Consulting www.boost-consulting.com

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

-- Dave Abrahams Boost Consulting www.boost-consulting.com

David Abrahams wrote:

As I said, I understand the general intent. It's the specific meaning that's unclear. It's especally unclear because you say "must implement the following interface..." and then show one member function with an implementation. You could provide that implementation by using a CRTP base class.

OK - I see this now. In fact, in the archives provided, it is implemented using the CRTP model. I concidered that as a feature of the implementation. I think I can adjust this to remove the implementations. I see now that they don't belong here and that it is confusing.

...

Note that his concept is different than others in the standard library in that one of the requirements is that it be a class with certain required member functions.

Most or all other concept definitions only specify operations for which the type is valid. As far as I know this doesn't include supporting specific member functions.

Actually, no. All of the concepts in the Container hierarchy share that property.

Of course. And I did study the SGI documentation in that regard. Now that I think about, I'm sure this is where I got the idea to do it this way.

Well, CRTP might be a good option for you, especially considering that function implementation I mentioned in the first paragraph above.

As I said, Ive used CRTP in the implementation. But I could imagine someone making an archive class that didn't use CRTP and still have the archive function. So I see it as an implementation detail rather than part of the concept.

So the question is, does the user actually need to use default arguments where your interface description does, or would overloads work just as well? Since you don't have a description of how to interpret your prototype, the answer is unclear.

The only default argument is register_type(T * t = NULL); and its only there because many compilers don't accept the syntax: ar.template register_type<T>() Its optional to support compilers that do support the above syntax. So I guess that should be explained. Robert Ramey

IMO you'd be better off simply leaving out "template" on those compilers, if you can get away with it.

ar.register_type<T>()

If I remember correctly, that's what I started out with. I had to do what I did in order to get all compilers to accept it. Robert Ramey

"Robert Ramey" <ramey@rrsd.com> writes:

Thinking about this while I was trying to sleep my thoughts on this clarified a little

There are really two things going on here.

a) There is a use using an archive.

?? sorry, can't parse that.

This is very similar to a user using a collection. He needs a description of the user interface for the collection class. This is essentially a list of requirements on the collection class template arguments and a list of member functions. The achive classes have no template arguments so the archive concept description fills the role. It is also very analogous to the standard class header.

b) There are facilities for creating a new archive class. These would be most analogus to the boost iterator. The archive classes use CRTP. Actually this part has been considered an implementation detail given the common interface to all archives described in the archive concept. Hence its not really described in the documentation at all.

I suspect that part of any confusion is that one is expecting to find b) but finds only a).

If one is looking for a) then he never notices that b) isn't there.

We don't care about any of that. All I -- and the people I'm working with, who can't figure this out either -- care about are the minimal requirements on a type in order for the serialization library to use it as an archive. *That* is the proper definition of the Archive concept. Let me add, if you haven't already got a lot of experience defining concepts rigorously, this is a bad time to try to innovate by using a pseudosignature approach. Write requirements tables using the standard columns, as applicable: valid expression result type requirements semantics preconditions/postconditions invariants notes One of the best ways to make sure you have the right concept specification is to use the Boost Concept Checking Library and define a concept archetype, then throw it at your code. That will at least check that you've got all the syntactic requirements right. Semantic requirements, you need to think through on your own. -- Dave Abrahams Boost Consulting www.boost-consulting.com

"Robert Ramey" <ramey@rrsd.com> writes:

I went back and checked in the tutorial. Previously I had just taken your assertion that it wasn't in there at face value. Under Tutorial/Pointers in the table of contents there is a code snippet and link to the complete demo which shows the serialization of a polymorphic base class. The text doesn't specifically say its a pointer to a polymorphic base class though this is obvious from the link to the full example.

So I think this concern can be addressed by enhancing the explanation in this section of the tutorial.

Great! It's always nice when there's an easy fix. Here's what I think you need to explain, in these words or yours: When you serialize a pointer to an object of polymorphic type, the serialization function for that object's full dynamic type is called. Except in special circumstances, that dynamic type must have been explicitly registered as explained in <link to that section of the reference> before the pointer is serialized, or serialization will fail with an unregistered_class exception. -- Dave Abrahams Boost Consulting www.boost-consulting.com

Rob Stewart <stewart@sig.com> writes:

From: David Abrahams <dave@boost-consulting.com>

...

When you serialize a pointer to an object of polymorphic type, the serialization function for that object's full dynamic type is

How does "full dynamic type" differ from "dynamic type?"

It gives a hint for any programmer who might not have a clue what the latter means.

(I think you should omit "full.")

However, it's your doc, if you choose to use it. Go ahead and omit anything you want if you think it conveys the same information. -- Dave Abrahams Boost Consulting www.boost-consulting.com