On 5/3/07, Anthony Williams <anthony_w.geo@yahoo.com> wrote:

Here I don't understand. Suppose Boost release X contains library A version 1.4, and library B version 1.23, and library A depends on library B. The developer of library B then releases a new version (1.24), and the author of library A doesn't. The next Boost release (Y) comes along, and now library A is no longer part of the Boost release? That strikes me as a bad plan --- the contents of Boost will vary from release to release as developers update their libraries at different rates.

As an alternative, how about this: if library A depends on version xyz of library B, then library B is pinned at version xyz for Boost releases until library A is updated. If library A is not updated for n consecutive Boost releases, library A is dropped from Boost as unmaintained.

How about this, also: a library developer can only release their library if it is built against the latest released version of all its dependent libraries. That way if a core library is updated, all other libraries will have to use the new version before they can release.

Didn't Beman's proposal address most or all of these issues? --Michael Fawcett

"Anthony Williams" <anthony_w.geo@yahoo.com> wrote in message news:ejlxj095.fsf@yahoo.com...

"Gennadiy Rozental" <gennadiy.rozental@thomson.com> writes:

[snipped lots of sensible stuff]

...

VI Boost release

The boost release is done automatically every predefined period of time (like three month). It requires NO testing, NO branching and NO merging/reverting. It's done by single script (a bit oversimplification, but close) that does these tasks:

1. It iterates through all components and check any component that was released since previous boost release. Very important: The component should NOT depend on older version of component that is being released. For example if library A depends on version 1.23 of library B and version 1.24 of library B was released, library A won't become part of the next boost release.

Here I don't understand. Suppose Boost release X contains library A version 1.4, and library B version 1.23, and library A depends on library B. The developer of library B then releases a new version (1.24), and the author of library A doesn't. The next Boost release (Y) comes along, and now library A is no longer part of the Boost release? That strikes me as a bad plan --- the contents of Boost will vary from release to release as developers update their libraries at different rates.

As an alternative, how about this: if library A depends on version xyz of library B, then library B is pinned at version xyz for Boost releases until library A is updated. If library A is not updated for n consecutive Boost releases, library A is dropped from Boost as unmaintained.

Hmm. All good points. How about this: 1. If a) library A depends on concrete version xyz of library B b) B is released library c) library A is not updated library A indeed in this case is not included, since it's intentionally points to the particular release of library B that is not included in new version ob boost anymore. 2. If library A depends on LATEST version of library. As soon as B is released, library A regression test will have to be resumed also (according to the procedure). If by the time we are ready to release boost library A regression is all green we do "automatic promotion/patching". We release patch version of library A that now depends on newly released library B. This process can be automatic. If regression failures are present we do need to postpone library B inclusion into boost release. Library B released in automatically made "not LATEST".

How about this, also: a library developer can only release their library if it is built against the latest released version of all its dependent libraries. That way if a core library is updated, all other libraries will have to use the new version before they can release.

No. I do not wont to restrict developers in how they do their development and what version they depend on. If you opt to depend on concrete version of the library B you need to understand the consequences: If B is released, you will have to do your own release manually or your library is not included in umbrella boost release. Gennadiy

Also, I think all the libraries would benefit from this approach. But it doesnt require that everyone agree. Others are free to continue with the current "tightly coupled" system. The only problem is if the serialization library depends on another whose author makes an interfacing breaking change. This should happen very, very rarely - and in fact doesn't happen often. If it happens too often, I'll reduce my dependency on such libraries. Robert Ramey

Robert Ramey <ramey <at> rrsd.com> writes:

Gennadiy Rozental wrote: [...]

...

I pitched them while ago once. And I still believe that the synchronization is the root of all evils. The only real solution to break this deadloop is independent libraries versioning. This should resolve both out release and testing issues (which are closely connected IMO)

This is on the right track - but seems way too complicated.

Here is what I plan to do from now on: [something along the lines of: develop locally against the latest Boost Official release]

e) When it passes all my tests on the compilers I have. I will do the following: i) check in my changes into whatever tree boost decides it wants to ii) zip up the files which are different from the last boost release and upload the ziped file to a place on my website. The website will contain instructions on how to set up one's include paths so that the latest validated serialization library can be used.

This is going to make it incredibly hard to use in the same project both serialization and any of the libraries that changed since you set up your environment. I believe there are two problems in Gennadiy's proposal: the granularity is too fine and the constraint of releasing Boost in a single whole is going to make things unnecessarily hard. A better approach would be to separate from core Boost some of the larger libraries once and for all. I'm thinking of Serialization, Spirit, Python, possibly a few others. In most cases other libraries should not depend on these (i.e. Preprocessor is not a good candidate). Where dependencies exist they should either be removed or moved to reside within the separate library (e.g. serialization support for core libraries should be supplied by serialization and not core [as I believe it is now, at least in many cases]). These libraries should be developed, tested and released separately, against the most recent release of core. It will be up to each library mantainers' team to decide whether to "port" one or more released versions of their library to new releases of Boost Core, while they work on a new major release. This will help shorten the Core release cycle, while allowing large libraries developers to proceed at their own pace. Users will also benefit from not having to download/install/build libraries they do not need. Cheers, Nicola Musatti

Robert Ramey wrote:

Nicola Musatti wrote: [...]

...

This is going to make it incredibly hard to use in the same project both serialization and any of the libraries that changed since you set up your environment.

Why? one would have on his machine either

a) last boost release version b) or last boost release version + validated upgrades.

The problems of using a separate version of Spirit make me weary. [...]

...

A better approach would be to separate from core Boost some of the larger libraries once and for all. I'm thinking of Serialization, Spirit, Python, possibly a few others. In most cases other libraries should not depend on these (i.e. Preprocessor is not a good candidate).

It seems that this is functionally equivalent to my proposal.

Except that this state of things would be officially recognized across Boost and there would be no separate patches. [...]

So, you're agreeing with me? or not?

I do agree with the idea of developing Serialization (and possibly other libraries) separately from what I called Boost Core. I wish this could happen in a more official, agreed upon way. Cheers, Nicola Musatti

Gennadiy Rozental wrote: [...]

...

I believe there are two problems in Gennadiy's proposal: the granularity is too fine

It's natural separation by independent libraries

I might agree if they really were independent, but in many cases they are not. On the other hand there are a number of rather large libraries that have fewer dependent ones.

...

and the constraint of releasing Boost in a single whole is going to make things unnecessarily hard.

Which constrain?

I'm under the impression that in your scheme you expect to be able to assemble a complete Boost release by choosing the appropriate releases of all the component libraries.

...

A better approach would be to separate from core Boost some of the larger libraries once and for all. I'm thinking of Serialization, Spirit, Python, possibly a few others. In most cases other libraries should not depend on these (i.e. Preprocessor is not a good candidate). Where dependencies exist they should either be removed or moved to reside within the separate library (e.g. serialization support for core libraries should be supplied by serialization and not core [as I believe it is now, at least in many cases]).

These libraries should be developed, tested and released separately, against the most recent release of core. It will be up to each library mantainers' team to decide whether to "port" one or more released versions of their library to new releases of Boost Core, while they work on a new major release.

1. This in no way address problem developing and releasing libraries that other depend on. And this is biggest problem IMO

All that is needed is to shift the release date of the split libraries some 2-3 months after the release of core, assuming a six month release cycle. Core developers will take advantage of the more manageable size of the library collection they work on, while split libraries' developers will gain from the resulting period of Core's guaranteed stability.

2. You proposition leads to the separated librarties to be potencially unusable with latest boost release. This is not a good thing IMO.

People that only use core will be able to switch to a new release immediately; those who need one or more of the separated libraries will have two wait up to three months. On the other hand by reducing Core Boost to a much more manageable size than whole Boost is nice, the chances of hitting planned release dates should increase. If you consider how long people have been waiting for the libraries that were introduced/improved in 1.34, not to mention those that are expected for 1.35...

3. Who make this decision? Which libraries are "core" and which are standalone?

This will have to be agreed upon, considering size, dependencies and breadth of applicability. Ideally library authors should offer to split off their libraries if they think it reasonable. In a way Robert Ramey is already heading in a similar direction with Serialization. I think he should be encouraged to do so, but within an agreed upon setup, rather than in total independence, so that other authors may benefit from the experience gained. Cheers, Nicola Musatti

Peter Dimov wrote:

Sorry for not following the thread properly, just a quick note:

...

...

...

(e.g. serialization support for core libraries should be supplied by serialization and not core [as I believe it is now, at least in many cases]).

This is not right. Serialization support for class X should be provided by X.hpp and it must be possible to do this without depending on any other header.

Appologies from me, too, as now we are getting even further from the original topic. I have seen at least one case where it was indeed X.hpp that introduced the dependency to serialization code. However, this generates another problem: If X is usable without serialization, users shouldn't be forced to also link to the serialization library, just because of this dependency. (In the particular case I have in mind this dependency was controlled by a preprocessor macro. That's not very practical, since packagers surely won't provide two sets of packages for X, one with and one without this dependency.) Thus, I'd suggest to encapsulate the X-serialization functionality into a separate library (may be header-only), such as X_serialization.hpp etc. Then I can still use X stand-alone, and drag in the rest whenever I need it. Thanks, Stefan PS: the library I'm thinking of is boost.wave, and there serialization was dragged in whenever BOOST_WAVE_SERIALIZATION is defined. -- ...ich hab' noch einen Koffer in Berlin...

Emil Dotchevski wrote:

If X is usable without serialization, users shouldn't be forced to also

...

link to the serialization library, just because of this dependency. (In the particular case I have in mind this dependency was controlled by a preprocessor macro. That's not very practical, since packagers surely won't provide two sets of packages for X, one with and one without this dependency.)

Thus, I'd suggest to encapsulate the X-serialization functionality into a separate library (may be header-only), such as X_serialization.hpp etc. Then I can still use X stand-alone, and drag in the rest whenever I need it.

This has been discussed before. You don't need X_serialization.hpp, if you don't use BOOST_CLASS_EXPORT. See http://www.archivesat.com/Boost_developers/thread2900871.htm.

I'm not sure how relevant that is. If X.hpp contains serialization-related code, it surely needs to include serialization header files, too, to drag in any relevant declarations. Thus there is a dependency from X to serialization. Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...

Emil Dotchevski wrote:

...

Emil Dotchevski wrote:

...

If X is usable without serialization, users shouldn't be forced to also

...

link to the serialization library, just because of this dependency. (In the particular case I have in mind this dependency was controlled by a preprocessor macro. That's not very practical, since packagers surely won't provide two sets of packages for X, one with and one without this dependency.)

Thus, I'd suggest to encapsulate the X-serialization functionality into a separate library (may be header-only), such as X_serialization.hpp etc. Then I can still use X stand-alone, and drag in the rest whenever I need it.

This has been discussed before. You don't need X_serialization.hpp, if you don't use BOOST_CLASS_EXPORT. See http://www.archivesat.com/Boost_developers/thread2900871.htm.

I'm not sure how relevant that is. If X.hpp contains serialization-related code, it surely needs to include serialization header files, too, to drag in any relevant declarations. Thus there is a dependency from X to serialization.

You need no declarations because the serialization is implemented as a function template:

class foo;

template<class Archive> void serialize(Archive & ar, foo & x, const unsigned int version) { ar & x.bar; .... }

The above code compiles without including any boost serialization headers.

I have not used boost serialization (I have my own serialization library), but as far as I can see the reason why foo.hpp that uses boost serailization needs to include serialization headers, is to be able to use BOOST_CLASS_EXPORT.

If you serialize classes with base classes, you need to include base_object.hpp, and if you want your serialization to be compatible with XML archives, you have to include nvp.hpp, so it's not possible to completely avoid including serialization headers.

For me this is good enough reason not to use BOOST_CLASS_EXPORT.

If you're about to serialize derived classes via pointers to base classes, BOOST_CLASS_EXPORT is pretty much a must. - Volodya

Stefan,

Appologies from me, too, as now we are getting even further from the original topic.

I have seen at least one case where it was indeed X.hpp that introduced the dependency to serialization code. However, this generates another problem:

If X is usable without serialization, users shouldn't be forced to also link to the serialization library, just because of this dependency. (In the particular case I have in mind this dependency was controlled by a preprocessor macro. That's not very practical, since packagers surely won't provide two sets of packages for X, one with and one without this dependency.)

Thus, I'd suggest to encapsulate the X-serialization functionality into a separate library (may be header-only), such as X_serialization.hpp etc. Then I can still use X stand-alone, and drag in the rest whenever I need it.

I'm not sure, if this is always possible.

PS: the library I'm thinking of is boost.wave, and there serialization was dragged in whenever BOOST_WAVE_SERIALIZATION is defined.

As far as Wave is concerned, the mentioned macro has no influence on the code generated for the library. You always can define BOOST_WAVE_SERIALIZATION to zero when compiling your code, even if the library was compiled with BOOST_WAVE_SERIALIZATION=1. The generated library has no dependency on Boost.Serialization. HTH Regards Hartmut

...

As far as Wave is concerned, the mentioned macro has no influence on the code generated for the library. You always can define BOOST_WAVE_SERIALIZATION to zero when compiling your code, even if the library was compiled with BOOST_WAVE_SERIALIZATION=1. The generated library has no dependency on Boost.Serialization.

Seems to me that unless it's essential to library function serialization support should always be turned off by default.

The job of a load function is to leave the object being loaded in a well defined state -- its invariants intact -- much like a constructor. Therefore, turning this functionality on or off doesn't make any more sense than turning a particular constructor on or off. You can define loading and saving without including any headers from boost serialization. Emil Dotchevski

One more thing. I just had a look at the Wave code. By default serialization support is switched off. So serialization dependencies are generated only if you explicitly specify BOOST_WAVE_SERIALIZATION=1. Regards Hartmut

-----Original Message----- From: Hartmut Kaiser [mailto:hartmut.kaiser@gmail.com] Sent: Friday, May 04, 2007 6:59 PM To: 'boost@lists.boost.org' Subject: RE: [boost] library dependencies

Stefan,

...

Appologies from me, too, as now we are getting even further from the original topic.

I have seen at least one case where it was indeed X.hpp that introduced the dependency to serialization code. However, this generates another problem:

If X is usable without serialization, users shouldn't be forced to also link to the serialization library, just because of this dependency. (In the particular case I have in mind this dependency was controlled by a preprocessor macro. That's not very practical, since packagers surely won't provide two sets of packages for X, one with and one without this dependency.)

Thus, I'd suggest to encapsulate the X-serialization functionality into a separate library (may be header-only), such as X_serialization.hpp etc. Then I can still use X stand-alone, and drag in the rest whenever I need it.

I'm not sure, if this is always possible.

...

PS: the library I'm thinking of is boost.wave, and there serialization was dragged in whenever BOOST_WAVE_SERIALIZATION is defined.

As far as Wave is concerned, the mentioned macro has no influence on the code generated for the library. You always can define BOOST_WAVE_SERIALIZATION to zero when compiling your code, even if the library was compiled with BOOST_WAVE_SERIALIZATION=1. The generated library has no dependency on Boost.Serialization.

HTH Regards Hartmut

Emil Dotchevski wrote:

...

One more thing.

I just had a look at the Wave code. By default serialization support is switched off. So serialization dependencies are generated only if you explicitly specify BOOST_WAVE_SERIALIZATION=1.

Is it possible to refactor the Wave code headers such that they do not invoke BOOST_CLASS_EXPORT, leaving this job to client code that does need to serialize Wave objects? In that case, would you still need to include serialization headers in the Wave code headers?

Wave code headers do not include serialization headers. Regards Hartmut

Emil Dotchevski wrote:

...

...

Is it possible to refactor the Wave code headers such that they do not invoke BOOST_CLASS_EXPORT, leaving this job to client code that does need to serialize Wave objects? In that case, would you still need to include serialization headers in the Wave code headers?

Wave code headers do not include serialization headers.

Regards Hartmut

So what does BOOST_WAVE_SERIALIZATION disable? A few function templates that wouldn't be instantiated unless the client code calls them?

Yes, mainly. Additionally it disables supporting code, such as versioning etc., i.e. code not needed without serialization support. Regards Hartmut

Stefan Seefeld wrote:

...

Wave code headers do not include serialization headers.

http://boost.cvs.sourceforge.net/boost/boost/boost/wave/cpp_co ntext.hpp?revision=1.35&view=markup

conditionally includes boost/serialization/serialization.hpp

Yes, I understand that this include directive is only seen if BOOST_WAVE_SERIALIZATION is defined to something other than 0. But still, I very much doubt packagers would package boost.wave without accounting for that dependency to boost.serialization.

Even if BOOST_WAVE_SERIALIZATION is defined to != 0 during compilation of the library (the *.cpp files) there will be no dependency on Boost.Serialization at runtime as long as the user doesn't define this constant during the compilation of his application. But I'll have a look and try to factor out the direct dependencies on serialization headers into separate headers. Regards Hartmut

Peter Dimov wrote:

Sorry for not following the thread properly, just a quick note:

...

...

...

(e.g. serialization support for core libraries should be supplied by serialization and not core [as I believe it is now, at least in many cases]).

This is not right. Serialization support for class X should be provided by X.hpp and it must be possible to do this without depending on any other header.

I don't agree, except in the rare cases where serialization is one of the core responsibilities of the class. The one thing that I realize is often unavoidable is providing the serialization mechanism access to a class's private state and I consider this rather unfortunate. Cheers, Nicola Musatti

-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Peter Dimov

If your serialization methods

- allow creation of objects whose invariants do not hold, or - expose implementation details in the external representation,

What do you mean by the second line? Sohail

----- Mensaje original ----- De: Peter Dimov <pdimov@mmltd.net> Fecha: Sábado, Mayo 5, 2007 12:17 pm Asunto: Re: [boost] Serialization support,Was: [BoostCon07][testing] Session reminder. Para: boost@lists.boost.org

Sohail Somani wrote:

...

...

-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Peter Dimov

...

If your serialization methods

- allow creation of objects whose invariants do not hold, or - expose implementation details in the external representation,

What do you mean by the second line?

Consider for example a set<int>. The proper way to serialize it is as a sequence of values. This is what the user sees, and this external format is robust against changes in the implementation. If you serialize it as a tree of Node classes, this would make it quite hard to switch to a skip list representation later.

I'd like to join this intrusive vs. non-intrusive serialization discussion from my experience in providing serialization support for Boost.MultiIndex. As I see things, there are two related but different ways in which serialization support can be said to be intrusive: a) data intrusive: the stuff serialized reflects the internal structure of the class. The set<int> example proposed by Peter above illustrates a case of (unwise) data intrusive serialization. b) interface intrusive: the serialization algorithms cannot be implemented by using the class public interface alone. For instance, up to Boost 1.33 (if my memory serves me well), serialization of shared_ptrs was provided in an interface intrusive way because no non-intrusive approach was found --this has fortunately been corrected now. Usually, data intrusive serialization implies interface intrusive serialization, but *not* the other way around: Boost.MultiIndex serialization support is indeed interface intrusive but not data intrusive; let me explain why I had to do things this way. No one doubts non-intrusive serialization is the preferred approach, when feasible. Now consider the process of serializing a multi_index_container: we naturally want deserialization to reconstruct the order in which the elements were arranged in *every* index of the container (hashed indices excluded from this guarantee, as relying in the arbitrary order they provide is unsound to begin with). If we follow the natural approach of saving the sequence of values as traversed by some of the indices (let's say index #0), this guarantee is not held, let's see the problems encountered with each index type: * Ordered indices: no problems with *unique* ordered indices, since the order there is strictly determined by the values contained. But what about non-unique ordered indices? Consider this container: multi_index_container< int, indexed_by< ordered_non_unique<int_modulo_3_extractor>, ordered_non_unique<int_modulo_3_extractor> >

where the elements are deemed equivalent if they have the same value modulo 3. Now suppose we have the values 0,3 and 6 in the container, and index #0 lists them in the following order: 0 3 6 What can we say from this info about the traversal when done through index #1? The answer is: absolutely nothing, whatever permutation of these elements could be validly exposed by index #1. These variations from index #0 result from the fact that the values could have been inserted with hinted insert() through either of both indices. The final state is a convoluted function of the insertion history. * Sequenced and random-access indices: here it is even clearer that the order maintained by these indices cannot be inferred by those of other indices, since after insertion elements can be freely relocated. So, in order to provide our desired level of functionality (perfect reconstruction of every index traversal order) we cannot just save the elements as traversed by index #0, but we must somehow codify the variations from every other index wrt to the traversal order implied by index #0, when these variations are not unique --this is indeed what's done in an efficient way, using LIS (longest increasing subsequence) algorithms. So far, this approach is not data intrusive, since the information stored does not reveal any particular implementation detail and reflects only the nature of traversal orders allowable by index semantics. Now the remaining question is: can we use this info to reconstruct the traversal orders in a non interface intrusive way, i.e. by resorting only to the public interface of each index type? * Ordered indices: No; once an element is inserted into a multi-index container, there is no way (with public interface methods) of changing its order with respect to other equivalent elements in a non-unique ordered index, short of extracting and reinserting the element again, which, besides being terribly inefficient, destroys the relative element position gotten so far in other indices, in a sort of cacth-22 situation. * Sequenced indices: Yes. splice() (or relocate()) facilities can be used to alter the traversal order of a sequenced index without touching the rest of indices. * Random-access indices: Yes, with problems. splice() member functions are available here as for sequenced indices, but relocating elements around is a O(n) operation, implying a O(n*n) complexity for the task of reconstructing the entire index. An interface intrusive scheme can do this in linear time. So, I had no other choice but to implement serialization support for Boost.MultiIndex in an interface intrusive way. The morale of the story is: for rich-state classes where the exact state of an object depends heavily on its past history, non-intrusive serialization can be either algorithmically unfeasible (it is hard or impossible to reconstruct the history from the current state) or potentially less efficient than a interface intrusive approach. This does not mean that the class interface or the serialization support implementation are "broken". Just my 2c, sorry about the long post. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

----- Mensaje original ----- De: Peter Dimov <pdimov@mmltd.net> Fecha: Sábado, Mayo 5, 2007 3:40 pm Asunto: Re: [boost] Serialization support,Was: [BoostCon07][testing] Session reminder. Para: boost@lists.boost.org

"JOAQUIN LOPEZ MU?Z" wrote:

[...]

...

a) data intrusive: the stuff serialized reflects the internal structure of the class. The set<int> example proposed by Peter above illustrates a case of (unwise) data intrusive serialization. b) interface intrusive: the serialization algorithms cannot be implemented by using the class public interface alone. For instance, up to Boost 1.33 (if my memory serves me well), serialization of shared_ptrs was provided in an interface intrusive way because no non-intrusive approach was found --this has fortunately been corrected now.

There is no such thing as "interface intrusive" serialization if you consider the serialization support a part of the documented interface of the class. I admit that this requires additional investment and is rarely done, but it's the only way to do it right.

I'm not getting you: of course if you provide serialization support for a given class T, this support is usually documented and thus becomes part of T's interface; what additional investment are you referring to? What more do I have to document except saying "T is serializable through Boost.Serialization"?

An opaque external representation (or serialization algorithm) always leads to problems in the long run.

What kind of problems? In which sense is treating the external representation as an opaque entity different from treating the *internal* representation as an implementation detail?

(shared_ptr's 1.32 serialization was "data intrusive".)

I stand corrected.

[...]

...

So, I had no other choice but to implement serialization support for Boost.MultiIndex in an interface intrusive way. The morale of the story is: for rich-state classes where the exact state of an object depends heavily on its past history, non-intrusive serialization can be either algorithmically unfeasible (it is hard or impossible to reconstruct the history from the current state) or potentially less efficient than a interface intrusive approach. This does not mean that the class interface or the serialization support implementation are "broken".

If the ability to reconstruct a MultiIndex container in a particular exact state is important, there should exist a documented way to do that. Deserialization is not a special case.

On the contrary, I think deserialization is synonymous with "reconstructing an object in a particular exact state". The only specificity about this is that one choses Boost.Serialization as the reconstructing API instead of something else. But this is as good a choice as any other API (or better, if Boost.Serialization gains momentum as the de facto standard serialization interface, which would be great). So, I think I basically agree with what you say next:

(This doesn't mean that the deserialization support cannot be this documented way, of course.)

Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

Hi Peter, ----- Mensaje original ----- De: Peter Dimov <pdimov@mmltd.net> Fecha: Sábado, Mayo 5, 2007 4:47 pm Asunto: Re: [boost] Serialization support Para: boost@lists.boost.org

"JOAQUIN LOPEZ MU?Z" wrote: [...]

...

I'm not getting you: of course if you provide serialization support for a given class T, this support is usually documented and thus becomes part of T's interface; what additional investment are you referring to? What more do I have to document except saying "T is serializable through Boost.Serialization"?

Well... here's what I would document given a simple class:

template<class A> void serialize( A & a, unsigned );

Effects: a & x; a & y; where x and y are the values returned by x() and y(), respectively.

"T is serializable through Boost.Serialization" doesn't tell me how to manipulate the result. I can neither construct a suitable input file nor read the output. [...] The external representation is observable and persistent. It can outlive both Boost.Serialization and Boost.MultiIndex (or their specific implementations) and it should be possible for an alternate implementation of the MultiIndex interface to read/write the same format. If you use serialization to send a Boost.MI object over the wire to another machine, it should be possible for the non-C++ program on the other end to reconstruct the data structure.

Think about how people would feel when they serialize a std::multiindex with implementation A and can't read it with implementation B, as will happen if the requirement for an implementation is just to be able to read back what it created, not create a particular documented external representation.

Ok now I understand, and basically share, your concerns about the lack of documentation on what goes into the serialization stream. Of course I can document my part in the case of B.MI, but for this to constitute a real cross-platform, cross- implementation spec one would additionally need that: 1. Other serializable types (serialization primitive and otherwise) do the same and document their way of stuffing themselves down the wire. 2. Boost.Serialization exposes its internal mechanism for serialization --I'm thinking here of things like versioning and pointer tracking-- at least to the extent that they reflect on what gets actually saved to the stream. This initiative is interesting (and would be almost mandatory if B.S was ever proposed to the standard) but its scope is community-wide and beyond the realm of an isolated class' definition IMHO. What does Robert think about this? Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

----- Mensaje original ----- De: Rene Rivera <grafikrobot@gmail.com> Fecha: Sábado, Mayo 5, 2007 7:05 pm Asunto: Re: [boost] Serialization support Para: boost@lists.boost.org

JOAQUIN LOPEZ MU?Z wrote:

...

Ok now I understand, and basically share, your concerns about the lack of documentation on what goes into the serialization stream.

I'm not sure you see the full scope of what Peter is raising, or perhaps I'm reading more into it :-) It is not enough to document the serialization for, nor the serialization procedure, as that leaves out non-Boost Serialization implementations.

Well the idea is, you can leverage the B.S interface to add yout non-Boost serialization support. As its crudest, you can do the following: void non_boost_save(const T& t,non_boost_archive& a) { std::ostringstream oss; { boost::archive::text_oarchive oa(oss); oa<<t; } a.save(os.str()); } Something more sensible could be done by defining your own utility B.S Archive class, you get the idea.

I happen to have one of those non-Boost implementations, and it's implemented non-intrusively. In thinking about the scope of designing a serializable class, I consider the case of writing an external copy algorithm. If I can't write a copy function that given one instance will create an equivalent instance, then the class isn't usefully serializable.

Here you lost me. I know you're not referring to the following, but from your description looks like you're asking for a function T create_copy(const T& t) { return t; } which of course is readily available whenever T is copy-constructible. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

"JOAQUIN LOPEZ MU?Z" wrote:

Ok now I understand, and basically share, your concerns about the lack of documentation on what goes into the serialization stream. Of course I can document my part in the case of B.MI, but for this to constitute a real cross-platform, cross- implementation spec one would additionally need that:

1. Other serializable types (serialization primitive and otherwise) do the same and document their way of stuffing themselves down the wire. 2. Boost.Serialization exposes its internal mechanism for serialization --I'm thinking here of things like versioning and pointer tracking-- at least to the extent that they reflect on what gets actually saved to the stream.

This initiative is interesting (and would be almost mandatory if B.S was ever proposed to the standard) but its scope is community-wide and beyond the realm of an isolated class' definition IMHO. What does Robert think about this?

My stated goal was to permit the serialization an de-serialization of any group of C++ data structures in the most expendient way. I wanted the system to be simple to use, complete and efficient. To this end, I made a determined effort to separate the external aspects in the "archives" and description of the class attributes related to serialization in "serialization" I refrained from and a-priori definition and description of the external format for a couple of reasons: a) it was too hard - I would require a huge amount of work and forsight. b) I believed - and still believe - that it conflicted with my stated goals above. c) I didn't think that the investment of time would be worth it in brining the package "to market" After some experience in seeing how the package is being used and how hard/easy it is to use, I don't regret my decisions. If one want's to add an externallly defined language independent format to the above goals, I think one will be doomed to failure. Of course I could be wrong and anyone is free to take a crack at it. Lots of people have. I'm not sure how all the other systems out there compare to boost serialization these days. So I don't see an externally documented format for this boost serialization. Hence I don't see anything like boost serialization ever appearing in the standard. Perhaps some system which might functionally similar but I think it would have to be grown from scratch with a different set of goals and priorities. Which is the reason that I think the whole concept of library standards have been over-applied and even detremental to the future success of C++. Robert Ramey

Jeff Garland wrote:

BTW, I have seen the lack of a documented format become a reason to not use Boost serialization on a project.

No doubt this has happened - and for good reason. I such a format is required - This "right choice" is probably not boost serialization. Boost Serializaition has (in my view) lots of appeal - and its applicable to a very wide class of problems. But it can't be all things to all people (an all host languages). Sorry, but I think different tool developed with a different set of priority goals is required for that.

...

Which is the reason that I think the whole concept of library standards have been over-applied and even detremental to the future success of C++.

This too I believe is wrong. Every human system of significance rests on standards. You and I couldn't be conversing now if we didn't have a pile of IEEE standards, posix standards, W3C standards, and yes, programming standards. For C++, there is a real effect of having something in the standard -- companies that refuse to use Boost will insist on the use of ISO standard C++.

I know my view on this subject isn't widely held. We've touched upon it before and maybe we will again - but for now we're frying other fish. Robert Ramey

Peter Dimov wrote: If you use

serialization to send a Boost.MI object over the wire to another machine, it should be possible for the non-C++ program on the other end to reconstruct the data structure.

Not every language can support all the data structures that C++ supports. Robert Ramey

----- Mensaje original ----- De: Jeff Garland <jeff@crystalclearsoftware.com> Fecha: Sábado, Mayo 5, 2007 5:14 pm Asunto: Re: [boost] Serialization support, Was: [BoostCon07][testing] Session reminder. Para: boost@lists.boost.org

JOAQUIN LOPEZ MU?Z wrote:

...

So, I had no other choice but to implement serialization support for Boost.MultiIndex in an interface intrusive way. [...] Thx for explaining this. I'd like to go back the the header inclusion dependency issue. The need for internal access isn't a reason why the serialization headers need to be 'included by default'. A friend class/function in separate serialization header could implement the serialization using the internal mechanisms that are otherwise an implementation detail.

Hello Jeff, Yep, the need for intrusive serialization support does not preclude its relocation to separate serialization headers. In the discussion we had two months ago I gave my rationales for not using separate headers, let me quote that for the readers' convenience: <quote> Well, I thought about this kind of problems when designing the serialization support of B.MI, and what you've got is the best I came up with. There is a rationale for not having boost/multi_index/serialization/*_index.hpp headers, let me explain: When you serialize a multi_index_container comprised of N indices, every index gets involved in the serialization process; so, if you have something like: tyepedef multi_index_container< element, indexed_by< ordered_unique<...>, hashed_non_unique<...>, sequenced<...> >

mic_t;

and want to serialize objects of type mic_t, you'd have (according to the serialization header model) to include the following: #include <boost/multi_index/serialization.hpp> #include <boost/multi_index/serialization/ordered_index.hpp> #include <boost/multi_index/serialization/hashed_index.hpp> #include <boost/multi_index/serialization/sequenced_index.hpp> Which looks like excessively cumbersome to me. Failing to include one of the headers won't result in less-capable serialization support, only in a compile-time error when trying to stream mic_t's. A twist on this could be: why not embed all the serialization support in one centralized header? #include <boost/multi_index/serialization.hpp> This is convenient from the user's point of view, but doesn't scale up well internally, cause adding a new type of index would cause this header to grow regardless of whether the new type of index is used or not. As things stand now, index types are entirely orthogonal with each other in terms of code base, which is good. So, my decision was to embed serialization support for each index (and the multi_index_container wrapper itself) directly in their corresponding headers, and I left the disabling macro just in case. Faced with the dilemma of whether this support should be on or off by default, I opted for "on" because, thanks to the autolinking feature, when the serialization capabilities are not invoked the user does not notice anything, perhaps some theoretical slowdown in compile times. </quote> What's your stance on this? If there's some agreement that the actual approach should be changed I can of course do it --it'd only raise small backwards-compatibility glitches. BTW, if the separate header is agreed on as the preferred mechanism for bringing in serialization capabilities, I think it'd be great to reach a consensus on the names of those convenience headers; there are various options: 1. boost/lib/T_serialization.hpp 2. boost/lib/T_serialize.hpp (date_time approach) 3. boost/lib/serialization/T.hpp (consistent with B.S approach) I like 3 best because _serialization or _serialize suffixes tend to give long file names, and we've got the 31 char limit rule still in effect. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

"JOAQUIN LOPEZ MU?Z" wrote:

Just my 2c, sorry about the long post.

LOL - as usual - another tour d'force. Robert Ramey

Nicola Musatti wrote:

Peter Dimov wrote: [...]

...

If your serialization methods

- allow creation of objects whose invariants do not hold, or

Of course. This is analogous to writing a buggy copy constructor.

...

- expose implementation details in the external representation,

This is more tricky: you may want to serialize an object which is in a state that in normal circumstances is not the direct outcome of construction, but rather is the result of some activity. How can you manage that? Either you provide a special purpose constructor or you open up the object's internal state.

You store information that allows you to reconstruct the state. There is no need to open the object's internal state, and it is not entirely clear what you mean by that.

"Nicola Musatti" <Nicola.Musatti@gmail.com> wrote in message news:f1frd3$jm8$1@sea.gmane.org...

Gennadiy Rozental wrote: [...]

...

...

I believe there are two problems in Gennadiy's proposal: the granularity is too fine

It's natural separation by independent libraries

I might agree if they really were independent, but in many cases they are not. On the other hand there are a number of rather large libraries that have fewer dependent ones.

Sorry. I was a bit unclear. By independent I mean "independently developed".

...

...

and the constraint of releasing Boost in a single whole is going to make things unnecessarily hard.

Which constrain?

I'm under the impression that in your scheme you expect to be able to assemble a complete Boost release by choosing the appropriate releases of all the component libraries.

Yes. So what is the problem?

...

...

A better approach would be to separate from core Boost some of the larger libraries once and for all. I'm thinking of Serialization, Spirit, Python, possibly a few others. In most cases other libraries should not depend on these (i.e. Preprocessor is not a good candidate). Where dependencies exist they should either be removed or moved to reside within the separate library (e.g. serialization support for core libraries should be supplied by serialization and not core [as I believe it is now, at least in many cases]).

These libraries should be developed, tested and released separately, against the most recent release of core. It will be up to each library mantainers' team to decide whether to "port" one or more released versions of their library to new releases of Boost Core, while they work on a new major release.

1. This in no way address problem developing and releasing libraries that other depend on. And this is biggest problem IMO

All that is needed is to shift the release date of the split libraries some 2-3 months after the release of core, assuming a six month release

What if we want to release "core" every 3 month? But what is more important there is independently developed libraries within core.

cycle. Core developers will take advantage of the more manageable size of the library collection they work on, while split libraries'

What advantage? Each developer really care about one's own library size and dependencies. Why do you believe that be splitting couple libraries we achieve anything?

developers will gain from the resulting period of Core's guaranteed stability.

What stability? Core is stable to 6 month next release library A is updated. For 6 month separated library B tested against outdated version and it become unusable once next "core" release occurs. It takes another couple month to make another release of library B compatible with A, only to become invalid again in a month or so when next version of "core" is released.

...

2. You proposition leads to the separated libraries to be potentially unusable with latest boost release. This is not a good thing IMO.

People that only use core will be able to switch to a new release immediately; those who need one or more of the separated libraries will have two wait up to three months. On the other hand by reducing Core

And something will always be missing: either update to separate library or new feature from "core" one.

Boost to a much more manageable size than whole Boost is nice, the chances of hitting planned release dates should increase. If you

How splitting couple libs will make it "much more manageable"? And if you split let's say 2/3 of libs how is it different from what I propose?

consider how long people have been waiting for the libraries that were introduced/improved in 1.34, not to mention those that are expected for 1.35...

...

3. Who make this decision? Which libraries are "core" and which are standalone?

This will have to be agreed upon, considering size, dependencies and breadth of applicability.

This will never going to be agreed upon.

Ideally library authors should offer to split off their libraries if they think it reasonable. In a way Robert Ramey is already heading in a similar direction with Serialization. I think he should be encouraged to do so, but within an agreed upon setup, rather than in total independence, so that other authors may benefit from the experience gained.

There is no real incentive for a library author to do extra work of keeping up with independent "core" releases. In my scheme you can't add "core" library release to the boost release until all the libs that depend on LATEST version of the library are tested against it. What you propose essentially leave every developer of separated libraries on their own. While the same time developers of the "core" libraries are still faces with current interdependencies problem. Gennadiy

Nicola Musatti wrote:

Gennadiy Rozental wrote:

Why do you think so? Spirit is already been developed independently and Robert R. clearly intends to do the same with Serialization. I'm sure that for them it would be easier to release their libraries against a stable release of the rest of Boost than against a moving target.

I wouldn't call what I plan to do as develop independently. I just want run my own tests against a stable code base that everyone has and make the changes available to anyone who wants them. I expect to check in changes with whatever system boost decides to use. A side effect of this is that I make the defaut test set smaller and add the tests that I now need to add but cannot because they take too long. DOES ANYONE OBJECT TO MY DOING THIS? WHY? Does this in anyway conflict with any of the proposals made so far? Anyone who doesn't agree with this can just wait for the normal boost process - he will see no change and won't have to be aware that I've tested stuff on my on machine and permitted those who are interested in using the lastest version. Robert Ramey

David Abrahams wrote:

...

DOES ANYONE OBJECT TO MY DOING THIS? WHY?

If you are going to "make the changes available to anyone who wants them" by checking them into a branch in the Boost SVN repo, I think it's fine. In fact, I think it will pretty much line up with Beman's proposal.

Actually, what I was thinking would be to create a *.zip file of the hanged files. As I explained before and has been suggested by other authors, local changes/updates/modifications of the Boost can be handled by including a "shadow" tree of include files before the tree of the official boost release. Robert Ramey

David Abrahams wrote:

on Mon May 07 2007, "Robert Ramey" <ramey-AT-rrsd.com> wrote:

...

David Abrahams wrote:

...

...

DOES ANYONE OBJECT TO MY DOING THIS? WHY?

If you are going to "make the changes available to anyone who wants them" by checking them into a branch in the Boost SVN repo, I think it's fine. In fact, I think it will pretty much line up with Beman's proposal.

Actually, what I was thinking would be to create a *.zip file of the hanged files.

I meant making the *.zip file in addition to checking into the head. The HEAD and its history would be as it always has been. Robert Ramey

Then I object. Please use the source control tools, if only so we have a historical record.

"Nicola Musatti" <Nicola.Musatti@gmail.com> wrote in message 
news:f1gb29$b3s$1@sea.gmane.org...
> I'm convinced that one of the problems that lead to 1.34 not being
> released for over a year beyond the original schedule has to do with the
> size and complexity of Boost as a whole and coordinating the personal
> schedules of many developers to reach a single coordinated goal. The one

My proposal doesn't require ANY coordination during boost release. And 
that's the key. Once time has come you collect what's ready at this time, 
combine and release. No testing is done at the time of boost release 
whatsoever. And no waiting.

> advantage of this situation is that each developer has been able to
> react to changes as soon as s/he realized that her/his library was 
> impacted.
>
> By splitting at the library level, but expecting to still issue one
> coherent whole you don't solve the problem, but you give up the one
> advantage. I don't see what your approach would gain us.

I am still in a dark: what exactly did I give up?

>>>>> A better approach would be to separate from core Boost some of the
>>>>> larger
>>>>> libraries once and for all. I'm thinking of Serialization, Spirit,
>>>>> Python,
>>>>> possibly a few others. In most cases other libraries should not depend
>>>>> on
>>>>> these
>>>>> (i.e. Preprocessor is not a good candidate). Where dependencies exist
>>>>> they
>>>>> should either be removed or moved to reside within the separate 
>>>>> library
>>>>> (e.g.
>>>>> serialization support for core libraries should be supplied by
>>>>> serialization and
>>>>> not core [as I believe it is now, at least in many cases]).
>>>>>
>>>>> These libraries should be developed, tested and released separately,
>>>>> against the
>>>>> most recent release of core. It will be up to each library mantainers'
>>>>> team to
>>>>> decide whether to "port" one or more released versions of their 
>>>>> library
>>>>> to
>>>>> new
>>>>> releases of Boost Core, while they work on a new major release.
>>>> 1. This in no way address problem developing and releasing libraries 
>>>> that
>>>> other depend on. And this is biggest problem IMO
>>> All that is needed is to shift the release date of the split libraries
>>> some 2-3 months after the release of core, assuming a six month release
>>
>> What if we want to release "core" every 3 month? But what is more 
>> important
>> there is independently developed libraries within core.
>
> So? At worst users of specific libraries will be one core release
> behind. They'll have to face the trade off of being able to use the one
> library they need, but having to wait some more time before they can
> adopt some newly introduced core feature. Assuming a more realistic six
> month release cycle for core plus an extra three month for split
> libraries these people would still be much better of than those that
> right now would like to use a library that was reviewed a year ago.

No. It will never work from user's prospective IMO. I am using boost xyz. I 
need lib A. Now I am stack and can't upgrade just because library A 
developer has no incentive (or just busy) to build against next version of 
boost.

>>> cycle. Core developers will take advantage of the more manageable size
>>> of the library collection they work on, while split libraries'
>>
>> What advantage? Each developer really care about one's own library size 
>> and
>> dependencies. Why do you believe that be splitting couple libraries we
>> achieve anything?
>
> For instance people not using those libraries won't have to wait for
> them to reach stability before a release can be made.

What about those who do need them? They will always be in "one (at least) 
release too late" position.

> By just setting
> apart Serialization regression testing would take half the time, leading

Today serialization. Tomorrow GUI lib, XML parser, HTTP server etc.

> to a much shorter turn around time. The problem is rather that the split
> libraries would need testing resources too.

Yes. Testing. You did not address it. It's completely unclear how do you 
plan to test separated library that depends on some long back release.

>>> developers will gain from the resulting period of Core's guaranteed
>>> stability.
>>
>> What stability? Core is stable to 6 month next release library A is 
>> updated.
>> For 6 month separated library B tested against outdated version and it
>> become unusable once next "core" release occurs. It takes another couple
>> month to make another release of library B compatible with A, only to 
>> become
>> invalid again in a month or so when next version of "core" is released.
>
> Again, this a problem we keep having to face whenever we have to use a
> piece of software that depends on others.

No. Boost release *has to be consistent*. Otherwise separated component is 
just another third-party library we don't care about.

>>>> 2. You proposition leads to the separated libraries to be potentially
>>>> unusable with latest boost release. This is not a good thing IMO.
>>> People that only use core will be able to switch to a new release
>>> immediately; those who need one or more of the separated libraries will
>>> have two wait up to three months. On the other hand by reducing Core
>>
>> And something will always be  missing: either update to separate library 
>> or
>> new feature from "core" one.
>
> If the result is a shorter overall cycle than we have now,, this will be
> a non problem.

It is. Large number of users tend to stick to single version and use it 
until they jump to another. You propose me to have 10 different versions of 
boost "core" for 10 separated libraries developed at different speed.

>>> Boost to a much more manageable size than whole Boost is nice, the
>>> chances of hitting planned release dates should increase. If you
>>
>> How splitting couple libs will make it "much more manageable"? And if you
>> split let's say 2/3 of libs how is it different from what I propose?
>
> Again, just as an example, consider splitting off Serialization, Spirit,
> Wave, Python, Mpi. This not just a couple of libs, it's a significant
> portion of the Boost code base, and you still don't have to find out
> which release of Bind goes with which release of Function and so on.

1. What about long term? Big libraries will continue to appear in boost.
2. How do you decide which library qualify for separation? Does Multi-index 
big-enough or complex enough? How about Date/Time? Or MPL? Regex? Thread? 
One of the biggest is Preprocessor. Does it qualify? On the other hand there 
many small components that other libraries has very little dependency upon. 
Why not separate them?

>>> consider how long people have been waiting for the libraries that were
>>> introduced/improved in 1.34, not to mention those that are expected for
>>> 1.35...
>>>
>>>> 3. Who make this decision? Which libraries are "core" and which are
>>>> standalone?
>>> This will have to be agreed upon, considering size, dependencies and
>>> breadth of applicability.
>>
>> This will never going to be agreed upon.
>
> Why do you think so? Spirit is already been developed independently and
> Robert R. clearly intends to do the same with Serialization. I'm sure
> that for them it would be easier to release their libraries against a
> stable release of the rest of Boost than against a moving target.

IMO what you proposing is a moving target ;). The examples you provide, just 
prove my point: independent library development is the way to go. The only 
difference is that you propose strange (IMO) combination of existing mess 
and proper way. If we decide that independent development is the way to go, 
why not make it a rule: all libraries are developed independently.

>>> Ideally library authors should offer to split
>>> off their libraries if they think it reasonable. In a way Robert Ramey
>>> is already heading in a similar direction with Serialization. I think he
>>> should be encouraged to do so, but within an agreed upon setup, rather
>>> than in total independence, so that other authors may benefit from the
>>> experience gained.
>>
>> There is no real incentive for a library author to do extra work of 
>> keeping
>> up with independent "core" releases. In my scheme you can't add "core"
>> library release to the boost release until all the libs that depend on
>> LATEST version of the library are tested against it. What you propose
>> essentially leave every developer of separated libraries on their own. 
>> While
>> the same time developers of the "core" libraries are still faces with
>> current interdependencies problem.
>
> Keeping up with independent core releases is *less* work than keeping up
>  with a continuously moving target and the reason I'm writing all this
> is because I don't *want* split library authors to be left on their own
> or to go their separate ways. Rather I want Boost to be split into a
> small number of packages and I'm convinced that this would improve
> noticeably the manageability of the whole lot.

Where did you see a moving target in my proposal?

In my makefile I specify: my lib depends on A:abc B:xyz. It can be concrete 
versions for a long time. Once my development is completed, I may switch to 
the LATEST versions of dependent components. With some luck it just 
compiles, otherwise I had to make fixes to comply. Alternatively I may 
decide that I do not want to continue development and don't switch to 
LATEST. In later case my library is not included in next version of boost. 
The only tough case is if library A that depends on LATEST version of 
library B and it was not fixed. Who is at fault here? My proposal is 
postpone library A release. There will be come complications here. But I 
believe they all can be resolved.
   The key advantage above is separation of my own development from 
development of dependent libraries.

Gennadiy

"Nicola Musatti" <Nicola.Musatti@gmail.com> wrote in message news:f1hdjg$pnq$1@sea.gmane.org...

Rereading your proposal I realize that what I have in mind is not all that different:

First, I think that there's a set of libraries that appear to be too closely interconnected to handle them as separate components; I'm thinking of bind/function/signals, but there may be others.

Yes. I agree. I also thought that some small numeber of mosr widely used components should constitute Boost.Core component. Boost.Signals I would split though. It's special purpose (though generic) library. Smart_ptr/config/utility/type_traits are good candidates for core.

Second, I wish that Boost was also made available as a small number of packages, so that users could choose to only install those that they actually need.

Yes. I believe we will need to consider procedure for packaging of independent library releases. This is a second stage though.

For the rest, I'm convinced that the techniques you propose stand an even better chance of working at a coarser degree of granularity.

Gennadiy

Gennadiy Rozental wrote:

"Robert Ramey" <ramey@rrsd.com> wrote in message This doesn't address neither testing against new version of your library by developers of other libraies

Sure it does - they can either test against the HEAD as they do now or they update their local installation

nor dealing with testing agains updates of component your library depends on.

This amounts to using the tests for one library as tests of the dependent library. There are two problems with this. First its very inefficient and arbitrary as the tests in my library aren't presume that the rest of boost works so they aren't designed to test the rest of boost. And if an error occurs then it has to be determined which of the components, libraries, build system, etc which are simultaneously changing is the cause of the failure. This wastes huge amount of time. That being said. If the rest of boost things that the serialization library is a good way test the other libraries - fine with me. Its not an issue for me at all.

Formalization of independent versioning should allow us deal with it.

I think boost as a concept has to change from tightly coupled group of "standard" libraries to a loosely coupled group of "interoperable" libraries. At its core I think this is the issue. Good news is that its not something that we have to agree upon. Its happening now as it must as boost get beyond the scale where it can be digested as a whole. Robert Ramey

on Thu May 03 2007, "Gennadiy Rozental" <gennadiy.rozental-AT-thomson.com> wrote:

Hi,

I do plan to attend this session. I've got some ideas on the subject. I pitched them while ago once. And I still believe that the synchronization is the root of all evils. The only real solution to break this deadloop is independent libraries versioning. This should resolve both out release and testing issues (which are closely connected IMO)

INDEPENDENT LIBRARY VERTIONING

I just want to point out that the BoostCon "Testing Boost" sprint is about our testing processes and infrastructure, not about the structure of Boost itself and its release process. I agree that the latter is an important topic and I hope we'll discuss it at BoostCon, but it's a much tougher issue (to form consensus on, and to solve) than is the topic of the sprint. In order to ensure that the sprint is successful, I strongly request that in Rene's session, we concentrate on the topic at hand. Thank you, -- Dave Abrahams Boost Consulting www.boost-consulting.com Don't Miss BoostCon 2007! ==> http://www.boostcon.com

"David Abrahams" <dave@boost-consulting.com> wrote in message news:87lkg4d5sj.fsf@valverde.peloton...

on Thu May 03 2007, "Gennadiy Rozental" <gennadiy.rozental-AT-thomson.com> wrote:

...

Hi,

I do plan to attend this session. I've got some ideas on the subject. I pitched them while ago once. And I still believe that the synchronization is the root of all evils. The only real solution to break this deadloop is independent libraries versioning. This should resolve both out release and testing issues (which are closely connected IMO)

INDEPENDENT LIBRARY VERTIONING

I just want to point out that the BoostCon "Testing Boost" sprint is about our testing processes and infrastructure, not about the structure of Boost itself and its release process. I agree that the latter is an important topic and I hope we'll discuss it at BoostCon, but it's a much tougher issue (to form consensus on, and to solve) than is the topic of the sprint. In order to ensure that the sprint is successful,

I strongly request that in Rene's session, we concentrate on the topic at hand.

While in general I agree, IMO discussing how we are going to test the libraries before deciding how the process is organized in general is like putting cart in front of a horse. Part of my proposition directly affect the way testing needs to be organized. Gennadiy.

David Abrahams wrote:

Okay, here are some issues I think ought to be solved, in no particular order, some (much) more important than others:

David, I'm happy to see you bring up much the same points I mentioned. Here is one I haven't yet got to, and would like to explore a bit further:

* generating XML by parsing the jam log is fragile and prevents the use of multiple build processes (-jN). This one should be almost embarassingly easy to fix.

I think there is more to this than only the ability to run tests in parallel. For example, it would help to robustify the testing harness if the 'test database' could be inspected without actually executing any test. By that I mean the ability to: * See all tests, as part of the test database structure (i.e. their organization into test suites) * See meta data associated with tests, such as - what kind of test - expected outcome, per platform - dependencies, prerequisites, etc. In fact, this wishlist is influenced by the fact that I'm working on QMTest (http://www.codesourcery.com/qmtest), where these aspects are an important part of the overall design. (In fact, I have been trying to convince Vladimir to make it possible to hook bbv2 (at least the part related to testing) up with QMTest in support of the above features. Note also that test report generation and a graphical user interface are an integral part of QMTest.) Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...

David Abrahams wrote:

...

* See all tests, as part of the test database structure (i.e. their organization into test suites)

* See meta data associated with tests, such as

- what kind of test - expected outcome, per platform - dependencies, prerequisites, etc.

How would that help with robustness?

There have been a number of cases where the regression harness picked up (and reported) stale results, just because old executables / results were lying around in the build tree. It didn't 'know' that the test actually had been removed from the test suite. But even in a somewhat broader sense, being able to inspect the test database with all its metadata IMO contributes to robustness. Robustness through transparency... Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...

(sorry, dropped the cc by mistake) David Abrahams wrote:

Okay, here are some issues I think ought to be solved, in no particular order, some (much) more important than others:

[...] * No way for a test to fail with a "skipped" status (gray box). This is similar to

* no mechanism for communicating that tests are known not to work on some platform (toolset+os) to the build system, and so avoid building them

but it can rely on Boost.Config macros, which may not be easily accessible to the build system.

on Fri May 04 2007, "Gennadiy Rozental" <gennadiy.rozental-AT-thomson.com> wrote:

"David Abrahams" <dave@boost-consulting.com> wrote in message news:876478d51y.fsf@valverde.peloton...

...

* No way for a developer to request testing results for a specific branch

And against particular branch.

How is "for a specific branch" different from "against a particular branch?" it seems as though you're repeating what I meant to say (?)

That's what independent versioning should address.

I don't see a relationship between the two issues. The ability to request test results against a specific branch has obvious utility; you can try out changes and see how portable they are without disturbing general test results. Independent versioning, by it self, does not address that need. You'd still need to implement all the same testing infrastructure, so I don't see the relevance here. -- Dave Abrahams Boost Consulting www.boost-consulting.com Don't Miss BoostCon 2007! ==> http://www.boostcon.com

on Sat May 05 2007, "Gennadiy Rozental" <gennadiy.rozental-AT-thomson.com> wrote:

"David Abrahams" <dave@boost-consulting.com> wrote in message news:87zm4k87rw.fsf@valverde.peloton...

...

on Fri May 04 2007, "Gennadiy Rozental" <gennadiy.rozental-AT-thomson.com> wrote:

...

"David Abrahams" <dave@boost-consulting.com> wrote in message news:876478d51y.fsf@valverde.peloton...

...

* No way for a developer to request testing results for a specific branch

And against particular branch.

How is "for a specific branch" different from "against a particular branch?" it seems as though you're repeating what I meant to say (?)

My library depend in lib A, B, C. I want to test against new_dev branch of A, HEAD branch of B and 1_33 branch of C.

So assemble a branch that contains everything you want, check it in, and request the test. SVN is really good for that sort of thing because of the way it copies "by reference".

...

...

That's what independent versioning should address.

I don't see a relationship between the two issues. The ability to request test results against a specific branch has obvious utility; you can try out changes and see how portable they are without disturbing general test results. Independent versioning, by it self, does not address that need. You'd still need to implement all the same testing infrastructure, so I don't see the relevance here.

Yes. independent versioning has little to do with "testing for a specific branch". But has everything to do with "testing against particular branch" ;)

I still don't see it, sorry. -- Dave Abrahams Boost Consulting www.boost-consulting.com Don't Miss BoostCon 2007! ==> http://www.boostcon.com

David Abrahams wrote:

on Sat May 05 2007, "Gennadiy Rozental" <gennadiy.rozental-AT-thomson.com> wrote:

...

"David Abrahams" <dave@boost-consulting.com> wrote in message news:87zm4k87rw.fsf@valverde.peloton...

...

on Fri May 04 2007, "Gennadiy Rozental" <gennadiy.rozental-AT-thomson.com> wrote:

...

"David Abrahams" <dave@boost-consulting.com> wrote in message news:876478d51y.fsf@valverde.peloton...

...

* No way for a developer to request testing results for a specific branch And against particular branch. How is "for a specific branch" different from "against a particular branch?" it seems as though you're repeating what I meant to say (?) My library depend in lib A, B, C. I want to test against new_dev branch of A, HEAD branch of B and 1_33 branch of C.

So assemble a branch that contains everything you want, check it in, and request the test. SVN is really good for that sort of thing because of the way it copies "by reference".

Switching branches for testing practically implies non-incremental testing. That can't be done efficiently without significant changes to how we run tests or to the amount of hardware available for testing. Regards, m

on Sun May 06 2007, "Gennadiy Rozental" <gennadiy.rozental-AT-thomson.com> wrote:

"David Abrahams" <dave@boost-consulting.com> wrote in message news:87tzur78cq.fsf@valverde.peloton...

...

on Sat May 05 2007, "Gennadiy Rozental" <gennadiy.rozental-AT-thomson.com> wrote:

...

My library depend in lib A, B, C. I want to test against new_dev branch of A, HEAD branch of B and 1_33 branch of C.

So assemble a branch that contains everything you want, check it in, and request the test. SVN is really good for that sort of thing because of the way it copies "by reference".

Does it means that my branch will keep track of changes made in referenced branches? If not it's useless in practice.

You could do that with svn:externals.