[modularization] Improving/splitting up detail

John Maddock

1 Nov 2013 1 Nov '13

7:15 p.m.

Lets say we wanted boost/detail to depend on nothing but config, here's what it would take: boost/detail/algorithm.hpp Used by graph lib only, suggest it's moved there. boost/detail/allocator_utilities.hpp Used by multi index, statechart and flyweight, suggest move to new "common" module. boost/detail/atomic_count.hpp Depends on smart_ptr's details only, suggest those details are moved to "detail" module. boost/detail/binary_search.hpp Used by python and test, move to "common". boost/detail/bitmask.hpp Only external dependency is boost/cstdint.hpp, either move to "common", or else move cstdint.hpp to "config". boost/detail/call_traits.hpp Move to call_traits (should be there already). boost/detail/catch_exceptions.hpp Only external dependency is boost/cstdint.hpp, suggest latter is moved to config. boost/detail/compressed_pair.hpp Move to compressed pair (should be there already). boost/detail/endian.hpp Move to "predef" boost/detail/has_default_constructor.hpp File appears to be unused, and not very useful, suggest we remove it. boost/detail/identifier.hpp File appears to be unused, suggest we remove it. boost/detail/indirect_traits.hpp Used by iterator and python, move to "common" boost/detail/is_incrementable.hpp Move to common, or better modify the libraies using this to use the newer type_traits equivalents. boost/detail/is_sorted.hpp Used by graph and range, move to "common" boost/detail/is_xxx.hpp Used by parameter and python, move to "common" boost/detail/iterator.hpp I suspect this may be obsolete now compiler requirements have been bumped, but whatever it's still used by multiple libs: move to "common" boost/detail/lcast_precision.hpp Only used by lexical_cast, move it there. boost/detail/lightweight_mutex.hpp Only extern dependency is smart_ptr's details, suggest move those to detail. boost/detail/named_template_params.hpp Appears to be unused, suggest we remove. boost/detail/numeric_traits.hpp Used by graph and iterator, move to "common" boost/detail/ob_compressed_pair.hpp Move to compressed pair, might also be obsolete by now. boost/detail/quick_allocator.hpp Only external dependency is smart_ptr's details, suggest those are moved here. Or since only serialization uses this, move this file there... boost/detail/reference_content.hpp Used by optional and variant, move to "common". boost/detail/winapi/* Depends on boost/cstdint.hpp only, suggest that belongs to config. Everything else in detail has no dependencies other than config. Thoughts? John.

Show replies by date

Ahmed Charles

1 Nov 1 Nov

7:28 p.m.

...

From: john@johnmaddock.co.uk To: boost@lists.boost.org Date: Fri, 1 Nov 2013 19:15:17 +0000 Subject: [boost] [modularization] Improving/splitting up detail

Lets say we wanted boost/detail to depend on nothing but config, here's what it would take:

boost/detail/algorithm.hpp

Used by graph lib only, suggest it's moved there.

boost/detail/allocator_utilities.hpp

Used by multi index, statechart and flyweight, suggest move to new "common" module.

boost/detail/atomic_count.hpp

Depends on smart_ptr's details only, suggest those details are moved to "detail" module.

boost/detail/binary_search.hpp

Used by python and test, move to "common".

boost/detail/bitmask.hpp

Only external dependency is boost/cstdint.hpp, either move to "common", or else move cstdint.hpp to "config".

boost/detail/call_traits.hpp

Move to call_traits (should be there already).

boost/detail/catch_exceptions.hpp

Only external dependency is boost/cstdint.hpp, suggest latter is moved to config.

boost/detail/compressed_pair.hpp

Move to compressed pair (should be there already).

boost/detail/endian.hpp

Move to "predef"

boost/detail/has_default_constructor.hpp

File appears to be unused, and not very useful, suggest we remove it.

boost/detail/identifier.hpp

File appears to be unused, suggest we remove it.

boost/detail/indirect_traits.hpp

Used by iterator and python, move to "common"

boost/detail/is_incrementable.hpp

Move to common, or better modify the libraies using this to use the newer type_traits equivalents.

boost/detail/is_sorted.hpp

Used by graph and range, move to "common"

boost/detail/is_xxx.hpp

Used by parameter and python, move to "common"

boost/detail/iterator.hpp

I suspect this may be obsolete now compiler requirements have been bumped, but whatever it's still used by multiple libs: move to "common"

boost/detail/lcast_precision.hpp

Only used by lexical_cast, move it there.

boost/detail/lightweight_mutex.hpp

Only extern dependency is smart_ptr's details, suggest move those to detail.

boost/detail/named_template_params.hpp

Appears to be unused, suggest we remove.

boost/detail/numeric_traits.hpp

Used by graph and iterator, move to "common"

boost/detail/ob_compressed_pair.hpp

Move to compressed pair, might also be obsolete by now.

boost/detail/quick_allocator.hpp

Only external dependency is smart_ptr's details, suggest those are moved here. Or since only serialization uses this, move this file there...

boost/detail/reference_content.hpp

Used by optional and variant, move to "common".

boost/detail/winapi/*

Depends on boost/cstdint.hpp only, suggest that belongs to config.

Everything else in detail has no dependencies other than config.

Thoughts?

These look good. My only question is: what is the "common" that you refer to? An existing library or something that will be created?

John Maddock

7:45 p.m.

...

These look good. My only question is: what is the "common" that you refer to? An existing library or something that will be created?

Yes a new module, consisting of those parts of detail that are shared between multiple libraries, but have heavier dependencies. John.

Andrey Semashev

7:50 p.m.

On Fri, Nov 1, 2013 at 11:15 PM, John Maddock wrote:

...

Lets say we wanted boost/detail to depend on nothing but config, here's what it would take:

A general question: how do "common" and "detail" modules correlate? It seems they both contain things too little to form a full fledged library. Maybe there should be a single module, "common"? Or "utility"?

...

boost/detail/winapi/*

Depends on boost/cstdint.hpp only, suggest that belongs to config.

Not sure about that. On one hand these headers do relate to configuring on Windows. OTOH, that is more than defining a few macros.

Peter Dimov

8:14 p.m.

John Maddock wrote:

...

Lets say we wanted boost/detail to depend on nothing but config, ...

Perhaps I'm missing your point entirely, but... I don't think that we have a Detail module. All detail headers are actually parts of other modules.

Daniel James

9:09 p.m.

On 1 November 2013 20:14, Peter Dimov wrote:

...

Perhaps I'm missing your point entirely, but... I don't think that we have a Detail module.

We do now. https://github.com/boostorg/detail

Daniel James

8:19 p.m.

On 1 November 2013 19:15, John Maddock wrote:

...

Lets say we wanted boost/detail to depend on nothing but config, here's what it would take:

It would be good to establish a convention for modules which aren't "official" Boost libraries (i.e. haven't gone through a review, aren't considered public but are needed by other libraries).

...

boost/detail/allocator_utilities.hpp

Used by multi index, statechart and flyweight, suggest move to new "common" module.

We might be able to deprecate this header, it's used to support old allocators. I stopped using it in Boost.Unordered 2 years ago and no one noticed.

...

boost/detail/endian.hpp

Move to "predef"

The old version of endian.hpp (i.e. up to r85229) needs to stay in detail.

Rene Rivera

2 Nov 2 Nov

2:17 a.m.

On Fri, Nov 1, 2013 at 3:19 PM, Daniel James wrote:

...

On 1 November 2013 19:15, John Maddock wrote:

...
boost/detail/endian.hpp

Move to "predef"

The old version of endian.hpp (i.e. up to r85229) needs to stay in detail.

Eventually the libraries that use andian.hpp should be adjusted to use the predef equivalent directly instead of through the detail/endian.hpp compatibility header. And hence this header would disappear entirely. -- -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim - grafikrobot/yahoo

Jeremiah Willcock

1 Nov 1 Nov

9:15 p.m.

On Fri, 1 Nov 2013, John Maddock wrote:

...

Lets say we wanted boost/detail to depend on nothing but config, here's what it would take:

boost/detail/algorithm.hpp

Used by graph lib only, suggest it's moved there.

I'm fine with that, but all uses of them can probably be removed anyway (there are only two functions in that file with four overloads).

...

boost/detail/is_sorted.hpp

Used by graph and range, move to "common"

Doesn't this belong in Algorithm? It might be there already by some other name. -- Jeremiah Willcock

Marshall Clow

3 Nov 3 Nov

3:40 p.m.

On Nov 1, 2013, at 2:15 PM, Jeremiah Willcock wrote:

...

On Fri, 1 Nov 2013, John Maddock wrote:

...
Lets say we wanted boost/detail to depend on nothing but config, here's what it would take:

boost/detail/algorithm.hpp

Used by graph lib only, suggest it's moved there.

I'm fine with that, but all uses of them can probably be removed anyway (there are only two functions in that file with four overloads).

...
boost/detail/is_sorted.hpp

Used by graph and range, move to "common"

Doesn't this belong in Algorithm? It might be there already by some other name.

Yes. #include And has two advantages: 1) Range-based overloads. 2) Uses the standard library versions in c++11 (and beyond) -- Marshall Marshall Clow Idio Software mailto:mclow.lists@gmail.com A.D. 1517: Martin Luther nails his 95 Theses to the church door and is promptly moderated down to (-1, Flamebait). -- Yu Suzuki

Daniel Pfeifer

1 Nov 1 Nov

10:10 p.m.

Hi John, Thanks for the analysis! Could you adapt the mapping in repositories.txt accordingly and submit a pull request on Boost2Git (Preferable in several small commits)? Alternatively, I can make the changes in repositories.txt if you give me the *exact* information what to change (Maybe after the discussion).

...

boost/detail/xxx.hpp

File appears to be unused, suggest we remove it.

Since we modularize the complete history, we need to define where a file belongs. Even if we delete it.

...

boost/detail/xxx.hpp

Used by YYY and ZZZ, move to "common"

Isn't there already a dependency from YYY to ZZZ? Could we put xxx.hpp to ZZZ instead? Maybe a "common" is not even needed. -- Daniel

Daniel James

10:34 p.m.

On 1 November 2013 22:10, Daniel Pfeifer wrote:

...

...
boost/detail/xxx.hpp

Used by YYY and ZZZ, move to "common"

Isn't there already a dependency from YYY to ZZZ? Could we put xxx.hpp to ZZZ instead? Maybe a "common" is not even needed.

We should be careful about making changes just because they fit the current dependencies. Dependencies are not constant, we don't want to be in a situation where the dependency graph falls apart because of a future change to one of these headers. Also, if a header is in detail it should be potentially useful for future libraries which wouldn't want to depend on ZZZ (if it isn't, then it shouldn't have been put in detail in the first place). Lumping together unrelated code because it happens to fit the dependency graph could easily result in a mess which makes no logical sense. Reducing dependencies between header-only libraries is not that important a concern.

Ahmed Charles

2 Nov 2 Nov

6:18 a.m.

...

Date: Fri, 1 Nov 2013 22:34:07 +0000 From: daniel@calamity.org.uk To: boost@lists.boost.org Subject: Re: [boost] [modularization] Improving/splitting up detail

On 1 November 2013 22:10, Daniel Pfeifer wrote:

...
...
boost/detail/xxx.hpp

Used by YYY and ZZZ, move to "common"

Isn't there already a dependency from YYY to ZZZ? Could we put xxx.hpp to ZZZ instead? Maybe a "common" is not even needed.

We should be careful about making changes just because they fit the current dependencies. Dependencies are not constant, we don't want to be in a situation where the dependency graph falls apart because of a future change to one of these headers. Also, if a header is in detail it should be potentially useful for future libraries which wouldn't want to depend on ZZZ (if it isn't, then it shouldn't have been put in detail in the first place).

The goal is to create two 'modules' or libraries: 1. "detail" - which only depend on config 2. "common" which has minimal dependencies and contains code useful to multiple libraries (hopefully it doesn't depend on things that depend on it). So, by definition, there can't be a change to detail which intentionally adds dependencies, since that is forbidden by the goals of the library. Of course, people can make mistakes, but they would then be fixed and the desired invariant would hold again. Saying that future code may ruin the dependencies of the proposed library is like suggesting that future code may cause shared_ptr to leak memory. That's true, but it doesn't stop us from designing a class that manages memory without leaking.

...

Lumping together unrelated code because it happens to fit the dependency graph could easily result in a mess which makes no logical sense. Reducing dependencies between header-only libraries is not that important a concern.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Daniel Pfeifer

7:23 a.m.

2013/11/2 Ahmed Charles :

...

...
Date: Fri, 1 Nov 2013 22:34:07 +0000 From: daniel@calamity.org.uk To: boost@lists.boost.org Subject: Re: [boost] [modularization] Improving/splitting up detail

On 1 November 2013 22:10, Daniel Pfeifer wrote:

...
...
boost/detail/xxx.hpp

Used by YYY and ZZZ, move to "common"

Isn't there already a dependency from YYY to ZZZ? Could we put xxx.hpp to ZZZ instead? Maybe a "common" is not even needed.

We should be careful about making changes just because they fit the current dependencies. Dependencies are not constant, we don't want to be in a situation where the dependency graph falls apart because of a future change to one of these headers. Also, if a header is in detail it should be potentially useful for future libraries which wouldn't want to depend on ZZZ (if it isn't, then it shouldn't have been put in detail in the first place).

The goal is to create two 'modules' or libraries:

We had that originally. They were called "detail" and "core". How do we describe those libraries to users? Those might be the Frequently Questioned Answers: A: It is a Boost library that you should not use. Q: If it is not useful, why does it exist? A: It is used internally by several Boost libraries. Q: If it is so useful, why should I not use it too? I prefer to have no such "detail" libraries at all. Everything that is useful to a broader audience should be in utility. Everything else should be an implementation detail of one library. When one library depends on the implementation detail of another library (ie. includes a 'detail' header from it), that is a smell, OK. But at least it is hidden from users. -- Daniel

John Maddock

8:35 a.m.

...

...
The goal is to create two 'modules' or libraries:

We had that originally. They were called "detail" and "core".

How do we describe those libraries to users? Those might be the Frequently Questioned Answers:

A: It is a Boost library that you should not use. Q: If it is not useful, why does it exist?

A: It is used internally by several Boost libraries. Q: If it is so useful, why should I not use it too?

I prefer to have no such "detail" libraries at all. Everything that is useful to a broader audience should be in utility.

Funnily enough I was about to suggest just that: lets do away with detail and have: core: everything in: boost/detail/ boost/pending/ boost/utility/ That has no dependencies. Utility: As above, but with dependencies (mostly to mpl/pp/type_traits). throw_exception.hpp and dependencies should go in core too, as would a few headers under boost/ such as cstdint.hpp etc. John.

Rob Stewart

12:20 p.m.

On Nov 2, 2013, at 4:35 AM, "John Maddock" wrote:

...

...
...
The goal is to create two 'modules' or libraries:

We had that originally. They were called "detail" and "core". [snip]

I prefer to have no such "detail" libraries at all. Everything that is useful to a broader audience should be in utility.

Funnily enough I was about to suggest just that: lets do away with detail and have:

core: everything in: boost/detail/ boost/pending/ boost/utility/ That has no dependencies.

Utility: As above, but with dependencies (mostly to mpl/pp/type_traits).

throw_exception.hpp and dependencies should go in core too, as would a few headers under boost/ such as cstdint.hpp etc.

That seems like a good idea provided that everything now in namespace detail remains in that namespace. ___ Rob (Sent from my portable computation engine)

John Maddock

12:24 p.m.

...

...
Funnily enough I was about to suggest just that: lets do away with detail and have:

core: everything in: boost/detail/ boost/pending/ boost/utility/ That has no dependencies.

Utility: As above, but with dependencies (mostly to mpl/pp/type_traits).

throw_exception.hpp and dependencies should go in core too, as would a few headers under boost/ such as cstdint.hpp etc.

That seems like a good idea provided that everything now in namespace detail remains in that namespace.

Yes, no code changes, we're just talking about which module "owns" which header. John.

Daniel James

3 Nov 3 Nov

9:54 a.m.

On 2 November 2013 08:35, John Maddock wrote:

...

Funnily enough I was about to suggest just that: lets do away with detail and have:

For recreating old versions of boost, we're going to need to have something mapped to the location 'libs/detail' that contains the detail test files. It won't necessarily have to be called 'detail'.

Daniel James

2 Nov 2 Nov

9:27 a.m.

On 2 November 2013 07:23, Daniel Pfeifer wrote:

...

I prefer to have no such "detail" libraries at all. Everything that is useful to a broader audience should be in utility.

The problem there is that these things haven't been reviewed. The solution for such things was to put them in 'pending' (if they were going to be reviewed) or 'detail' otherwise. We need to establish a better convention.

Andrey Semashev

9:47 a.m.

On Saturday 02 November 2013 08:23:05 Daniel Pfeifer wrote:

...

How do we describe those libraries to users? Those might be the Frequently Questioned Answers:

A: It is a Boost library that you should not use. Q: If it is not useful, why does it exist?

A: It is used internally by several Boost libraries. Q: If it is so useful, why should I not use it too?

A: Because they are Boost implementation details. They are undocumented and can be changed or removed without notice and backward compatibility. That said I agree that the detail module looks awkward.

...

I prefer to have no such "detail" libraries at all. Everything that is useful to a broader audience should be in utility. Everything else should be an implementation detail of one library.

There are things not intended for public use and yet useful for multiple libraries. I don't think these components should be made a public part of utility, but making them a private part of it seems ok to me.

Gavin Lambert

5 Nov 5 Nov

6:17 a.m.

On 2/11/2013 22:47, Quoth Andrey Semashev:

...

There are things not intended for public use and yet useful for multiple libraries. I don't think these components should be made a public part of utility, but making them a private part of it seems ok to me.

If something has proven useful for multiple Boost libraries, why wouldn't it be useful for application code? The only reason I can think of is simply that whoever wrote it doesn't want to pay the costs of documenting, reviewing, and at least somewhat preserving compatibility between releases -- but without documentation, review, and some kind of cross-release compatibility, why would any Boost library authors want to depend on it either? Especially once Boost goes modular and a library author might make potentially breaking changes to some of the common code without even having downloaded some of the other libraries that depend on it (possibly thereby not realising the change was breaking, at least until the testers have cycled).

Andrey Semashev

6:36 a.m.

On Tue, Nov 5, 2013 at 10:17 AM, Gavin Lambert wrote:

...

On 2/11/2013 22:47, Quoth Andrey Semashev:

...
There are things not intended for public use and yet useful for multiple libraries. I don't think these components should be made a public part of utility, but making them a private part of it seems ok to me.

If something has proven useful for multiple Boost libraries, why wouldn't it be useful for application code?

Take a look at boost/detail contents. Except one or two headers, I don't see why would someone need those things outside Boost.

...

The only reason I can think of is simply that whoever wrote it doesn't want to pay the costs of documenting, reviewing, and at least somewhat preserving compatibility between releases -- but without documentation, review, and some kind of cross-release compatibility, why would any Boost library authors want to depend on it either?

All these are not big issues for internal components. The comments are enough documentation and in case of incompatible changes all Boost libraries can be updated relatively easily.

...

Especially once Boost goes modular and a library author might make potentially breaking changes to some of the common code without even having downloaded some of the other libraries that depend on it (possibly thereby not realising the change was breaking, at least until the testers have cycled).

Modularization makes changes to common components more difficult, I admit. But it doesn't mean these components should be made public.

Daniel James

2 Nov 2 Nov

9:16 a.m.

On 2 November 2013 06:18, Ahmed Charles wrote:

...

...
Date: Fri, 1 Nov 2013 22:34:07 +0000 From: daniel@calamity.org.uk To: boost@lists.boost.org Subject: Re: [boost] [modularization] Improving/splitting up detail

On 1 November 2013 22:10, Daniel Pfeifer wrote:

...
...
boost/detail/xxx.hpp

Used by YYY and ZZZ, move to "common"

Isn't there already a dependency from YYY to ZZZ? Could we put xxx.hpp to ZZZ instead? Maybe a "common" is not even needed.

We should be careful about making changes just because they fit the current dependencies. Dependencies are not constant, we don't want to be in a situation where the dependency graph falls apart because of a future change to one of these headers. Also, if a header is in detail it should be potentially useful for future libraries which wouldn't want to depend on ZZZ (if it isn't, then it shouldn't have been put in detail in the first place).

The goal is to create two 'modules' or libraries:

I understand that very well. I was responding to the idea that xxx.hpp should be put into ZZZ. Which is why I quoted that part. That's how quoting works.

4023

Age (days ago)

4027

Last active (days ago)

List overview

Download

22 comments

11 participants

participants (11)

Ahmed Charles
Andrey Semashev
Daniel James
Daniel Pfeifer
Gavin Lambert
Jeremiah Willcock
John Maddock
Marshall Clow
Peter Dimov
Rene Rivera
Rob Stewart