2013/10/19 Andrey Semashev :

On Saturday 19 October 2013 11:22:58 John Maddock wrote:

...

Trying to get this discussion somewhat back on track...

There are many libraries in Boost that have "glue" headers: by this I mean small headers that allow library A to interoperate with library B. Often these are simply specializations of traits classes or function overloads, are relatively trivial, and are not included unless you need them to be. By this I mean that: suppose library B wants to interoperate with lib A. It defines a special header which #includes what it needs from lib A and defines whatever specializations are required to get the two libraries working together.

Now here's the thing: there is no dependency from lib B to lib A *unless you are already using both libraries*.

A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.

So where does the glue header belong?

In most cases if you want B to interoperate with A then you define a header which:

* Includes headers from both libraries. * Uses the public interface of A, but may access the internals of B (think serialization).

Which would imply that the header belongs with B. Pure and simple. Except it's not, because these two libraries may be peers, and interoperability may be more complex than simply unidirectional.

I don't have an answer here, but it seems to me that "dependency" is a more complex thing than simply what-includes-what (and that's without getting into header file, vs source file, vs test file dependencies).

I think a notion of an optional dependency is needed. I don't know how such dependencies should be expresses in the technical sense though. Ideally, ryppl (or whatever other dependency tracking system) should offer a way to select the dependencies to install.

I agree.

2013/10/19 Andrey Semashev :

...

I think a notion of an optional dependency is needed. I don't know how such dependencies should be expresses in the technical sense though. Ideally, ryppl (or whatever other dependency tracking system) should offer a way to select the dependencies to install. I agree.

Lets use the example of Boost.UUID and Boost.Serialization.

My gut-feeling says that serialization support for UUID belongs to UUID. But lets consider the alternatives:

If serialization support for all libraries belongs to Serialization, the Serialization library is required to keep serialization support updated for all current and future Boost libraries. Agreed, this could not work.

If we say the glue code shall be a distinct component, we and up with an explosion of glue-components, one for each edge in the worst case. If you don't add the glue-component either you don't have the information of the possible added dependencies or you add them to the UUID component :( The glue files must belong to some component, isn't it?

So, having serialization support as an optional part of the supported components seems most viable.

Simply scanning all headers for #includes gives a list of all possible dependencies. Telling whether such a dependency is actually required or just optional cannot be easily decided.

A smarter dependency scanner might be able to distinguish optional dependencies for libraries that provide an "include all" header. I don't think the optional dependency is the way to go. We have an

Le 19/10/13 15:58, Daniel Pfeifer a écrit : optional component that the user can choose or not, but its dependencies are not optional.

Example: "boost/asio.hpp" pulls in most headers from Boost.ASIO, but not all. "boost/asio/ssl.hpp" is an example of such an exception. Hence, SSL is an optional dependency of Boost.ASIO.

I suspect that then we need a new component asio/ssl that can have other dependencies than asio. Best, Vicente

On 19 October 2013 14:58, Daniel Pfeifer wrote:

Lets use the example of Boost.UUID and Boost.Serialization.

My gut-feeling says that serialization support for UUID belongs to UUID.

FWIW that's how hash support is added. Support is usually in the same header as the object being hashed. Since it just requires writing a 'hash_value' function, a library might not even need to include any of hash's headers. To support writing generic hash functions, there is a lightweight forward declaration header[1] which only depends on config and workaround. If a library uses that header, bcp will understand this, but a tool which works on the module level won't. [1] http://www.boost.org/boost/functional/hash/hash_fwd.hpp It isn't a big problem for hash, as it doesn't have many dependencies (by boost's standards at least). But might be elsewhere.

A smarter dependency scanner might be able to distinguish optional dependencies for libraries that provide an "include all" header.

Example: "boost/asio.hpp" pulls in most headers from Boost.ASIO, but not all. "boost/asio/ssl.hpp" is an example of such an exception. Hence, SSL is an optional dependency of Boost.ASIO.

"include all" headers aren't always appropriate. There's no established naming convention, so there will need to be some explicit specification (e.g. I don't think boost/exception/exception.hpp or boost/bind/bind.hpp are "include all" headers). I think I'd prefer to explicitly specify optional dependencies, since they are the special case. Counting everything in a module is the safe option, so it should be the default. It also isn't clear how to handle the case when the glue for modules A and B depends C, but nothing else in A or B do. Would have to explicitly specify that the glue was being used, or have the dependency tool somehow understand this? Or maybe just require one of A or B to depend on C.

On 19 October 2013 17:40, Andrey Semashev wrote:

On Saturday 19 October 2013 18:23:54 Mathias Gaunard wrote:

...

On 19/10/13 12:22, John Maddock wrote:

...

A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.

So where does the glue header belong?

In a submodule of the library that depends on both the rest of the library and serialization.

I don't see any need for optional dependencies here.

IMHO, having glue headers in separate repositories is very inconvenient.

We could set it up so that a single repo could contain multiple submodules (i.e. a module for dependency checking is not the same thing as a git submodule).

On Sat, Oct 19, 2013 at 1:19 PM, Bjørn Roald wrote:

On 10/19/2013 06:59 PM, Daniel James wrote:

...

On 19 October 2013 17:40, Andrey Semashev wrote:

...

On Saturday 19 October 2013 18:23:54 Mathias Gaunard wrote:

...

On 19/10/13 12:22, John Maddock wrote:

...

A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.

So where does the glue header belong?

In a submodule of the library that depends on both the rest of the library and serialization.

I don't see any need for optional dependencies here.

IMHO, having glue headers in separate repositories is very inconvenient.

We could set it up so that a single repo could contain multiple submodules (i.e. a module for dependency checking is not the same thing as a git submodule).

agree,

I am not sure I like using the term submodule though, it sort of imply it is part of something, and thus that something have the properties of the part, hence we do not get rid of the dependency.

Why not just call them separate components, or modules or something else. They just happen to live in the repository of the library they can add features to if they are used.

-- Bjørn

I've been dealing with this issue very frequently in the last couple years and I've tried a number of the layouts people suggested. Sadly nothing leaves a complete feeling of satisfaction but some are definitely better than others. The one that seems to work best is creating an optional component in a separate header of the same repository. The optional component, rather than the full module or repository comes with a required dependency. For instance, I used an extensions folder in a library with a subfolder for each of the components that add functionality but require additional dependencies. When I tried separate repositories for the core of a single library and every component it quickly became unwieldy. One reason is that it isn't as simple to move files between repositories and reviewing the history of changes in files and structure becomes significantly less convenient. Most of my experience in this sort of structure is with mercurial w/ subrepositories + CMake, so Git may be better at some things. Another problem I'm still hoping to find a good solution for is what to do with headers with no dependencies that either define a constant or provide some other basic definition that makes sense to use in many libraries. I know there are a couple of headers that are exactly like this in Boost that are widely included. Does anyone have insight into good ways to handle these issues as well? There are other cases where some useful components could be split off from within libraries. For instance, many libraries could make use of boost::math::constants in cases where it doesn't make as much sense to have all of the statistical functions come along as another dependency. There are also a number of headers in other libraries that set up a design of some sort that could be useful outside the scope of the individual libraries as well, such as the tag dispatching utilities in boost::geometry. Obviously there are limits to what makes sense to split up, but perhaps there are improved ways we can think about and make these divisions. For example, could there be a smart way to split off some of the general low level concepts and utility objects/functions so they can be documented and used more widely themselves? Cheers! Andrew Hundt