[modularization] What do we do with "glue" headers?
Trying to get this discussion somewhat back on track... There are many libraries in Boost that have "glue" headers: by this I mean small headers that allow library A to interoperate with library B. Often these are simply specializations of traits classes or function overloads, are relatively trivial, and are not included unless you need them to be. By this I mean that: suppose library B wants to interoperate with lib A. It defines a special header which #includes what it needs from lib A and defines whatever specializations are required to get the two libraries working together. Now here's the thing: there is no dependency from lib B to lib A *unless you are already using both libraries*. A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header. So where does the glue header belong? In most cases if you want B to interoperate with A then you define a header which: * Includes headers from both libraries. * Uses the public interface of A, but may access the internals of B (think serialization). Which would imply that the header belongs with B. Pure and simple. Except it's not, because these two libraries may be peers, and interoperability may be more complex than simply unidirectional. I don't have an answer here, but it seems to me that "dependency" is a more complex thing than simply what-includes-what (and that's without getting into header file, vs source file, vs test file dependencies). Cheers, John.
On Saturday 19 October 2013 11:22:58 John Maddock wrote:
Trying to get this discussion somewhat back on track...
There are many libraries in Boost that have "glue" headers: by this I mean small headers that allow library A to interoperate with library B. Often these are simply specializations of traits classes or function overloads, are relatively trivial, and are not included unless you need them to be. By this I mean that: suppose library B wants to interoperate with lib A. It defines a special header which #includes what it needs from lib A and defines whatever specializations are required to get the two libraries working together.
Now here's the thing: there is no dependency from lib B to lib A *unless you are already using both libraries*.
A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.
So where does the glue header belong?
In most cases if you want B to interoperate with A then you define a header which:
* Includes headers from both libraries. * Uses the public interface of A, but may access the internals of B (think serialization).
Which would imply that the header belongs with B. Pure and simple. Except it's not, because these two libraries may be peers, and interoperability may be more complex than simply unidirectional.
I don't have an answer here, but it seems to me that "dependency" is a more complex thing than simply what-includes-what (and that's without getting into header file, vs source file, vs test file dependencies).
I think a notion of an optional dependency is needed. I don't know how such dependencies should be expresses in the technical sense though. Ideally, ryppl (or whatever other dependency tracking system) should offer a way to select the dependencies to install.
2013/10/19 Andrey Semashev <andrey.semashev@gmail.com>:
On Saturday 19 October 2013 11:22:58 John Maddock wrote:
Trying to get this discussion somewhat back on track...
There are many libraries in Boost that have "glue" headers: by this I mean small headers that allow library A to interoperate with library B. Often these are simply specializations of traits classes or function overloads, are relatively trivial, and are not included unless you need them to be. By this I mean that: suppose library B wants to interoperate with lib A. It defines a special header which #includes what it needs from lib A and defines whatever specializations are required to get the two libraries working together.
Now here's the thing: there is no dependency from lib B to lib A *unless you are already using both libraries*.
A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.
So where does the glue header belong?
In most cases if you want B to interoperate with A then you define a header which:
* Includes headers from both libraries. * Uses the public interface of A, but may access the internals of B (think serialization).
Which would imply that the header belongs with B. Pure and simple. Except it's not, because these two libraries may be peers, and interoperability may be more complex than simply unidirectional.
I don't have an answer here, but it seems to me that "dependency" is a more complex thing than simply what-includes-what (and that's without getting into header file, vs source file, vs test file dependencies).
I think a notion of an optional dependency is needed. I don't know how such dependencies should be expresses in the technical sense though. Ideally, ryppl (or whatever other dependency tracking system) should offer a way to select the dependencies to install.
I agree.
2013/10/19 Andrey Semashev <andrey.semashev@gmail.com>:
On Saturday 19 October 2013 11:22:58 John Maddock wrote:
Trying to get this discussion somewhat back on track...
There are many libraries in Boost that have "glue" headers: by this I mean small headers that allow library A to interoperate with library B. Often these are simply specializations of traits classes or function overloads, are relatively trivial, and are not included unless you need them to be. By this I mean that: suppose library B wants to interoperate with lib A. It defines a special header which #includes what it needs from lib A and defines whatever specializations are required to get the two libraries working together.
Now here's the thing: there is no dependency from lib B to lib A *unless you are already using both libraries*.
A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.
So where does the glue header belong?
In most cases if you want B to interoperate with A then you define a header which:
* Includes headers from both libraries. * Uses the public interface of A, but may access the internals of B (think serialization).
Which would imply that the header belongs with B. Pure and simple. Except it's not, because these two libraries may be peers, and interoperability may be more complex than simply unidirectional.
I don't have an answer here, but it seems to me that "dependency" is a more complex thing than simply what-includes-what (and that's without getting into header file, vs source file, vs test file dependencies).
I think a notion of an optional dependency is needed. I don't know how such dependencies should be expresses in the technical sense though. Ideally, ryppl (or whatever other dependency tracking system) should offer a way to select the dependencies to install.
I agree. Lets use the example of Boost.UUID and Boost.Serialization. My gut-feeling says that serialization support for UUID belongs to UUID. But lets consider the alternatives: If serialization support for all libraries belongs to Serialization, the Serialization library is required to keep serialization support updated for all current and future Boost libraries. If we say the glue code shall be a distinct component, we and up with an explosion of glue-components, one for each edge in the worst case. So, having serialization support as an optional part of the supported components seems most viable. Simply scanning all headers for #includes gives a list of all possible dependencies. Telling whether such a dependency is actually required or just optional cannot be easily decided. A smarter dependency scanner might be able to distinguish optional dependencies for libraries that provide an "include all" header. Example: "boost/asio.hpp" pulls in most headers from Boost.ASIO, but not all. "boost/asio/ssl.hpp" is an example of such an exception. Hence, SSL is an optional dependency of Boost.ASIO. cheers, Daniel
2013/10/19 Andrey Semashev <andrey.semashev@gmail.com>:
I think a notion of an optional dependency is needed. I don't know how such dependencies should be expresses in the technical sense though. Ideally, ryppl (or whatever other dependency tracking system) should offer a way to select the dependencies to install. I agree.
Lets use the example of Boost.UUID and Boost.Serialization.
My gut-feeling says that serialization support for UUID belongs to UUID. But lets consider the alternatives:
If serialization support for all libraries belongs to Serialization, the Serialization library is required to keep serialization support updated for all current and future Boost libraries. Agreed, this could not work.
If we say the glue code shall be a distinct component, we and up with an explosion of glue-components, one for each edge in the worst case. If you don't add the glue-component either you don't have the information of the possible added dependencies or you add them to the UUID component :( The glue files must belong to some component, isn't it?
So, having serialization support as an optional part of the supported components seems most viable.
Simply scanning all headers for #includes gives a list of all possible dependencies. Telling whether such a dependency is actually required or just optional cannot be easily decided.
A smarter dependency scanner might be able to distinguish optional dependencies for libraries that provide an "include all" header. I don't think the optional dependency is the way to go. We have an
Le 19/10/13 15:58, Daniel Pfeifer a écrit : optional component that the user can choose or not, but its dependencies are not optional.
Example: "boost/asio.hpp" pulls in most headers from Boost.ASIO, but not all. "boost/asio/ssl.hpp" is an example of such an exception. Hence, SSL is an optional dependency of Boost.ASIO.
I suspect that then we need a new component asio/ssl that can have other dependencies than asio. Best, Vicente
2013/10/19 Vicente J. Botet Escriba <vicente.botet@wanadoo.fr>:
Le 19/10/13 15:58, Daniel Pfeifer a écrit :
2013/10/19 Andrey Semashev <andrey.semashev@gmail.com>:
I think a notion of an optional dependency is needed. I don't know how such dependencies should be expresses in the technical sense though. Ideally, ryppl (or whatever other dependency tracking system) should offer a way to select the dependencies to install.
I agree.
Lets use the example of Boost.UUID and Boost.Serialization.
My gut-feeling says that serialization support for UUID belongs to UUID. But lets consider the alternatives:
If serialization support for all libraries belongs to Serialization, the Serialization library is required to keep serialization support updated for all current and future Boost libraries.
Agreed, this could not work.
If we say the glue code shall be a distinct component, we and up with an explosion of glue-components, one for each edge in the worst case.
If you don't add the glue-component either you don't have the information of the possible added dependencies or you add them to the UUID component :( The glue files must belong to some component, isn't it?
This is exactly my point.
So, having serialization support as an optional part of the supported components seems most viable.
Simply scanning all headers for #includes gives a list of all possible dependencies. Telling whether such a dependency is actually required or just optional cannot be easily decided.
A smarter dependency scanner might be able to distinguish optional dependencies for libraries that provide an "include all" header.
I don't think the optional dependency is the way to go. We have an optional component that the user can choose or not, but its dependencies are not optional.
The (required) dependencies of the optional component of the library introduce an optional dependency of the library. :-)
Example: "boost/asio.hpp" pulls in most headers from Boost.ASIO, but not all. "boost/asio/ssl.hpp" is an example of such an exception. Hence, SSL is an optional dependency of Boost.ASIO.
I suspect that then we need a new component asio/ssl that can have other dependencies than asio.
Asio already provides optional SSL support. As long as you don't include "boost/asio/ssl.hpp", SSL is not required. Are you suggesting that Asio should provide SSL support in a separate Git repository? I hope not! Cheers, Daniel
On 19 October 2013 14:58, Daniel Pfeifer <daniel@pfeifer-mail.de> wrote:
Lets use the example of Boost.UUID and Boost.Serialization.
My gut-feeling says that serialization support for UUID belongs to UUID.
FWIW that's how hash support is added. Support is usually in the same header as the object being hashed. Since it just requires writing a 'hash_value' function, a library might not even need to include any of hash's headers. To support writing generic hash functions, there is a lightweight forward declaration header[1] which only depends on config and workaround. If a library uses that header, bcp will understand this, but a tool which works on the module level won't. [1] http://www.boost.org/boost/functional/hash/hash_fwd.hpp It isn't a big problem for hash, as it doesn't have many dependencies (by boost's standards at least). But might be elsewhere.
A smarter dependency scanner might be able to distinguish optional dependencies for libraries that provide an "include all" header.
Example: "boost/asio.hpp" pulls in most headers from Boost.ASIO, but not all. "boost/asio/ssl.hpp" is an example of such an exception. Hence, SSL is an optional dependency of Boost.ASIO.
"include all" headers aren't always appropriate. There's no established naming convention, so there will need to be some explicit specification (e.g. I don't think boost/exception/exception.hpp or boost/bind/bind.hpp are "include all" headers). I think I'd prefer to explicitly specify optional dependencies, since they are the special case. Counting everything in a module is the safe option, so it should be the default. It also isn't clear how to handle the case when the glue for modules A and B depends C, but nothing else in A or B do. Would have to explicitly specify that the glue was being used, or have the dependency tool somehow understand this? Or maybe just require one of A or B to depend on C.
On 10/19/2013 12:22 PM, John Maddock wrote:
Trying to get this discussion somewhat back on track...
There are many libraries in Boost that have "glue" headers: by this I mean small headers that allow library A to interoperate with library B. Often these are simply specializations of traits classes or function overloads, are relatively trivial, and are not included unless you need them to be. By this I mean that: suppose library B wants to interoperate with lib A. It defines a special header which #includes what it needs from lib A and defines whatever specializations are required to get the two libraries working together.
Now here's the thing: there is no dependency from lib B to lib A *unless you are already using both libraries*.
A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.
So where does the glue header belong?
In most cases if you want B to interoperate with A then you define a header which:
* Includes headers from both libraries. * Uses the public interface of A, but may access the internals of B (think serialization).
Which would imply that the header belongs with B. Pure and simple. Except it's not, because these two libraries may be peers, and interoperability may be more complex than simply unidirectional.
I don't have an answer here, but it seems to me that "dependency" is a more complex thing than simply what-includes-what (and that's without getting into header file, vs source file, vs test file dependencies).
Could such glue headers simply be considered separate components by the dependency scanning as their dependencies do not incur as dependency before they are actually included in application source. B --> core A_with_B_glue --> core A_with_B_glue --> B becomes: A --> core B --> core AB_glue --> A AB_glue --> B Whether these should belong in one of the git repositories of which main libraries they depend on or somewhere else is a different question that has more to do more software configuration management than component dependencies. I am trying to say it may not be important to move those headers. -- Bjørn
On 10/19/2013 6:22 AM, John Maddock wrote:
Trying to get this discussion somewhat back on track...
There are many libraries in Boost that have "glue" headers: by this I mean small headers that allow library A to interoperate with library B. Often these are simply specializations of traits classes or function overloads, are relatively trivial, and are not included unless you need them to be. By this I mean that: suppose library B wants to interoperate with lib A. It defines a special header which #includes what it needs from lib A and defines whatever specializations are required to get the two libraries working together.
Now here's the thing: there is no dependency from lib B to lib A *unless you are already using both libraries*.
A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.
So where does the glue header belong?
In most cases if you want B to interoperate with A then you define a header which:
* Includes headers from both libraries. * Uses the public interface of A, but may access the internals of B (think serialization).
Which would imply that the header belongs with B. Pure and simple. Except it's not, because these two libraries may be peers, and interoperability may be more complex than simply unidirectional.
I don't have an answer here, but it seems to me that "dependency" is a more complex thing than simply what-includes-what (and that's without getting into header file, vs source file, vs test file dependencies).
I agree with what you have said and therefore in some cases it becomes wrong to simply move files from one place to another based on a graph of header file dependencies. Individual libraries need to be considered. Regarding the property_map library I was wrong in not noticing that it does indeed including graph library headers but I believe I am correct in that its files, in a proposed modular Boost, should not be moved with those of the graph library. Property maps are an independent concept from graphs, even if used by the graph library. I am actually doing work locally with the property_map library in order to move the graph-related distributed headers and specializations to the graph library and will report on my results when I am finished. These may indeed be "glue" headers but I will argue when I am done, if I am successful, that they belong with the graph library.
2013/10/19 John Maddock <john@johnmaddock.co.uk>:
Trying to get this discussion somewhat back on track...
There are many libraries in Boost that have "glue" headers: by this I mean small headers that allow library A to interoperate with library B. Often these are simply specializations of traits classes or function overloads, are relatively trivial, and are not included unless you need them to be. By this I mean that: suppose library B wants to interoperate with lib A. It defines a special header which #includes what it needs from lib A and defines whatever specializations are required to get the two libraries working together.
Now here's the thing: there is no dependency from lib B to lib A *unless you are already using both libraries*.
A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.
So where does the glue header belong?
In most cases if you want B to interoperate with A then you define a header which:
* Includes headers from both libraries. * Uses the public interface of A, but may access the internals of B (think serialization).
Which would imply that the header belongs with B. Pure and simple. Except it's not, because these two libraries may be peers, and interoperability may be more complex than simply unidirectional.
I don't have an answer here, but it seems to me that "dependency" is a more complex thing than simply what-includes-what (and that's without getting into header file, vs source file, vs test file dependencies).
Cheers, John.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 19/10/13 12:22, John Maddock wrote:
A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.
So where does the glue header belong?
In a submodule of the library that depends on both the rest of the library and serialization. I don't see any need for optional dependencies here.
On Saturday 19 October 2013 18:23:54 Mathias Gaunard wrote:
On 19/10/13 12:22, John Maddock wrote:
A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.
So where does the glue header belong?
In a submodule of the library that depends on both the rest of the library and serialization.
I don't see any need for optional dependencies here.
IMHO, having glue headers in separate repositories is very inconvenient.
On 19 October 2013 17:40, Andrey Semashev <andrey.semashev@gmail.com> wrote:
On Saturday 19 October 2013 18:23:54 Mathias Gaunard wrote:
On 19/10/13 12:22, John Maddock wrote:
A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.
So where does the glue header belong?
In a submodule of the library that depends on both the rest of the library and serialization.
I don't see any need for optional dependencies here.
IMHO, having glue headers in separate repositories is very inconvenient.
We could set it up so that a single repo could contain multiple submodules (i.e. a module for dependency checking is not the same thing as a git submodule).
On 10/19/2013 06:59 PM, Daniel James wrote:
On 19 October 2013 17:40, Andrey Semashev <andrey.semashev@gmail.com> wrote:
On Saturday 19 October 2013 18:23:54 Mathias Gaunard wrote:
On 19/10/13 12:22, John Maddock wrote:
A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.
So where does the glue header belong?
In a submodule of the library that depends on both the rest of the library and serialization.
I don't see any need for optional dependencies here.
IMHO, having glue headers in separate repositories is very inconvenient.
We could set it up so that a single repo could contain multiple submodules (i.e. a module for dependency checking is not the same thing as a git submodule).
agree, I am not sure I like using the term submodule though, it sort of imply it is part of something, and thus that something have the properties of the part, hence we do not get rid of the dependency. Why not just call them separate components, or modules or something else. They just happen to live in the repository of the library they can add features to if they are used. -- Bjørn
On Sat, Oct 19, 2013 at 1:19 PM, Bjørn Roald <bjorn@4roald.org> wrote:
On 10/19/2013 06:59 PM, Daniel James wrote:
On 19 October 2013 17:40, Andrey Semashev <andrey.semashev@gmail.com> wrote:
On Saturday 19 October 2013 18:23:54 Mathias Gaunard wrote:
On 19/10/13 12:22, John Maddock wrote:
A good example would be the serialization lib: a simple dependency tracker would show this up as a dependency to a large part of Boost, but in fact that's not true: in the vast majority of those cases there is no such dependency unless you actually want to use both libraries together and do so by including the "glue" header.
So where does the glue header belong?
In a submodule of the library that depends on both the rest of the library and serialization.
I don't see any need for optional dependencies here.
IMHO, having glue headers in separate repositories is very inconvenient.
We could set it up so that a single repo could contain multiple submodules (i.e. a module for dependency checking is not the same thing as a git submodule).
agree,
I am not sure I like using the term submodule though, it sort of imply it is part of something, and thus that something have the properties of the part, hence we do not get rid of the dependency.
Why not just call them separate components, or modules or something else. They just happen to live in the repository of the library they can add features to if they are used.
-- Bjørn
I've been dealing with this issue very frequently in the last couple years and I've tried a number of the layouts people suggested. Sadly nothing leaves a complete feeling of satisfaction but some are definitely better than others. The one that seems to work best is creating an optional component in a separate header of the same repository. The optional component, rather than the full module or repository comes with a required dependency. For instance, I used an extensions folder in a library with a subfolder for each of the components that add functionality but require additional dependencies. When I tried separate repositories for the core of a single library and every component it quickly became unwieldy. One reason is that it isn't as simple to move files between repositories and reviewing the history of changes in files and structure becomes significantly less convenient. Most of my experience in this sort of structure is with mercurial w/ subrepositories + CMake, so Git may be better at some things. Another problem I'm still hoping to find a good solution for is what to do with headers with no dependencies that either define a constant or provide some other basic definition that makes sense to use in many libraries. I know there are a couple of headers that are exactly like this in Boost that are widely included. Does anyone have insight into good ways to handle these issues as well? There are other cases where some useful components could be split off from within libraries. For instance, many libraries could make use of boost::math::constants in cases where it doesn't make as much sense to have all of the statistical functions come along as another dependency. There are also a number of headers in other libraries that set up a design of some sort that could be useful outside the scope of the individual libraries as well, such as the tag dispatching utilities in boost::geometry. Obviously there are limits to what makes sense to split up, but perhaps there are improved ways we can think about and make these divisions. For example, could there be a smart way to split off some of the general low level concepts and utility objects/functions so they can be documented and used more widely themselves? Cheers! Andrew Hundt
There are other cases where some useful components could be split off from within libraries. For instance, many libraries could make use of boost::math::constants in cases where it doesn't make as much sense to have all of the statistical functions come along as another dependency.
There are a number of libraries which use the "core" of boost.math but don't need the whole thing - one that causes cyclic dependencies because of this is lexical_cast which is both used by Boost.Math (including the constants code unfortunately), and also uses a few core Math lib functions. One way to avoid that would be to split off a "Math.Core" lib which contained only a few common headers (no tests or docs etc). Trouble is constants can't be part of the core as it would bring back those cyclic dependencies to lexical_cast. They could be their own module I guess, but I'd really like to avoid too many splits. With regard to glue headers - the conclusion I've come to is that we need to manually break (by which I mean ignore) some dependencies. Consider, if lib B has a glue header for interoperability with lib A, then there's not any need to automatically track dependencies to A. After all the whole point of the glue header is to use the two libraries together, so it's not like the user doesn't know that they need both modules. To consider another cyclic dependency: the Math and Multiprecision libs. In fact a human observer looking at the code would conclude the dependency graph goes in one direction - from Multiprecision to Math (again a small part of it). But there is one Math lib header which *may* under some circumstances (user sets a particular macro) include some multiprecision headers. Further that header only exists to assist in the generation of new numeric constants - whether by the user or by the Math lib's maintainers. So it's usage will be very infrequent, and since the user has to explicitly tell it to use Boost.Multiprecision it's not like they don't know that it's needed. I guess we could use macro's to obfuscate the #includes so there's no dependency tracking in these cases, but it seems a stupid thing to do (obfuscating the code, and possibly breaking build tool support) just to keep the modularization folks happy. So I think I'd like the folks pushing for modularization to provide a way for us to annotate headers with a "don't track here" direction. Thoughts? John.
participants (9)
-
Andrew Hundt -
Andrey Semashev -
Bjørn Roald -
Daniel James -
Daniel Pfeifer -
Edward Diener -
John Maddock -
Mathias Gaunard -
Vicente J. Botet Escriba