[bcp] [shared_ptr] Extracting subset of boost for shared_ptr - 3.2MB? - Boost-users

[bcp] [shared_ptr] Extracting subset of boost for shared_ptr - 3.2MB?

Tyler Mandry

18 Feb 2006 18 Feb '06

5:19 a.m.

I extracted a subset of boost using bcp so that my project could use shared_ptr. The exact line I used is this: $ bcp --boost=/usr/include boost/shared_ptr.hpp boost/ This gave me more than just the boost headers, but any other way and it wouldn't work. It wasn't a problem though; I could just use the files in the newly-created boost/boost directory and wipe out anything else. So I was left with, presumably, all the boost headers and files that depended on shared_ptr.hpp. But, after running du, I found that the extracted directory was a whopping 3.2MB - bigger than the existing code of my project altogether. bcp had given me 274 different header files. All for just one smart pointer class? Is this correct? Am I using bcp correctly? Are there some paths that bcp is taking that aren't really neccessary? Can I narrow this huge vat of files down to the ones that I *really* need, or is this it? How? Regards and thanks, -- Tyler Mandry

Show replies by date

Peter Dimov

18 Feb 18 Feb

4:20 p.m.

New subject: [bcp] [shared_ptr] Extracting subset of boost forshared_ptr - 3.2MB?

Tyler Mandry wrote:

...

Are there some paths that bcp is taking that aren't really neccessary?

Unfortunately bcp isn't smart enough to not look into optional dependencies that are guarded by #ifdef. In this particular case it follows the dependency to detail/quick_allocator.hpp, which in turn brings in the type_traits and mpl libraries. But quick_allocator.hpp is only used if you #define BOOST_SP_USE_QUICK_ALLOCATOR.

...

Can I narrow this huge vat of files down to the ones that I *really* need, or is this it? How?

The only reliable way that I know of is to copy the required headers into a pristine include directory one by one until your program compiles. Not particularly elegant, but it works. In shared_ptr's case you need boost/assert.hpp boost/current_function.hpp boost/checked_delete.hpp boost/throw_exception.hpp Most of boost/detail/, so I just copy it wholesale boost/config/*

John Maddock

4:32 p.m.

New subject: [bcp] [shared_ptr] Extracting subset of boostforshared_ptr - 3.2MB?

...

Unfortunately bcp isn't smart enough to not look into optional dependencies that are guarded by #ifdef. In this particular case it follows the dependency to detail/quick_allocator.hpp, which in turn brings in the type_traits and mpl libraries. But quick_allocator.hpp is only used if you #define BOOST_SP_USE_QUICK_ALLOCATOR.

Peter, this comes up so often, I wonder is it worth while "hiding" that optional dependency behind a #define, as in: #define BOOST_SP_QA #ifdef BOOST_SP_USE_QUICK_ALLOCATOR #include BOOST_SP_QA #endif Of course I have no data on how often that dependency is actually needed (or not). John.

Peter Dimov

5:08 p.m.

New subject: [bcp] [shared_ptr] Extracting subset ofboostforshared_ptr - 3.2MB?

John Maddock wrote:

...

...
Unfortunately bcp isn't smart enough to not look into optional dependencies that are guarded by #ifdef. In this particular case it follows the dependency to detail/quick_allocator.hpp, which in turn brings in the type_traits and mpl libraries. But quick_allocator.hpp is only used if you #define BOOST_SP_USE_QUICK_ALLOCATOR.

Peter, this comes up so often, I wonder is it worth while "hiding" that optional dependency behind a #define, as in:

#define BOOST_SP_QA

#ifdef BOOST_SP_USE_QUICK_ALLOCATOR #include BOOST_SP_QA #endif

Of course I have no data on how often that dependency is actually needed (or not).

How about something along the lines of: bcp boost/shared_ptr.hpp --exclude=quick_allocator.hpp Or: #ifdef BOOST_SP_USE_QUICK_ALLOCATOR //@bcp optional "quick_allocator is only used if explicitly requested" #include #endif which would then inform the user that an optional dependency wasn't being followed (if this is the default): bcp: an optional dependency isn't included in the subset because `quick_allocator is only used if explicitly requested' bcp: use `bcp boost/shared_ptr.hpp boost/detail/quick_allocator.hpp' to include it The advantage is that we don't have to decide whether to follow a particular dependency, the choice can be left to the person running bcp.

John Maddock

19 Feb 19 Feb

10:17 a.m.

New subject: [bcp] [shared_ptr] Extracting subsetofboostforshared_ptr - 3.2MB?

...

How about something along the lines of:

bcp boost/shared_ptr.hpp --exclude=quick_allocator.hpp

Or:

#ifdef BOOST_SP_USE_QUICK_ALLOCATOR

//@bcp optional "quick_allocator is only used if explicitly requested" #include

#endif

which would then inform the user that an optional dependency wasn't being followed (if this is the default):

bcp: an optional dependency isn't included in the subset because

`quick_allocator is only used if explicitly requested'

bcp: use `bcp boost/shared_ptr.hpp boost/detail/quick_allocator.hpp' to include it

The advantage is that we don't have to decide whether to follow a particular dependency, the choice can be left to the person running bcp.

I like it, shouldn't be too hard to add either..... John.

Jeff Garland

5:13 p.m.

New subject: [bcp] [shared_ptr] Extracting subsetofboostforshared_ptr - 3.2MB?

On Sun, 19 Feb 2006 10:17:12 -0000, John Maddock wrote

...

...
How about something along the lines of:

bcp boost/shared_ptr.hpp --exclude=quick_allocator.hpp

Or:

#ifdef BOOST_SP_USE_QUICK_ALLOCATOR

//@bcp optional "quick_allocator is only used if explicitly requested" #include

#endif

which would then inform the user that an optional dependency wasn't being followed (if this is the default):

bcp: an optional dependency isn't included in the subset because

`quick_allocator is only used if explicitly requested'

bcp: use `bcp boost/shared_ptr.hpp boost/detail/quick_allocator.hpp' to include it

The advantage is that we don't have to decide whether to follow a particular dependency, the choice can be left to the person running bcp.

I like it, shouldn't be too hard to add either.....

As you've already shown... Is 'optional' a 'fixed tag' or can we use it to slice and dice things? For example, suppose we wanted to create a boost-wide 'with[out]_serialization' capability. Could we do this? //@bcp with_serialization "serialization support for date-time is optional" #include #include "boost/serialization/split_free.hpp" There a bunch of boost libraries that would benefit from this type of optional flagging. For date-time it turns out that the --exclude=xyz.hpp capability could also serve the same purpose. Another possible idea. What if we created a new convention: <libname>_bcp.cfg that could be put in the library header directory. Then the library author could directly express the options using some trivial syntax of our own invention that would feed bcp and thus trivialize the command line for users. This syntax would be something like: option option_name option_description bcp_command_parameters option serialization "Provides support for serialization" --exclude date_time/gregorian/greg_serialize.hpp --exclude date_time/posix_time/time_serialize.hpp or even better: option serialization "Provides support for serialization" --exclude_library_depend serialization --exclude_library_depend archive Anyway, just some wild ideas ;-) Jeff

David Abrahams

6:28 p.m.

New subject: [bcp] [shared_ptr] Extracting subsetofboostforshared_ptr - 3.2MB?

"Jeff Garland" writes:

...

Another possible idea. What if we created a new convention: <libname>_bcp.cfg that could be put in the library header directory. Then the library author could directly express the options using some trivial syntax of our own invention that would feed bcp and thus trivialize the command line for users. This syntax would be something like:

option option_name option_description bcp_command_parameters

We probably ought to be thinking about how this could work with Boost.Build. I don't see it clearly yet, but I bet there's some kind of integration opportunity. -- Dave Abrahams Boost Consulting www.boost-consulting.com

John Maddock

4:24 p.m.

New subject: [bcp] [shared_ptr] Extracting subsetofboostforshared_ptr - 3.2MB?

...

Or:

#ifdef BOOST_SP_USE_QUICK_ALLOCATOR

//@bcp optional "quick_allocator is only used if explicitly requested" #include

#endif

I've just added this to bcp as an experiment (it's in mainline cvs), with the extra line above added to shared_ptr as well, the full dependency list is down to: $bcp --cvs --list shared_ptr.hpp Optional functionality won't be copied: quick_allocator is only used if explicitly requested Add the file boost/detail/quick_allocator.hpp to the list of dependencies to extract to copy this functionality. boost/assert.hpp boost/checked_delete.hpp boost/config.hpp boost/config/abi/borland_prefix.hpp boost/config/abi/borland_suffix.hpp boost/config/abi/msvc_prefix.hpp boost/config/abi/msvc_suffix.hpp boost/config/abi_prefix.hpp boost/config/abi_suffix.hpp boost/config/auto_link.hpp boost/config/compiler/borland.hpp boost/config/compiler/comeau.hpp boost/config/compiler/common_edg.hpp boost/config/compiler/compaq_cxx.hpp boost/config/compiler/digitalmars.hpp boost/config/compiler/gcc.hpp boost/config/compiler/gcc_xml.hpp boost/config/compiler/greenhills.hpp boost/config/compiler/hp_acc.hpp boost/config/compiler/intel.hpp boost/config/compiler/kai.hpp boost/config/compiler/metrowerks.hpp boost/config/compiler/mpw.hpp boost/config/compiler/sgi_mipspro.hpp boost/config/compiler/sunpro_cc.hpp boost/config/compiler/vacpp.hpp boost/config/compiler/visualc.hpp boost/config/no_tr1/complex.hpp boost/config/no_tr1/functional.hpp boost/config/no_tr1/memory.hpp boost/config/no_tr1/utility.hpp boost/config/platform/aix.hpp boost/config/platform/amigaos.hpp boost/config/platform/beos.hpp boost/config/platform/bsd.hpp boost/config/platform/cygwin.hpp boost/config/platform/hpux.hpp boost/config/platform/irix.hpp boost/config/platform/linux.hpp boost/config/platform/macos.hpp boost/config/platform/qnxnto.hpp boost/config/platform/solaris.hpp boost/config/platform/win32.hpp boost/config/posix_features.hpp boost/config/requires_threads.hpp boost/config/select_compiler_config.hpp boost/config/select_platform_config.hpp boost/config/select_stdlib_config.hpp boost/config/stdlib/dinkumware.hpp boost/config/stdlib/libcomo.hpp boost/config/stdlib/libstdcpp3.hpp boost/config/stdlib/modena.hpp boost/config/stdlib/msl.hpp boost/config/stdlib/roguewave.hpp boost/config/stdlib/sgi.hpp boost/config/stdlib/stlport.hpp boost/config/stdlib/vacpp.hpp boost/config/suffix.hpp boost/config/user.hpp boost/current_function.hpp boost/detail/atomic_count.hpp boost/detail/atomic_count_gcc.hpp boost/detail/atomic_count_pthreads.hpp boost/detail/atomic_count_win32.hpp boost/detail/bad_weak_ptr.hpp boost/detail/interlocked.hpp boost/detail/shared_count.hpp boost/detail/shared_ptr_nmt.hpp boost/detail/sp_counted_base.hpp boost/detail/sp_counted_base_cw_ppc.hpp boost/detail/sp_counted_base_gcc_ia64.hpp boost/detail/sp_counted_base_gcc_ppc.hpp boost/detail/sp_counted_base_gcc_x86.hpp boost/detail/sp_counted_base_nt.hpp boost/detail/sp_counted_base_pt.hpp boost/detail/sp_counted_base_w32.hpp boost/detail/sp_counted_impl.hpp boost/detail/workaround.hpp boost/non_type.hpp boost/shared_ptr.hpp boost/throw_exception.hpp boost/type.hpp boost/version.hpp The list of config files is still rather long, but I'd say that's a result - certainly not bad for 10 minutes programming :-) John.

Peter Dimov

20 Feb 20 Feb

2:24 p.m.

New subject: [bcp] [shared_ptr] Extractingsubsetofboostforshared_ptr - 3.2MB?

John Maddock wrote:

...

...
Or:

#ifdef BOOST_SP_USE_QUICK_ALLOCATOR

//@bcp optional "quick_allocator is only used if explicitly requested" #include

#endif

I've just added this to bcp as an experiment (it's in mainline cvs), with the extra line above added to shared_ptr as well, ...

This extra line doesn't seem to be in CVS yet, did you forget to commit, or should I add it for you? :-)

John Maddock

6:32 p.m.

New subject: [bcp] [shared_ptr]Extractingsubsetofboostforshared_ptr - 3.2MB?

...

This extra line doesn't seem to be in CVS yet, did you forget to commit, or should I add it for you? :-)

Apologies: I should have been clearer: bcp has been updated to recognise //@bcp optional message-text #include But I deliberately haven't changed any headers, please do go ahead and experiment with this and make changes to whatever headers you see fit to :-) There have also been some useful suggestions from Jeff Garland that I want to mull over, conceptually we could have some thing like: //@bcp default=enable name=Wow Optional feature Wow is enabled by default use --disable-Wow to suppress it's inclusion or: //@bcp default=disable name=Wow Optional feature Wow is disabled by default use --enable-Wow to force it's inclusion and //@bcp suppress which would unconditionally suppress the inclusion of the following include. But I haven't really thought all that through yet. Jeff's idea of a separate file for optional includes would make it easier for lib authors to exclude inclusion of headers through a whole set of files, but requires rather more modifications to bcp than may be wise this close to a release? There may be synergies with Boost.Build as well, as I believe that scans headers in a similar way to bcp. John.

David Abrahams

18 Feb 18 Feb

6:01 p.m.

New subject: [bcp] [shared_ptr] [wave] Extracting subset of boostforshared_ptr - 3.2MB?

"John Maddock" writes:

...

...
Unfortunately bcp isn't smart enough to not look into optional dependencies that are guarded by #ifdef. In this particular case it follows the dependency to detail/quick_allocator.hpp, which in turn brings in the type_traits and mpl libraries. But quick_allocator.hpp is only used if you #define BOOST_SP_USE_QUICK_ALLOCATOR.

Peter, this comes up so often, I wonder is it worth while "hiding" that optional dependency behind a #define, as in:

#define BOOST_SP_QA

#ifdef BOOST_SP_USE_QUICK_ALLOCATOR #include BOOST_SP_QA #endif

If doing that works to fool bcp, we need a better system. It would be a lot of work, but maybe it would be the first real use of Wave as a library rather than just as a preprocessor. We could use the graph library to do a real search with some knowledge of compiler and configuration #defines. It would be a conservative search, so it would follow *all* PP #ifdef branches unless it could prove that they shouldn't be followed. Wow, that sounds like an interesting but difficult dynamic programming problem. I wonder if it's even tractable? -- Dave Abrahams Boost Consulting www.boost-consulting.com

David Abrahams

6:16 p.m.

New subject: [bcp] [shared_ptr] [wave] Extracting subset of boostforshared_ptr - 3.2MB?

David Abrahams writes:

...

"John Maddock" writes:

...
...
Unfortunately bcp isn't smart enough to not look into optional dependencies that are guarded by #ifdef. In this particular case it follows the dependency to detail/quick_allocator.hpp, which in turn brings in the type_traits and mpl libraries. But quick_allocator.hpp is only used if you #define BOOST_SP_USE_QUICK_ALLOCATOR.

Peter, this comes up so often, I wonder is it worth while "hiding" that optional dependency behind a #define, as in:

#define BOOST_SP_QA

#ifdef BOOST_SP_USE_QUICK_ALLOCATOR #include BOOST_SP_QA #endif

If doing that works to fool bcp, we need a better system. It would be a lot of work, but maybe it would be the first real use of Wave as a library rather than just as a preprocessor. We could use the graph library to do a real search with some knowledge of compiler and configuration #defines. It would be a conservative search, so it would follow *all* PP #ifdef branches unless it could prove that they shouldn't be followed. Wow, that sounds like an interesting but difficult dynamic programming problem. I wonder if it's even tractable?

I like Peter's idea much better in the near term ;-) -- Dave Abrahams Boost Consulting www.boost-consulting.com

John Maddock

4:29 p.m.

New subject: [bcp] [shared_ptr] Extracting subset of boost forshared_ptr - 3.2MB?

...

Is this correct? Am I using bcp correctly? Are there some paths that bcp is taking that aren't really neccessary? Can I narrow this huge vat of files down to the ones that I *really* need, or is this it? How?

I updated the docs recently because of this very case: "File dependencies are found as follows: * C++ source files are scanned for #includes, all #includes present in the boost source tree will then be scanned for their dependencies and so on. * C++ source files are associated with the name of a library, if that library has source code (and possibly build data), then include that source in the dependencies. * C++ source files are checked for dependencies on Boost.test (for example to see if they use cpp_main as an entry point). * HTML files are scanned for immediate dependencies (images and style sheets, but not links). It should be noted that in practice bcp can produce a rather "fat" list of dependencies, reasons for this include: * It searches for library names first, so using "regex" as a name will give you everything in the libs/regex directory and everything that depends on. This can be a long list as all the regex test and example programs will get scanned for their dependencies. If you want a more minimal list, then try using the names of the headers you are actually including, or use the --scan option to scan your source code. * If you include the header of a library with separate source, then you get that libraries source and all it's dependencies. This is deliberate and in general those extra dependencies are needed. * When you include a header, bcp doesn't know what compiler you're using, so it follows all possible preprocessor paths. If you're distributing a subset of Boost with you're application then that is what you want to have happen in general. The last point above can result in a substantial increase in the number of headers found compared to most peoples expectations. For example bcp finds 274 header dependencies for boost/shared_ptr.hpp: by running bcp in report mode we can see why all these headers have been found as dependencies: * All of the Config library headers get included (52 headers, would be about 6 for one compiler only). * A lot of MPL and type traits code that includes workarounds for broken compilers that you may or may not need. Tracing back through the code shows that most of these aren't needed unless the user has defined BOOST_SP_USE_QUICK_ALLOCATOR, however bcp isn't aware of whether that preprocessor path will be taken or not, so the headers get included just in case. This adds about 48 headers (type traits), plus another 49 from MPL. * The Preprocessor library gets used heavily by MPL: this adds another 96 headers. * The Shared Pointer library contains a lot of platform specific code, split up into around 22 headers: normally your compiler would need only a couple of these files. As you can see the number of dependencies found are much larger than those used by any single compiler, however if you want to distribute a subset of Boost that's usable in any configuration, by any compiler, on any platform then that's exactly what you need. If you want to figure out which Boost headers are being used by your specific compiler then the best way to find out is to prepocess the code and scan the output for boost header includes. You should be aware that the result will be very platform and compiler specific, and may not contain all the headers needed if you so much as change a compiler switch (for example turn on threading support)." So... to answer your specific question: if you want your code to be portable to *any* compiler then yes you need all those headers, otherwise using sed to process the preprocessed source would give you a more slimline set of headers. HTH, John

6841

Age (days ago)

6843

Last active (days ago)

List overview

Download

12 comments

5 participants

participants (5)

David Abrahams
Jeff Garland
John Maddock
Peter Dimov
Tyler Mandry