BOOST_PP_ITERATE for huge mpl and fusion
"Not to 50!!!" - Princess Bride Things like mpl::vector and fusion::map have a configured limit like BOOST_MPL_LIMIT_VECTOR_SIZE. The documentation is dodgy and doesn't mention that while it is configurable you can't really configure it to be more than 50. Many people have stumbled here: http://boost.2283326.n4.nabble.com/msm-Boost-MPL-vector-limit-size-exceeded-... http://lists.boost.org/Archives/boost/2004/08/70900.php http://comments.gmane.org/gmane.comp.lib.boost.user/55679 http://lists.boost.org/Archives/boost/2010/09/170443.php Appearantly there are some scripts to generate these headers, but they may not work anymore. I had to rush hack things in my code to make several containers go over 50 in one compilation scope. #define ULTIMATE_N 70 #include <boost/mpl/vector/vector50.hpp> namespace boost { namespace mpl { # define BOOST_PP_ITERATION_PARAMS_1 \ (3,(51, ULTIMATE_N, <boost/mpl/vector/aux_/numbered.hpp>)) # include BOOST_PP_ITERATE() }} I'd really like to fix this in boost. AFAICT all I have to do is find all *50.hpp files and replace the 50 with the appropriate config limit. I count 74 instances of this stuff going on! Mostly it's phoenix, mpl, and fusion. So to wrap it up my questions are: 1) Do the scripts still work? Does anyone use them? Are they regularly tested? 2) Does anyone see a problem with my simple fix? Chris
On 03/07/2012 01:08 PM, Hite, Christopher wrote:
"Not to 50!!!" - Princess Bride
Things like mpl::vector and fusion::map have a configured limit like BOOST_MPL_LIMIT_VECTOR_SIZE.
The documentation is dodgy and doesn't mention that while it is configurable you can't really configure it to be more than 50.
Just disable preprocessed headers.
Mathias Gaunard wrote:
The documentation is dodgy and doesn't mention that while it is configurable you can't really configure it to be more than 50.
Just disable preprocessed headers.
I found this but wasn't able to find it in the new docs: http://www.boost.org/doc/libs/1_33_1/libs/mpl/doc/refmanual/cfg-no-preproces... I tried this. It's how I'd expect it to work. #define BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS #define BOOST_MPL_LIMIT_VECTOR_SIZE 60 #include <boost/mpl/vector.hpp> Here's what I get (gcc-4.6.0): /fs/tools/L3/boost_1_43_0/boost/mpl/vector.hpp:36:73: fatal error: boost/mpl/vector/vector60.hpp: No such file or directory compilation terminated. What's BOOST_MPL_PREPROCESSING_MODE? It just seems to break things more. Does fusion have a separate config option? Chris
I'm very certain it is not possible to use sizes large than 50 with mpl/fusion/phoenix. I count two impls 1) has 20 elements all generated boost_1_49_0/boost/mpl/aux_/preprocessed/gcc/vector.hpp 2) stops at 50 /fs/tools/L3/boost_1_49_0/boost/mpl/vector/vector50.hpp I don't know if there's a way to generate more. Probably you need python. IMHO this should be documented so users can do it. This doesn't work: #define BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS #define BOOST_MPL_LIMIT_VECTOR_SIZE 60 #include <boost/mpl/vector.hpp> boost_1_49_0/boost/mpl/vector.hpp:36:73: fatal error: boost/mpl/vector/vector60.hpp: No such file or directory Can someone please help. I've upgraded my copy of boost. I've tried figuring out how these headers work. I'm pretty sure I could fix it if I really had to by replacing the 50 in the last file with the *_SIZE. I have this ugly hack in my code to keep things workiing. Would someone please help?!? Chris
Hi Christopher, On Wed, 07 Mar 2012 06:08:01 -0600, Hite, Christopher <Christopher.Hite@partner.commerzbank.com> wrote:
"Not to 50!!!" - Princess Bride
:)
Things like mpl::vector and fusion::map have a configured limit like BOOST_MPL_LIMIT_VECTOR_SIZE.
Just to clarify, BOOST_MPL_LIMIT_VECTOR_SIZE determines the maximum arity of the vector's "constructor" in its variadic form (i.e. vector<>). Somewhat contrary to its name, it does *not* determine the vector's maximum size. Rather, the latter is governed by what's available to the library on the compiler side: - on compilers that support typeof [1], there is really no predefined limit for the vector's size other than general compiler limitations on the template nesting depth and the likes. - on compilers w/o typeof support the default limit is 50, as you've discovered. The original rationale for that limit is cited in one of the posts you are linking to below: Increasing the default limit to, let's say, 100 elements means plus ~120 KB to the size of the distribution, and I'm not sure the need is widespread enough to make everybody pay for it. Given the current size of the distribution nowadays, this looks like a non-issue.
The documentation is dodgy and doesn't mention that while it is configurable you can't really configure it to be more than 50.
The docs on this topic are definitely lacking. Let me know if this post clarifies things for you, and I'll rework it into a doc page.
Many people have stumbled here: http://boost.2283326.n4.nabble.com/msm-Boost-MPL-vector-limit-size-exceeded-... http://lists.boost.org/Archives/boost/2004/08/70900.php http://comments.gmane.org/gmane.comp.lib.boost.user/55679 http://lists.boost.org/Archives/boost/2010/09/170443.php Appearantly there are some scripts to generate these headers, but they may not work anymore.
They do, at least on Windows, and they are here: http://svn.boost.org/svn/boost/trunk/libs/mpl/preprocessed/
I had to rush hack things in my code to make several containers go over 50 in one compilation scope.
#define ULTIMATE_N 70 #include <boost/mpl/vector/vector50.hpp>
namespace boost { namespace mpl {
# define BOOST_PP_ITERATION_PARAMS_1 \ (3,(51, ULTIMATE_N, <boost/mpl/vector/aux_/numbered.hpp>)) # include BOOST_PP_ITERATE()
}}
This is actually a legit way of going over the predefined limit; there is really nothing wrong with putting this in a header and including it where you need vectors over 50 elements. The only downside of using this instead of pre-generated preprocessed headers is potential loss of compilation speed, emphasis on potential.
I'd really like to fix this in boost. AFAICT all I have to do is find all *50.hpp files and replace the 50 with the appropriate config limit.
I count 74 instances of this stuff going on! Mostly it's phoenix, mpl, and fusion.
So to wrap it up my questions are: 1) Do the scripts still work? Does anyone use them? Are they regularly tested?
The scripts still work, they are used when a fix is made in one of the headers affecting the preprocessed output, and no, they are not regularly tested, but they also don't regularly break :).
2) Does anyone see a problem with my simple fix?
I can't really speak for the other libraries, but I have no problem with bumping up the default MPL limit to, say, 100. [1] ... and decltype, once you apply a four-lines patch. -- Aleksey Gurtovoy MetaCommunications Engineering
Aleksey Gurtovoy wrote:
Things like mpl::vector and fusion::map have a configured limit like BOOST_MPL_LIMIT_VECTOR_SIZE.
Just to clarify, BOOST_MPL_LIMIT_VECTOR_SIZE determines the maximum arity of the vector's "constructor" in its variadic form (i.e. vector<>). Somewhat contrary to its name, it does *not* determine the vector's maximum size. Rather, the latter is governed by what's available to the library on the compiler side:
- on compilers that support typeof [1], there is really no predefined limit for the vector's size other than general compiler limitations on the template nesting depth and the likes.
- on compilers w/o typeof support the default limit is 50, as you've discovered. The original rationale for that limit is cited in one of the posts you are linking to below:
Increasing the default limit to, let's say, 100 elements means plus ~120 KB to the size of the distribution, and I'm not sure the need is widespread enough to make everybody pay for it.
Given the current size of the distribution nowadays, this looks like a non-issue.
The docs on this topic are definitely lacking. Let me know if this post clarifies things for you, and I'll rework it into a doc page. Thanks, those few lines there already help a lot!
I know you're not responsible for fusion, but I also don't know if it needs the "constructor".
I had to rush hack things in my code to make several containers go over 50 in one compilation scope.
This is actually a legit way of going over the predefined limit; there is really nothing wrong with putting this in a header and including it where you need vectors over 50 elements. The only downside of using this instead of pre-generated preprocessed headers is potential loss of compilation > > speed, emphasis on potential. It may work for MPL I just upgraded boost and fusion now checks the limit and complains:
boost_1_49_0/boost/fusion/container/map/detail/preprocessed/as_map.hpp:21:2: error: #error "FUSION_MAX_MAP_SIZE out of bounds for preprocessed headers" I spent about an hour trying to hack around that. It looks like I'll either have to change boost or copy paste a good deal of fusion's details into a header. I realize you're not repsonsible for fusion, but as far as I can tell the other libraries copied your techniques and if we figure this out for MPL we can apply it to them too. I think we could fix all of these in the library by just changing the loops like: # define BOOST_PP_ITERATION_PARAMS_1 \ (3,(41, 50, <boost/mpl/map/aux_/numbered.hpp>)) to # define BOOST_PP_ITERATION_PARAMS_1 \ (3,(41, BOOST_MPL_LIMIT_MAP_SIZE, <boost/mpl/map/aux_/numbered.hpp>)) so the last file loops as often as the user wants. It just seems like you snapped defeat from the jaws of victory by putting the limit in. Is there something I missed like some compiler's preprocessor not handling that variable? Would it require you to change your scripts?
They do, at least on Windows, and they are here: http://svn.boost.org/svn/boost/trunk/libs/mpl/preprocessed/ ... The scripts still work, they are used when a fix is made in one of the headers affecting the preprocessed output, and no, they are not regularly tested, but they also don't regularly break :).
Are you talking about boost_1_49_0/libs/mpl/preprocessed/preprocess.cmd ? Where do I put the N?
I can't really speak for the other libraries, but I have no problem with bumping up the default MPL limit to, say, 100. That would take care of my project, but I'd hate to come back later and ask you for more.
[1] ... and decltype, once you apply a four-lines patch. Perhaps I should go fight for c++11. Would the four-lines be #ifdef'ed in future versions of boost?
Thanks for helping me! Chris
On 3/20/2012 7:35 PM, Hite, Christopher wrote:
It may work for MPL I just upgraded boost and fusion now checks the limit and complains:
boost_1_49_0/boost/fusion/container/map/detail/preprocessed/as_map.hpp:21:2: error: #error "FUSION_MAX_MAP_SIZE out of bounds for preprocessed headers"
I spent about an hour trying to hack around that. It looks like I'll either have to change boost or copy paste a good deal of fusion's details into a header.
I realize you're not repsonsible for fusion, but as far as I can tell the other libraries copied your techniques and if we figure this out for MPL we can apply it to them too.
No, Fusion does not use MPL's techniques at all. It does not use any scripts to generate the prepreocessed headers. Instead, Fusion uses Wave. Ditto for Phoenix-3. Going beyond the limts is very simple: simply #define BOOST_FUSION_DONT_USE_PREPROCESSED_FILES. Regards, -- Joel de Guzman http://www.boostpro.com http://boost-spirit.com
On 03/20/12 06:51, Joel de Guzman wrote:
On 3/20/2012 7:35 PM, Hite, Christopher wrote:
It may work for MPL I just upgraded boost and fusion now checks the limit and complains:
boost_1_49_0/boost/fusion/container/map/detail/preprocessed/as_map.hpp:21:2: error: #error "FUSION_MAX_MAP_SIZE out of bounds for preprocessed headers"
I spent about an hour trying to hack around that. It looks like I'll either have to change boost or copy paste a good deal of fusion's details into a header.
I realize you're not repsonsible for fusion, but as far as I can tell the other libraries copied your techniques and if we figure this out for MPL we can apply it to them too.
No, Fusion does not use MPL's techniques at all. It does not use any scripts to generate the prepreocessed headers. Instead, Fusion uses Wave. Ditto for Phoenix-3. Going beyond the limts is very simple: simply #define BOOST_FUSION_DONT_USE_PREPROCESSED_FILES.
Regards, Hmm. Aleksey, could MPL use Fusion's technique? It sounds like having a uniform way of generating headers would ease maintenance and maybe avoid the pain that Christopher is now experiencing.
-regards, Larry
Joel de Guzman wrote:
No, Fusion does not use MPL's techniques at all. It does not use any scripts to generate the prepreocessed headers. Instead, Fusion uses Wave. Ditto for Phoenix-3. Going beyond the limts is very simple: simply #define BOOST_FUSION_DONT_USE_PREPROCESSED_FILES.
Awesome!!! That's what I was looking for. I didn't find it because of the order gcc's stack of header includes. We should document it. AFAICT it's not in the docs. We could refer the user to it in the #error. On the other hand why don't we just set * DONT_USE_PREPROCESSED_FILES when the size>50? Chris
Hite, Christopher wrote:
Joel de Guzman wrote:
No, Fusion does not use MPL's techniques at all. It does not use any scripts to generate the prepreocessed headers. Instead, Fusion uses Wave. Ditto for Phoenix-3. Going beyond the limts is very simple: simply #define BOOST_FUSION_DONT_USE_PREPROCESSED_FILES.
Awesome!!! That's what I was looking for. I didn't find it because of the order gcc's stack of header includes.
Wait sorry, I'm still not out of the woods yet. If I leave out the #if'ed stuff below it doesn't compile. It's still much better than it was before, but it seems like I shouldn't have to do this either. #define BOOST_FUSION_DONT_USE_PREPROCESSED_FILES #define FUSION_MAX_MAP_SIZE ULTIMATE_N #define FUSION_MAX_VECTOR_SIZE ULTIMATE_N #if 1 // required for some reason #include <boost/fusion/include/vector50.hpp> namespace boost { namespace fusion { struct vector_tag; struct fusion_sequence_tag; struct random_access_traversal_tag; // expand vector41 to vector50 #define BOOST_PP_FILENAME_1 <boost/fusion/container/vector/detail/vector_n.hpp> #define BOOST_PP_ITERATION_LIMITS (51, ULTIMATE_N) #include BOOST_PP_ITERATE() }} #endif #include <boost/fusion/container/vector.hpp> 129 errors: In file included from /fs/tools/L3/boost_1_49_0/boost/fusion/container/vector/vector.hpp:11:0, from /fs/tools/L3/boost_1_49_0/boost/fusion/container/vector.hpp:26, from decode_separate_xetra.cpp:68: /fs/tools/L3/boost_1_49_0/boost/fusion/container/vector/detail/vector_n_chooser.hpp:65:1: error: 'vector70' does not name a type In file included from /fs/tools/L3/boost_1_49_0/boost/preprocessor/iteration/detail/iter/forward1.hpp:302:0, from /fs/tools/L3/boost_1_49_0/boost/fusion/container/vector/detail/vector_n_chooser.hpp:77, from /fs/tools/L3/boost_1_49_0/boost/fusion/container/vector/vector.hpp:11, from /fs/tools/L3/boost_1_49_0/boost/fusion/container/vector.hpp:26, from decode_separate_xetra.cpp:68: /fs/tools/L3/boost_1_49_0/boost/fusion/container/vector/detail/vector_n_chooser.hpp:103:1: error: 'vector51' does not name a type In file included from /fs/tools/L3/boost_1_49_0/boost/preprocessor/iteration/detail/iter/forward1.hpp:307:0, from /fs/tools/L3/boost_1_49_0/boost/fusion/container/vector/detail/vector_n_chooser.hpp:77, from /fs/tools/L3/boost_1_49_0/boost/fusion/container/vector/vector.hpp:11, from /fs/tools/L3/boost_1_49_0/boost/fusion/container/vector.hpp:26, Chris
participants (5)
-
Aleksey Gurtovoy -
Hite, Christopher -
Joel de Guzman -
Larry Evans -
Mathias Gaunard