- Boost-users - lists.preview.boost.org

[Interprocess] Allocing memory that can be unmapped
by Paul Harris 18 Nov '10

18 Nov '10

Hi, I need to allocate multiple large chunks of memory (many 100MB arrays), and my app is running out of RAM on 32 bit Windows operating systems. Works fine on Linux, as usual. I only need to access maybe 5 of those arrays at a time, so I was hoping to do something like this... create_shared_memory_in_pagefile(); for (each array) { array_offsets[i] = alloc_array_in_shm(); ptr = map_array(array_offsets[i]); do stuff with ptr; unmap_array(ptr); } then later on, when I need access to the array, ptr = map_array(array_offsets[i]); do stuff with ptr; unmap_array(ptr); That would limit the amount of mapped memory, and thus would survive on a 32 bit OS. Note that I don't actually need to share the memory at all, it would be used for only one process. I have been reading up on boost::interprocess, and it looks like it would help me manage the shared memory in a portable way. But what I'm really interested in is a simple allocator that will help me manage the shared memory. But I've noticed that all the allocators and managed memory requires that the entire shared memory chunk is mapped - this is what I'm trying to avoid. Does anyone have any ideas on how I can approach this? I was thinking: it seems that the allocator needs the entire memory mapped as it keeps its datastructures there. Instead, could the allocator store its data structures in RAM, and then I just need to map/unmap the correct addresses when required? thanks Paul

2 1

[proto] : Proto transform with state
by Manjunath Kudlur 18 Nov '10

18 Nov '10

I am trying to create a transform where I replace occurrences of proto::terminal<term> with proto::terminal<newterm<mpl::int_<N> > >. I want N to increment for every proto::terminal<term> encountered. So far I have the following code : #include <boost/proto/proto.hpp> #include <boost/mpl/int.hpp> #include <boost/mpl/arithmetic.hpp> #include <iostream> using namespace std; namespace proto=boost::proto; namespace mpl=boost::mpl; struct term {}; template<typename> struct newterm {}; proto::terminal<term>::type const a = {{}}; proto::terminal<term>::type const b = {{}}; proto::terminal<term>::type const c = {{}}; struct xform_callable : proto::callable { template<typename> struct result; template<typename This, typename N> struct result<This(const term &, N)> { typedef typename proto::result_of::make_expr<proto::tag::terminal, newterm<N> >::type type; }; template<typename This, typename tag, typename LHS, typename RHS> struct result<This(tag, LHS, RHS)> { typedef typename proto::result_of::make_expr<tag, LHS, RHS>::type type; }; template<typename N> typename proto::result_of::make_expr<proto::tag::terminal, newterm<N> >::type operator()(const term &, N) { return proto::make_expr<proto::tag::terminal>(newterm<N>()); } template<typename tag, typename LHS, typename RHS> typename proto::result_of::make_expr<tag, LHS, RHS>::type operator()(tag, LHS l, RHS r) const { return proto::make_expr<tag>(l, r); } }; struct xform : proto::or_< proto::when< proto::terminal<term>, xform_callable(proto::_value, mpl::next<proto::_state>())>, proto::when< proto::plus<xform, xform>, xform_callable(proto::tag_of<proto::_expr>(), xform(proto::_left, proto::_state), xform(proto::_right, proto::_state))> > {}; int main() { mpl::plus<mpl::int_<0>, mpl::int_<1> >(); proto::display_expr(xform()(a+b/c, mpl::int_<0>())); return 0; } I pass in mpl::int_<0>() as the initial state and call mpl::next when I match proto::terminal<term>. As the output would indicate, this is not working. Every time I think I understand proto transforms, something basic like this stumps me. I will be grateful for any hints on how to accomplish this. Thanks, Manjunath

3 3

[serialization] does not compile on PPC or Intel Mac with architecture=combined
by Philipp Münzel 18 Nov '10

18 Nov '10

Hi folks, the build of serialization in boost 1.44 fails when compiling for PPC Mac. On a native Power PC with gcc 4.0.1 the build fails: darwin.compile.c++ bin.v2/libs/serialization/build/darwin-4.0.1/release/link-static/ threading-multi/basic_xml_archive.o darwin.compile.c++ bin.v2/libs/serialization/build/darwin-4.0.1/release/link-static/ threading-multi/binary_iarchive.o ./boost/archive/basic_binary_iarchive.hpp:74: error: invalid application of 'sizeof' to incomplete type 'boost::STATIC_ASSERTION_FAILURE<false>' "g++" -ftemplate-depth-128 -O3 -finline-functions -Wno-inline -Wall -no-cpp-precomp -gdwarf-2 -Wno-long-double -DBOOST_ALL_NO_LIB=1 -DNDEBUG -I"." -c -o "bin.v2/libs/serialization/build/darwin-4.0.1/release/link-static/ threading-multi/binary_iarchive.o" "libs/serialization/src/binary_iarchive.cpp" ...failed darwin.compile.c++ bin.v2/libs/serialization/build/darwin-4.0.1/release/link-static/ threading-multi/binary_iarchive.o... ...failed updating 1 target... ...updated 74 targets... On an Intel Mac, with OSX 10.5 and gcc 4.2.1 the build fails when architecture=combined is added to the bjam command: bin.v2/libs/serialization/build/darwin-4.2.1/release/ architecture-combined/link-static/threading-multi/binary_iarchive.o In file included from ./boost/archive/binary_iarchive_impl.hpp:22, from ./boost/archive/binary_iarchive.hpp:20, from libs/serialization/src/binary_iarchive.cpp:14: ./boost/archive/basic_binary_iarchive.hpp:74: error: invalid application of ‘sizeof’ to incomplete type ‘boost::STATIC_ASSERTION_FAILURE<false>’ lipo: can't figure out the architecture type of: /var/tmp//cc5y68Un.out "g++" -ftemplate-depth-128 -O3 -finline-functions -Wno-inline -Wall -no-cpp-precomp -gdwarf-2 -arch i386 -arch ppc -DBOOST_ALL_NO_LIB=1 -DNDEBUG -I"." -c -o "bin.v2/libs/serialization/build/darwin-4.2.1/release/architecture- combined/link-static/threading-multi/binary_iarchive.o" "libs/serialization/src/binary_iarchive.cpp" ...failed darwin.compile.c++ bin.v2/libs/serialization/build/darwin-4.2.1/release/architecture- combined/link-static/threading-multi/binary_iarchive.o... ...failed updating 1 target... ...updated 96 targets... When using bjam without architecture=combined on an Intel Mac, the build succeeds. That makes boost.serialization unusable on PCC, because it can neither be build native, nor for a universal binary. I wonder how this could slip through the rigid testing procedures of boost... Regards, Philipp

2 1

Re: [Boost-users] Hybrid parallelism, no more + mpi+serialization, many questions
by Hicham Mouline 18 Nov '10

18 Nov '10

-----Original Message-----From: "Brian Budge" [brian.budge(a)gmail.com]Date: 18/11/2010 03:20 PMTo: boost-users(a)lists.boost.orgSubject: Re: [Boost-users] Hybrid parallelism, no more + mpi+serialization, many questions>> The large calculation that I currently do serially and that I intend to>> parallelize is the maximum of the return values of a large number of>> evaluations of a given "function" in the mathematical sense.>> The number of arguments of the function is only known at runtime.>> Let's say it is determined at runtime that the number of arguments is 10, ie>> we have 10 arguments x0, ..., x9>> Each argument can take a different number of values, for e.g. x0 can be>> x0_0, x0_1 .... x0_n0>> x1 can be x1_0, x1_1, ...x1_n1 and so on...n0 and n1 are typically known at>> runtime and different>>>> so serially, I run>> f(x0_0, x1_0, ..., x9_0)>> f(x0_0, x1_0, ..., x9_1)>> ...>> f(x0_0, x1_0, ..., x9_n9)>> then with all the x8 then all the x7 ... then all the x0.>> There is n0*n1*...*n9 runs>> Then I get the maximum of the return values.>>>>>> Imagining I have N mpi processes, ideally each process would run>> n0*n1*...*n9/N function evaluations.>> How do I split?>>>> In terms of current implementation, each of the x is a boost::variant over 4>> types:>> a double, a <min,max> pair, a <min, max, increment> triplet or a>> vector<double>>> A visitor is applied recursively to the variants in order to traverse the>> whole parameter space. >> apply_visitor on x0 => say if x0 is a triplet, then>> for (x0 from min to max with increment)>> apply_visitor on x1>> and so on until x9, then we actually call the f function with all the>> arguments collected so far.>>>> How can one parallelize such a beast?>>>> rds,>>>>>> _______________________________________________>This partly depends on how many processors/machines you have>available. You need to find a way of partitioning your state space>into tasks, and then doling those tasks out to processes/threads. How>expensive is f()? How much memory is used?>Brian f() takes about 1ms. I have a couple of billion n-tuples in the parameter space for now but thinking of reaching easily a couple of trillions.In the serial mode, only about 20Mbytes of RAM is used. The parameter space is not generated ahead. It looks like a tree.The tree is generated, when I reach a leaf, I evaluate f(), and compare the return to the current maximum.So I'm not storing much.rds,

2 1

Re: [Boost-users] Hybrid parallelism, no more + mpi+serialization, many questions
by Hicham Mouline 18 Nov '10

18 Nov '10

hello,Subsequent to a previous thread asking whether to merge MPI and Openmp to parallelize a large problem, I've been advised to go through MPI only as it would be simpler and that MPI implementations on the same box use shared memory which doesn't have a huge cost (still some compared to a uniprocess multithread where objects are actually shared naturally.... writing this, actually a question comes up:1. in the "shared memory" of many mpi processes on the same box, is an object (say a list of numbers) actually shared between the 2 processes address spaces? I guess not unless one explicitly make it so with the "shared memory API" (unix specific?)So, I currently have a serial application with a GUI that runs some calculations.My next step is to use OpenMPI with the help of the Boost.MPI wrapper library in C++ to parallelize those calculations.There is a set of static data objects created once at startup or loaded from files.2. what are the pros/cons of loading the static data objects individually from each separate mpi process vs broadcasting the static data via MPI itself after only the master reads/sets up the static data?3. Is it possible to choose the binary archive instead of the text archive when serializing my user-defined types?Where do I deal with the endianness issue given that I may have Intel/Sparc/PowerPC CPUs?regards,

6 8

compile error on Solaris 10
by Kelvin Cheng 18 Nov '10

18 Nov '10

Hi, My environment is Solaris 10 , g++ 3.3.2 I try to use the header files directly using the definition -DBOOST_REGEX_NO_LIB I got the following errors doing a simple compilation using regex. g++ -fPIC -c -DUNIX -DSOLARIS -Dsun -D_PTHREADS -D_GLIBCXX_HAVE_MBSTATE_T -DBOOST_REGEX_NO_LIB -I./ -I../shared/ -I../shared/boost ndbdriver.cpp -o ndbdriver.o In file included from ../shared/boost/boost/regex/v4/regex_traits.hpp:35, from ../shared/boost/boost/regex/regex_traits.hpp:27, from ../shared/boost/boost/regex/v4/regex.hpp:39, from ../shared/boost/boost/regex.hpp:31, from ndbdriver.cpp:39: ../shared/boost/boost/regex/v4/regex_traits_defaults.hpp:79: error: redefinition of `bool boost::re_detail::is_combining(charT) [with charT = char]' ../shared/boost/boost/regex/v4/regex_traits_defaults.hpp:74: error: `bool boost::re_detail::is_combining(charT) [with charT = char]' previously declared here In file included from ../shared/boost/boost/regex/v4/regex.hpp:58, from ../shared/boost/boost/regex.hpp:31, from ndbdriver.cpp:39: ../shared/boost/boost/regex/v4/states.hpp:60: error: redefinition of `struct boost::re_detail::is_byte<char>' ../shared/boost/boost/regex/v4/states.hpp:58: error: previous definition of ` struct boost::re_detail::is_byte<char>' In file included from ../shared/boost/boost/functional/hash.hpp:6, from ../shared/boost/boost/regex/v4/basic_regex.hpp:23, from ../shared/boost/boost/regex/v4/regex.hpp:67, from ../shared/boost/boost/regex.hpp:31, from ndbdriver.cpp:39: ../shared/boost/boost/functional/hash/hash.hpp: In function `size_t boost::hash_value(char)': ../shared/boost/boost/functional/hash/hash.hpp:152: error: redefinition of ` size_t boost::hash_value(char)' ../shared/boost/boost/functional/hash/hash.hpp:142: error: `size_t boost::hash_value(char)' previously defined here ../shared/boost/boost/functional/hash/hash.hpp:152: error: redefinition of ` size_t boost::hash_value(char)' ../shared/boost/boost/functional/hash/hash.hpp:142: error: `size_t boost::hash_value(char)' previously defined here ../shared/boost/boost/functional/hash/hash.hpp: In member function `size_t boost::hash<char>::operator()(char) const': ../shared/boost/boost/functional/hash/hash.hpp:405: error: call of overloaded ` hash_value(char&)' is ambiguous ../shared/boost/boost/functional/hash/hash.hpp:137: error: candidates are: size_t boost::hash_value(bool) ../shared/boost/boost/functional/hash/hash.hpp:142: error: size_t boost::hash_value(char) ../shared/boost/boost/functional/hash/hash.hpp:147: error: size_t boost::hash_value(unsigned char) ../shared/boost/boost/functional/hash/hash.hpp:157: error: size_t boost::hash_value(short int) ../shared/boost/boost/functional/hash/hash.hpp:162: error: size_t boost::hash_value(short unsigned int) ../shared/boost/boost/functional/hash/hash.hpp:167: error: size_t boost::hash_value(int) ../shared/boost/boost/functional/hash/hash.hpp:172: error: size_t boost::hash_value(unsigned int) ../shared/boost/boost/functional/hash/hash.hpp:177: error: size_t boost::hash_value(long int) ../shared/boost/boost/functional/hash/hash.hpp:182: error: size_t boost::hash_value(long unsigned int) ../shared/boost/boost/functional/hash/hash.hpp:188: error: size_t boost::hash_value(wchar_t) ../shared/boost/boost/functional/hash/hash.hpp:195: error: size_t boost::hash_value(long long int) ../shared/boost/boost/functional/hash/hash.hpp:200: error: size_t boost::hash_value(long long unsigned int) ../shared/boost/boost/functional/hash/hash.hpp:314: error: size_t boost::hash_value(float) ../shared/boost/boost/functional/hash/hash.hpp:319: error: size_t boost::hash_value(double) ../shared/boost/boost/functional/hash/hash.hpp:324: error: size_t boost::hash_value(long double) ../shared/boost/boost/functional/hash/hash.hpp:152: error: size_t boost::hash_value(char) ../shared/boost/boost/functional/hash/hash.hpp: At global scope: ../shared/boost/boost/functional/hash/hash.hpp:406: error: redefinition of ` struct boost::hash<char>' ../shared/boost/boost/functional/hash/hash.hpp:405: error: previous definition of `struct boost::hash<char>' All the errors are from related to the basic types when expanded from the templates. Any idea what's going wrong ? -- Kelvin KF

2 2

Interest in B-tree library for Boost?
by Beman Dawes 17 Nov '10

17 Nov '10

A Boost B-tree library would provide disk-based associative containers that scale all the way from really, really, small to really, really, large. B-trees perform well on hardware ranging from ancient floppy disk drives all the way up to humongous disk arrays. They are the technology behind most high-performance disk file systems and databases. If you don't know what a B-tree is, or think that "B-tree" is an abbreviation for "Binary tree", you might want to read the Wikipedia B-Tree article: http://en.wikipedia.org/wiki/B-tree. Knuth and other computer science texts also supply descriptions. See http://mysite.verizon.net/beman/btree/index.html for a more complete description of the proposed library. A preliminary implementation is available. Download http://mysite.verizon.net/beman/btree/btree-preview-1.zip, and unpack into a current Boost distribution. A few existing Jamfiles will be overwritten. The only compilers tested and working so far are VC++ 10 and GCC/MinGW 4.5. You can view the library online at http://github.com/Beman/Boost-Btree There is no detailed documentation yet, but if you know how to use std::map and other standard library associative containers, you already know most of what you need to know. Critical differences are documented in the Library Characteristics section of the index.html page referenced above. Initial timing tests have been very encouraging. The time test compares the btree_map implementation to std::map. As expected, a btree_map is many times slower than a std::map, given a cold operating system file cache and constraining the btree_map's caching to a minimum. But at the other extreme, when the O/S file cache is warm and the btree_map's cache is as large as the file, the btree_map is very similar in speed to std::map.The btree library's code is still very new, so timings and everything else may be subject to lots of revision. Any interest? --Beman

5 11

Valgrind false positive in make_shared
by Dave Steffen 17 Nov '10

17 Nov '10

Hi Folks I've been tracking down some almost-certainly-false-positives from Valgrind. Consider the following code: #include <iostream> #include <boost/shared_ptr.hpp> #include <boost/make_shared.hpp> using namespace std; using boost::make_shared; struct StateSpace { public: StateSpace(int spaceType) { if (spaceType == 4) cout << "Some Output" << endl; } }; class State { public: State() : m_rep(make_shared<StateSpace>(0) ) {} boost::shared_ptr<StateSpace> m_rep; }; int main() { State state; } Valgrind 3.6 reports this: ==8198== Memcheck, a memory error detector ==8198== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==8198== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for copyright info ==8198== Command: testState ==8198== ==8198== Conditional jump or move depends on uninitialised value(s) ==8198== at 0x401659: StateSpace::StateSpace(int) (testState.cpp:14) ==8198== by 0x401796: boost::shared_ptr<StateSpace> boost::make_shared<StateSpace, int, >(int&&, ) (make_shared.hpp:150) ==8198== by 0x4016BD: State::State() (testState.cpp:22) ==8198== by 0x401317: main (testState.cpp:29) ==8198== ==8198== ==8198== HEAP SUMMARY: ==8198== in use at exit: 464 bytes in 5 blocks ==8198== total heap usage: 240 allocs, 236 frees, 45,162 bytes allocated ==8198== ==8198== LEAK SUMMARY: ==8198== definitely lost: 0 bytes in 0 blocks ==8198== indirectly lost: 0 bytes in 0 blocks ==8198== possibly lost: 0 bytes in 0 blocks ==8198== still reachable: 464 bytes in 5 blocks ==8198== suppressed: 0 bytes in 0 blocks ==8198== Reachable blocks (those to which a pointer was found) are not shown. ==8198== To see them, rerun with: --leak-check=full --show-reachable=yes ==8198== ==8198== For counts of detected and suppressed errors, rerun with: -v ==8198== Use --track-origins=yes to see where uninitialised values come from ==8198== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5) Interestingly, if I change State() : m_rep(make_shared<StateSpace>(0) ) {} to State() : m_rep(new StateSpace(0)) {} the Valgrind error goes away. This is, of course, a greatly boiled-down version of our actual code, which we're very confident works correctly. Any ideas about why Valgrind doesn't like the call to make_shared? Any ideas about how to silence this warning (other than adding it to Valgrind's suppressions file)? Thanks very much. -- Dave Steffen, Ph.D. - Software Engineer Numerica Corporation <http://www.numerica.us> 4850 Hahns Peak Drive, Suite 200 Loveland, Colorado 80538 main (970) 461-2000 x 227 direct (970) 612-2327 Email: dave.steffen(a)numerica.us fax (970) 461-2004

3 5

[BGL] Getting/deducing type of custom (internal) properties
by Cedric Laczny 17 Nov '10

17 Nov '10

Hi, this is to substitute the thread: [BGL] Accessing internal properties and returning a "template"-type I think I can now deliver even more details on my problem. Again, I have the following concept: template < class GRAPH, class EDGE_PREDICATE, class VERTEX_PREDICATE > class FilteredGraph { public: /// Defines an alias type to represent the filtered_graph typedef filtered_graph< GRAPH, EDGE_PREDICATE, VERTEX_PREDICATE > FilteredGraphContainer; // // some c'tors, member functions etc. // template< class VERTEXPROPERTIES > VERTEXPROPERTIES& properties(const Vertex& v) const { typename property_map<FilteredGraphContainer, vertex_properties_t>::const_type param = get(vertex_properties, graph_); return (param[v]); } // // some more stuff // private: FilteredGraphContainer graph_; }; The compiler can't obviously deduce the type of the return-value of properties( Vertex v) as this type is not specified in the argument list. This is what I understood so far by finding this: http://www.parashift.com/c++-faq-lite/templates.html#faq-35.4 I have the same properties-function in my Graph-class, which uses custom properties as in http://live.boost.org/doc/libs/1_43_0/libs/graph/doc/using_adjacency_list.h… list-properties The definition of this class basically is: enum vertex_properties_t { vertex_properties }; enum edge_properties_t { edge_properties }; namespace boost { BOOST_INSTALL_PROPERTY(vertex, properties); BOOST_INSTALL_PROPERTY(edge, properties); } /* the graph base class template */ template < typename VERTEXPROPERTIES, typename EDGEPROPERTIES > class Graph { public: /// an adjacency_list like we need it typedef adjacency_list< setS, // disallow parallel edges vecS, // vertex container undirectedS, // undirected graph property<vertex_properties_t, VERTEXPROPERTIES>, property<edge_properties_t, EDGEPROPERTIES> > GraphContainer; // //... // private: GraphContainer graph_; }; This is all, more or less, similar to the template-definitions used in the BGL, or at least that was my intention. Interestingly, when I have a function: template< class GRAPH > void some_function( const GRAPH& g){ GRAPH::Vertex v; // Get some included vertex out of g and assign it to v; v = someVertexFromG( g ); filtered_graph< GRAPH, keep_all, keep_all > fg(g, keep_all(), keep_all()); get(vertex_properties, fg, v); } it works without any problem. Now the question, is it possible to solve this issue for FilteredGraph and keep the same concept as for Graph, or do I need to specify the type of VERTEXPROPERTIES somewhere _beforehand_ in order to let the compiler get this type (not preferred, for obvious reasons)? Best, Cedric

1 1

[StateChart] event processing order in async machine
by Igor R 16 Nov '10

16 Nov '10

Hello, I've got an async. state machine (1.44, MSVC10), and sometimes I observe a very strange behavior: events being processed not in the order they are queued. I spent some time trying to investigate this issue, but due to the asynchronous nature of the processing, it's quite not trivial. So, before I start sinking too deep, I'd like to ensure I don't miss something trivial. Unfortunately, I don't have at the moment any self-contained code, but a very simplified model of my FSM is as follows. Assume there are 2 events: Ev1, Ev2. The both are queued to the FMS from the same thread, always in the same order: Ev1, Ev2. The FSM processes them in another thread. The FMS has some "entry" chain of states that defer the both events: S1 /[defer(Ev1, Ev2)] --> S2 /[defer(Ev1, Ev2)] and finally 2 orthogonal sub-states, one of them ignores the both events, and the other processes the both: ...S2 --> S3<S3_1, S3_2>, S3_1 /[custom_reaction(Ev1), in_state_reaction(Ev2)] What I see is that *sometimes* the reaction for Ev2 is invoked *before* the reaction for Ev1. This "sometimes" really puzzles me: I'm absolutely sure that the order of queuing is always the same - so where can be a "race"? Any idea would be appreciated!

2 19