November 2010 - Boost-users - lists.preview.boost.org

circular_buffer, debug and invalid iterator checking
by Anthony Foglia 19 Nov '10

19 Nov '10

The invalid iterator checking in the circular_buffer library is excessively strict. It checks not when an iterator is dereferenced, but whenever any operator involving that iterator is used. I have an class defined to keep the last few data points, up to either a maximum number, or some condition on the stored data (e.g. the sum must be below a threshold) is met. So I figured I'd store the data in a circular buffer, and push new data onto the front, and store my own iterator into it to use as the end. If my end iterator is pointing to the next to last element in the buffer, when a new item is pushed onto the front of the circular buffer, that iterator becomes invalid. But that's not a problem for me, because I don't read past it. _Except_ the validity of the iterator is checked even when using it in operator== (and other iterator comparison operators), so my lopp condition tests now throw because the end of my ranges are no longer valid. Why isn't iterator invalidation only checked on dereferencing? Is there any suggested convention for changing my code to work under debug? I would try checking for BOOST_CB_ENABLE_DEBUG and catching these cases before the push_front occurs, but that macro is undefined at the end of the header. Or I could try something more complicated without the macro check, such as storing my ends in reverse iterators, and converting them whenever I need to do comparisons, though that is rather painful. I've attached a simple example which demonstrates the problem, not that it fails when the distance from my_end to end and the next element is pushed into the front. -- Anthony Foglia Princeton Consultants (609) 987-8787 x233

4 6

[bind] apply and member function
by Tore Halvorsen 19 Nov '10

19 Nov '10

Hi, Is boost::apply supposed to work with member functions? #include <boost/bind.hpp> #include <boost/bind/apply.hpp> struct foo { void bar() { } }; void quux(foo* fp) { } int main(int argc, char* argv[]) { foo f; foo* fp(&f); // Works as expected boost::bind(boost::apply<void>(), _1, fp)(&quux); // Fails to compile boost::bind(boost::apply<void>(), _1, fp)(&foo::bar); } -- Eld på åren og sol på eng gjer mannen fegen og fjåg. [Jøtul] <demo> 2010 Tore Halvorsen || +052 0553034554

1 0

[interprocess] Fault recovery in managed mapped file
by Kevin Arunski 18 Nov '10

18 Nov '10

I have found that managed mapped file can get stuck in a spinlock if the file is not closed and fully flushed to disk. For example, if the power is pulled from computer while a segment is open and before the first page in the segment has been committed. In this case it is common for a journaled filesystem to preserve the fact that the file was created, but it has lost the contents of the file and now the file appears to be zero'd out. I have observed this behaviour on Linux systems running ext4, for example. This loop is where we get stuck, in managed_open_or_create_impl::priv_open_or_create while(value == InitializingSegment || value == UninitializedSegment){ detail::thread_yield(); value = detail::atomic_read32(patomic_word); } At this point I have opened the file in open_only mode. *patomic_word is 0 (UninitializedSegment) at this point. It appears this code is waiting for some other process or thread to initialize the segment, but in fact there is no such process doing so. Some possible solutions: If, at this point we have opened a file and not created it, why wait for an UnitializedSegment to change state? If the segment is Unitialized here then simply throw an error. Make it the caller's responsibility to ensure the segment is created/initialized before it opened in a read-only mode. Perhaps, if you want to allow multiple processes to do open and create simultaneously without any additional synchronization mechanism, you could accomplish that by adding a count of open mappings into the shared segment. If the reference count is 1 at this point, don't attempt this spinlock because the state of the file is never going to change. In this case throw if the *patomic_word is != InitializedSegment. -- KEVIN ARUNSKI

2 2

[Interprocess] Allocing memory that can be unmapped
by Paul Harris 18 Nov '10

18 Nov '10

Hi, I need to allocate multiple large chunks of memory (many 100MB arrays), and my app is running out of RAM on 32 bit Windows operating systems. Works fine on Linux, as usual. I only need to access maybe 5 of those arrays at a time, so I was hoping to do something like this... create_shared_memory_in_pagefile(); for (each array) { array_offsets[i] = alloc_array_in_shm(); ptr = map_array(array_offsets[i]); do stuff with ptr; unmap_array(ptr); } then later on, when I need access to the array, ptr = map_array(array_offsets[i]); do stuff with ptr; unmap_array(ptr); That would limit the amount of mapped memory, and thus would survive on a 32 bit OS. Note that I don't actually need to share the memory at all, it would be used for only one process. I have been reading up on boost::interprocess, and it looks like it would help me manage the shared memory in a portable way. But what I'm really interested in is a simple allocator that will help me manage the shared memory. But I've noticed that all the allocators and managed memory requires that the entire shared memory chunk is mapped - this is what I'm trying to avoid. Does anyone have any ideas on how I can approach this? I was thinking: it seems that the allocator needs the entire memory mapped as it keeps its datastructures there. Instead, could the allocator store its data structures in RAM, and then I just need to map/unmap the correct addresses when required? thanks Paul

2 1

[proto] : Proto transform with state
by Manjunath Kudlur 18 Nov '10

18 Nov '10

I am trying to create a transform where I replace occurrences of proto::terminal<term> with proto::terminal<newterm<mpl::int_<N> > >. I want N to increment for every proto::terminal<term> encountered. So far I have the following code : #include <boost/proto/proto.hpp> #include <boost/mpl/int.hpp> #include <boost/mpl/arithmetic.hpp> #include <iostream> using namespace std; namespace proto=boost::proto; namespace mpl=boost::mpl; struct term {}; template<typename> struct newterm {}; proto::terminal<term>::type const a = {{}}; proto::terminal<term>::type const b = {{}}; proto::terminal<term>::type const c = {{}}; struct xform_callable : proto::callable { template<typename> struct result; template<typename This, typename N> struct result<This(const term &, N)> { typedef typename proto::result_of::make_expr<proto::tag::terminal, newterm<N> >::type type; }; template<typename This, typename tag, typename LHS, typename RHS> struct result<This(tag, LHS, RHS)> { typedef typename proto::result_of::make_expr<tag, LHS, RHS>::type type; }; template<typename N> typename proto::result_of::make_expr<proto::tag::terminal, newterm<N> >::type operator()(const term &, N) { return proto::make_expr<proto::tag::terminal>(newterm<N>()); } template<typename tag, typename LHS, typename RHS> typename proto::result_of::make_expr<tag, LHS, RHS>::type operator()(tag, LHS l, RHS r) const { return proto::make_expr<tag>(l, r); } }; struct xform : proto::or_< proto::when< proto::terminal<term>, xform_callable(proto::_value, mpl::next<proto::_state>())>, proto::when< proto::plus<xform, xform>, xform_callable(proto::tag_of<proto::_expr>(), xform(proto::_left, proto::_state), xform(proto::_right, proto::_state))> > {}; int main() { mpl::plus<mpl::int_<0>, mpl::int_<1> >(); proto::display_expr(xform()(a+b/c, mpl::int_<0>())); return 0; } I pass in mpl::int_<0>() as the initial state and call mpl::next when I match proto::terminal<term>. As the output would indicate, this is not working. Every time I think I understand proto transforms, something basic like this stumps me. I will be grateful for any hints on how to accomplish this. Thanks, Manjunath

3 3

[serialization] does not compile on PPC or Intel Mac with architecture=combined
by Philipp Münzel 18 Nov '10

18 Nov '10

Hi folks, the build of serialization in boost 1.44 fails when compiling for PPC Mac. On a native Power PC with gcc 4.0.1 the build fails: darwin.compile.c++ bin.v2/libs/serialization/build/darwin-4.0.1/release/link-static/ threading-multi/basic_xml_archive.o darwin.compile.c++ bin.v2/libs/serialization/build/darwin-4.0.1/release/link-static/ threading-multi/binary_iarchive.o ./boost/archive/basic_binary_iarchive.hpp:74: error: invalid application of 'sizeof' to incomplete type 'boost::STATIC_ASSERTION_FAILURE<false>' "g++" -ftemplate-depth-128 -O3 -finline-functions -Wno-inline -Wall -no-cpp-precomp -gdwarf-2 -Wno-long-double -DBOOST_ALL_NO_LIB=1 -DNDEBUG -I"." -c -o "bin.v2/libs/serialization/build/darwin-4.0.1/release/link-static/ threading-multi/binary_iarchive.o" "libs/serialization/src/binary_iarchive.cpp" ...failed darwin.compile.c++ bin.v2/libs/serialization/build/darwin-4.0.1/release/link-static/ threading-multi/binary_iarchive.o... ...failed updating 1 target... ...updated 74 targets... On an Intel Mac, with OSX 10.5 and gcc 4.2.1 the build fails when architecture=combined is added to the bjam command: bin.v2/libs/serialization/build/darwin-4.2.1/release/ architecture-combined/link-static/threading-multi/binary_iarchive.o In file included from ./boost/archive/binary_iarchive_impl.hpp:22, from ./boost/archive/binary_iarchive.hpp:20, from libs/serialization/src/binary_iarchive.cpp:14: ./boost/archive/basic_binary_iarchive.hpp:74: error: invalid application of ‘sizeof’ to incomplete type ‘boost::STATIC_ASSERTION_FAILURE<false>’ lipo: can't figure out the architecture type of: /var/tmp//cc5y68Un.out "g++" -ftemplate-depth-128 -O3 -finline-functions -Wno-inline -Wall -no-cpp-precomp -gdwarf-2 -arch i386 -arch ppc -DBOOST_ALL_NO_LIB=1 -DNDEBUG -I"." -c -o "bin.v2/libs/serialization/build/darwin-4.2.1/release/architecture- combined/link-static/threading-multi/binary_iarchive.o" "libs/serialization/src/binary_iarchive.cpp" ...failed darwin.compile.c++ bin.v2/libs/serialization/build/darwin-4.2.1/release/architecture- combined/link-static/threading-multi/binary_iarchive.o... ...failed updating 1 target... ...updated 96 targets... When using bjam without architecture=combined on an Intel Mac, the build succeeds. That makes boost.serialization unusable on PCC, because it can neither be build native, nor for a universal binary. I wonder how this could slip through the rigid testing procedures of boost... Regards, Philipp

2 1

Re: [Boost-users] Hybrid parallelism, no more + mpi+serialization, many questions
by Hicham Mouline 18 Nov '10

18 Nov '10

-----Original Message-----From: "Brian Budge" [brian.budge(a)gmail.com]Date: 18/11/2010 03:20 PMTo: boost-users(a)lists.boost.orgSubject: Re: [Boost-users] Hybrid parallelism, no more + mpi+serialization, many questions>> The large calculation that I currently do serially and that I intend to>> parallelize is the maximum of the return values of a large number of>> evaluations of a given "function" in the mathematical sense.>> The number of arguments of the function is only known at runtime.>> Let's say it is determined at runtime that the number of arguments is 10, ie>> we have 10 arguments x0, ..., x9>> Each argument can take a different number of values, for e.g. x0 can be>> x0_0, x0_1 .... x0_n0>> x1 can be x1_0, x1_1, ...x1_n1 and so on...n0 and n1 are typically known at>> runtime and different>>>> so serially, I run>> f(x0_0, x1_0, ..., x9_0)>> f(x0_0, x1_0, ..., x9_1)>> ...>> f(x0_0, x1_0, ..., x9_n9)>> then with all the x8 then all the x7 ... then all the x0.>> There is n0*n1*...*n9 runs>> Then I get the maximum of the return values.>>>>>> Imagining I have N mpi processes, ideally each process would run>> n0*n1*...*n9/N function evaluations.>> How do I split?>>>> In terms of current implementation, each of the x is a boost::variant over 4>> types:>> a double, a <min,max> pair, a <min, max, increment> triplet or a>> vector<double>>> A visitor is applied recursively to the variants in order to traverse the>> whole parameter space. >> apply_visitor on x0 => say if x0 is a triplet, then>> for (x0 from min to max with increment)>> apply_visitor on x1>> and so on until x9, then we actually call the f function with all the>> arguments collected so far.>>>> How can one parallelize such a beast?>>>> rds,>>>>>> _______________________________________________>This partly depends on how many processors/machines you have>available. You need to find a way of partitioning your state space>into tasks, and then doling those tasks out to processes/threads. How>expensive is f()? How much memory is used?>Brian f() takes about 1ms. I have a couple of billion n-tuples in the parameter space for now but thinking of reaching easily a couple of trillions.In the serial mode, only about 20Mbytes of RAM is used. The parameter space is not generated ahead. It looks like a tree.The tree is generated, when I reach a leaf, I evaluate f(), and compare the return to the current maximum.So I'm not storing much.rds,

2 1

Re: [Boost-users] Hybrid parallelism, no more + mpi+serialization, many questions
by Hicham Mouline 18 Nov '10

18 Nov '10

hello,Subsequent to a previous thread asking whether to merge MPI and Openmp to parallelize a large problem, I've been advised to go through MPI only as it would be simpler and that MPI implementations on the same box use shared memory which doesn't have a huge cost (still some compared to a uniprocess multithread where objects are actually shared naturally.... writing this, actually a question comes up:1. in the "shared memory" of many mpi processes on the same box, is an object (say a list of numbers) actually shared between the 2 processes address spaces? I guess not unless one explicitly make it so with the "shared memory API" (unix specific?)So, I currently have a serial application with a GUI that runs some calculations.My next step is to use OpenMPI with the help of the Boost.MPI wrapper library in C++ to parallelize those calculations.There is a set of static data objects created once at startup or loaded from files.2. what are the pros/cons of loading the static data objects individually from each separate mpi process vs broadcasting the static data via MPI itself after only the master reads/sets up the static data?3. Is it possible to choose the binary archive instead of the text archive when serializing my user-defined types?Where do I deal with the endianness issue given that I may have Intel/Sparc/PowerPC CPUs?regards,

6 8

compile error on Solaris 10
by Kelvin Cheng 18 Nov '10

18 Nov '10

Hi, My environment is Solaris 10 , g++ 3.3.2 I try to use the header files directly using the definition -DBOOST_REGEX_NO_LIB I got the following errors doing a simple compilation using regex. g++ -fPIC -c -DUNIX -DSOLARIS -Dsun -D_PTHREADS -D_GLIBCXX_HAVE_MBSTATE_T -DBOOST_REGEX_NO_LIB -I./ -I../shared/ -I../shared/boost ndbdriver.cpp -o ndbdriver.o In file included from ../shared/boost/boost/regex/v4/regex_traits.hpp:35, from ../shared/boost/boost/regex/regex_traits.hpp:27, from ../shared/boost/boost/regex/v4/regex.hpp:39, from ../shared/boost/boost/regex.hpp:31, from ndbdriver.cpp:39: ../shared/boost/boost/regex/v4/regex_traits_defaults.hpp:79: error: redefinition of `bool boost::re_detail::is_combining(charT) [with charT = char]' ../shared/boost/boost/regex/v4/regex_traits_defaults.hpp:74: error: `bool boost::re_detail::is_combining(charT) [with charT = char]' previously declared here In file included from ../shared/boost/boost/regex/v4/regex.hpp:58, from ../shared/boost/boost/regex.hpp:31, from ndbdriver.cpp:39: ../shared/boost/boost/regex/v4/states.hpp:60: error: redefinition of `struct boost::re_detail::is_byte<char>' ../shared/boost/boost/regex/v4/states.hpp:58: error: previous definition of ` struct boost::re_detail::is_byte<char>' In file included from ../shared/boost/boost/functional/hash.hpp:6, from ../shared/boost/boost/regex/v4/basic_regex.hpp:23, from ../shared/boost/boost/regex/v4/regex.hpp:67, from ../shared/boost/boost/regex.hpp:31, from ndbdriver.cpp:39: ../shared/boost/boost/functional/hash/hash.hpp: In function `size_t boost::hash_value(char)': ../shared/boost/boost/functional/hash/hash.hpp:152: error: redefinition of ` size_t boost::hash_value(char)' ../shared/boost/boost/functional/hash/hash.hpp:142: error: `size_t boost::hash_value(char)' previously defined here ../shared/boost/boost/functional/hash/hash.hpp:152: error: redefinition of ` size_t boost::hash_value(char)' ../shared/boost/boost/functional/hash/hash.hpp:142: error: `size_t boost::hash_value(char)' previously defined here ../shared/boost/boost/functional/hash/hash.hpp: In member function `size_t boost::hash<char>::operator()(char) const': ../shared/boost/boost/functional/hash/hash.hpp:405: error: call of overloaded ` hash_value(char&)' is ambiguous ../shared/boost/boost/functional/hash/hash.hpp:137: error: candidates are: size_t boost::hash_value(bool) ../shared/boost/boost/functional/hash/hash.hpp:142: error: size_t boost::hash_value(char) ../shared/boost/boost/functional/hash/hash.hpp:147: error: size_t boost::hash_value(unsigned char) ../shared/boost/boost/functional/hash/hash.hpp:157: error: size_t boost::hash_value(short int) ../shared/boost/boost/functional/hash/hash.hpp:162: error: size_t boost::hash_value(short unsigned int) ../shared/boost/boost/functional/hash/hash.hpp:167: error: size_t boost::hash_value(int) ../shared/boost/boost/functional/hash/hash.hpp:172: error: size_t boost::hash_value(unsigned int) ../shared/boost/boost/functional/hash/hash.hpp:177: error: size_t boost::hash_value(long int) ../shared/boost/boost/functional/hash/hash.hpp:182: error: size_t boost::hash_value(long unsigned int) ../shared/boost/boost/functional/hash/hash.hpp:188: error: size_t boost::hash_value(wchar_t) ../shared/boost/boost/functional/hash/hash.hpp:195: error: size_t boost::hash_value(long long int) ../shared/boost/boost/functional/hash/hash.hpp:200: error: size_t boost::hash_value(long long unsigned int) ../shared/boost/boost/functional/hash/hash.hpp:314: error: size_t boost::hash_value(float) ../shared/boost/boost/functional/hash/hash.hpp:319: error: size_t boost::hash_value(double) ../shared/boost/boost/functional/hash/hash.hpp:324: error: size_t boost::hash_value(long double) ../shared/boost/boost/functional/hash/hash.hpp:152: error: size_t boost::hash_value(char) ../shared/boost/boost/functional/hash/hash.hpp: At global scope: ../shared/boost/boost/functional/hash/hash.hpp:406: error: redefinition of ` struct boost::hash<char>' ../shared/boost/boost/functional/hash/hash.hpp:405: error: previous definition of `struct boost::hash<char>' All the errors are from related to the basic types when expanded from the templates. Any idea what's going wrong ? -- Kelvin KF

2 2

Interest in B-tree library for Boost?
by Beman Dawes 17 Nov '10

17 Nov '10

A Boost B-tree library would provide disk-based associative containers that scale all the way from really, really, small to really, really, large. B-trees perform well on hardware ranging from ancient floppy disk drives all the way up to humongous disk arrays. They are the technology behind most high-performance disk file systems and databases. If you don't know what a B-tree is, or think that "B-tree" is an abbreviation for "Binary tree", you might want to read the Wikipedia B-Tree article: http://en.wikipedia.org/wiki/B-tree. Knuth and other computer science texts also supply descriptions. See http://mysite.verizon.net/beman/btree/index.html for a more complete description of the proposed library. A preliminary implementation is available. Download http://mysite.verizon.net/beman/btree/btree-preview-1.zip, and unpack into a current Boost distribution. A few existing Jamfiles will be overwritten. The only compilers tested and working so far are VC++ 10 and GCC/MinGW 4.5. You can view the library online at http://github.com/Beman/Boost-Btree There is no detailed documentation yet, but if you know how to use std::map and other standard library associative containers, you already know most of what you need to know. Critical differences are documented in the Library Characteristics section of the index.html page referenced above. Initial timing tests have been very encouraging. The time test compares the btree_map implementation to std::map. As expected, a btree_map is many times slower than a std::map, given a cold operating system file cache and constraining the btree_map's caching to a minimum. But at the other extreme, when the O/S file cache is warm and the btree_map's cache is as large as the file, the btree_map is very similar in speed to std::map.The btree library's code is still very new, so timings and everything else may be subject to lots of revision. Any interest? --Beman

5 11