July 2014 - Boost - lists.preview.boost.org

local_time seems not monotonic
by Gaetano Mendola 31 Jul '14

31 Jul '14

The following snippet seems to generate non monotonic local_date. I'm using boost 1.55 on linux. #include <boost/date_time/local_time/local_time.hpp> #include <boost/ptr_container/ptr_vector.hpp> int main() { const boost::local_time::time_zone_ptr theTimeZone( new boost::local_time::posix_time_zone( "CET+01CEST+01,M3.5.0/02:00,M10.5.0/03:00") ); boost::local_time::local_date_time myOldValue( boost::local_time::local_microsec_clock::local_time(theTimeZone)); for (size_t i = 0; ; ++i) { const boost::local_time::local_date_time myLocalTime = boost::local_time::local_microsec_clock::local_time(theTimeZone); if (myLocalTime < myOldValue) { std::cout << myOldValue << std::endl; std::cout << myLocalTime << std::endl; std::cout << boost::local_time::local_microsec_clock::local_time(theTimeZone) << std::endl; std::cout << "====================" << std::endl; } myOldValue = myLocalTime; } } As you can see the program is not supposed to print nothing ever, however this is what I'm getting: 2014-Jul-31 00:24:56.005625 CEST 2014-Jul-31 00:24:55.005631 CEST 2014-Jul-31 00:24:56.005946 CEST ==================== 2014-Jul-31 00:24:58.005625 CEST 2014-Jul-31 00:24:57.005629 CEST 2014-Jul-31 00:24:58.005824 CEST ==================== 2014-Jul-31 00:25:02.005624 CEST 2014-Jul-31 00:25:01.005628 CEST 2014-Jul-31 00:25:02.005838 CEST ==================== 2014-Jul-31 00:25:04.005625 CEST 2014-Jul-31 00:25:03.005630 CEST 2014-Jul-31 00:25:04.005826 CEST ==================== 2014-Jul-31 00:25:06.005624 CEST 2014-Jul-31 00:25:05.005633 CEST 2014-Jul-31 00:25:06.005853 CEST ==================== 2014-Jul-31 00:25:07.005625 CEST 2014-Jul-31 00:25:06.005631 CEST 2014-Jul-31 00:25:07.005846 CEST ==================== 2014-Jul-31 00:25:12.005625 CEST 2014-Jul-31 00:25:11.005631 CEST 2014-Jul-31 00:25:12.005822 CEST ==================== as you can see when the local_date is near 0.005631 second fraction it goes back of one second and then forward again on the following call. Am I missing something ? Regards Gaetano Mendola

3 2

Request for feedback on design of concurrent_unordered_map, plus notes on use of memory transactions
by Niall Douglas 31 Jul '14

31 Jul '14

Dear Boost, Finally I have found some time to write up my experimentations with using transactional GCC or Intel TSX TM to make an improved concurrent_unordered_map. The quick tl;dr summary is this: 1. Transactional GCC is great iff you are happy accepting up to a 50% performance reduction in normal code execution. For a greenfield multithreaded project this is absolutely worth it due to the enormous savings in debug effort. 2. First generation Intel TSX isn't worth using in almost any common use case. The only common use place it makes sense is for almost always read only data which is *always* highly contended where speedups can approach linear to execution cores, and even then the refactoring you need to make to avoid writes to any cache line touched by any other thread will make your code slower than throwing everything inside a standard spin lock. Spin locks are well optimised and allow simple implementations, plus spin locks are very fast for uncontended use. Any Intel TSX (HLE, RTM) introduces penalties to uncontended use and seems to force a pipeline stall which can reduce single threaded performance by as much as 40%. 3. Trying to implement atomics-based logic to have the CPU select non-TSX during uncontended and TSX during contended comes out at best as slightly slower than always using TSX. This, for me, simply kills TSX v1.0, I no longer consider it worth thinking about till v2.0 is made by Intel. For reference for those wishing to repeat my experimentations, you can see the history of my four attempts at a concurrent_unordered_map at https://github.com/ned14/boost.spinlock/commits/concurrent_unordered_m ap2 plus benchmark results at https://ci.nedprod.com/view/Boost%20Thread-Expected-Permit/job/Memory% 20Transactions%20Test%20Linux%20GCC%204.8/. So, onto my proposed new concurrent_unordered_map design, which is NOT a split ordered list design as Intel TBB or Microsoft PPL does it. I had four criteria for my concurrent_unordered_map design which were based on my needs for proposed Boost.AFIO: 1. erase() is safe. This is for me the big showstopper with split ordered list designs, and I think it probably is for most people. 2. insert(), find(), and erase() all had to have single threaded performance similar to a std::unordered_map<> with a spin lock around it i.e. fast. 3. Everything apart from rehashing should be able to be concurrent. This includes iteration, inserting, finding, erasing, sizing. 4. Memory transaction friendly. This means transactional GCC right now, but maybe some future Intel TSX v2.0 support later. This criterion, incidentally, *requires* C++ 11 noexcept support, or at least I can't see a way to avoid it otherwise (just to be clear, I am claiming that for the implementation to _efficiently_ use memory transactions it needs noexcept move construction for both key and mapped type). Designs tried and benchmarked and rejected: 1. Items in a bucket were kept via an atomic forward pointer to the next item and one used cmpxchg to insert and remove items. This had really great, even superb, performance for load factors near one, but became pathological for badly distributed hash functions (I'm looking at you libstdc++). Once you rehashed the hash function to make it better distributed, performance nose dived. I was tempted to override std::hash with a decent implementation ... 2. Items in a bucket were kept as as a linear list of hashes and atomic pointers to value_type's. Each atomic pointer's bottom bit was its spin lock to keep each item footprint small, so you could lock each individual item during usage. The problem here really was that how do you safely resize the table without a per-bucket lock, and that halved performance over a spinlocked unordered_map as transactions really didn't deliver as expected here. 3. I created a specialisation only for mapped types of some std::shared_ptr<> which let me implement a completely lock free design. Unfortunately it was also rather slow, also about half the performance of a spinlocked unordered_map. Atomic increments and decrements of the same cache line across execution cores are slow :) Final proposed design (comments welcome): * item_type is allocated via a rebind of the supplied allocator, and looks like: struct item_type { value_type p; size_t hash; }; If value_type has a big sizeof(), a very good idea is to make the above a unique_ptr as otherwise you can't pack many item_type's into a cache line (AFIO's value_type is a std::pair<size_t, std::shared_ptr<>> which is 24 bytes, so item_type is 32 bytes which works well for my purposes). * bucket_type looks like this: struct bucket_type { spinlock<bool> lock; atomic<unsigned> count; // count is used items in there std::vector<item_type, item_type_allocator_type> items; char pad[64-sizeof(spinlock<bool>)-sizeof(atomic<unsigned>)-sizeof(std::vec tor<item_type, item_type_allocator_type>)]; }; static_assert(sizeof(bucket_type)==64, "bucket_type is not 64 bytes long!"); Note that items has its capacity directly managed, and is always expanded to powers of two. The size of actual items stored can float between zero and capacity. * And finally, concurrent_unordered_map itself merely contains: template<class Key, class T, class Hash=std::hash<Key>, class Pred=std::equal_to<Key>, class Alloc=std::allocator<std::pair<const Key, T>>> class concurrent_unordered_map { spinlock<bool> _rehash_lock; hasher _hasher; key_equal _key_equal; std::vector<bucket_type> _buckets; }; * To insert or find an item involves determining the bucket from the modulus of the hash by the number of buckets, locking the bucket and scanning the linear list for an empty item or the right item as appropriate. We know empty items from a magic hash value which is defined to some magic constant (advice on the best magic hash value least likely to turn up from std::hash is welcome). find() starts from the start of the list forwards, insert() starts from the end of the list backwards to help with cache line contention. As insert() needs to check for key matches as well as for empty slots, it needs to happen anyway, but if we knew key collision could never happen we could detect when the table is full from the atomic count and skip the table scan. If insert doesn't find a match nor an empty slot, if there is capacity it appends the item, if no capacity a capacity doubling is done and then an append. If capacity doubling occurs, all references to items in that bucket become invalid. I intend an insert_stable() API which will refuse to modify capacity, plus a bucket_reserve() API. insert() and find() have O(avrg bucket size (not count)). On transactional GCC find() and the initial seach for key equivalence in insert() are implemented as transactions without locking the bucket. All modifications are done by locking the bucket. This was found to have the best performance through trial and error. * To erase an item is easy: iterators store an iterator to the bucket and an offset into the linear table. Erasure involves constructing a temporary value_type, locking the bucket and performing a *move* construction out of the table into the temporary value_type, resetting the hash to empty. Destruction can then occur without holding the bucket lock. If the item just erased is the last in the bucket, the bucket's size (not capacity) is reduced by one, and that is repeated until the bucket size is the minimal possible. I currently violate STL allocator requirements here because I don't use the allocator for the temporary value_type to avoid a usually pointless malloc. Any suggestions on how to fix this would be welcome. You'll note that erasure here is safe as its storage never moves, though if you concurrently use and erase the same item you'll get a race onto a destroyed item which is as expected. erase() is O(1). * size() under this design becomes O(bucket count) as we sum the count members of each bucket. empty() is worst case O(bucket count) when the map is empty, best case O(1) when the first bucket is not empty, average case O(bucket count/item count/2). * Iteration under this design is queerly inversely dependent on load factor - with the higher the load factor, the better the performance. This is because we don't bother with the usual forward pointer per item as those are expensive for concurrent use and doubly linked pointers are rather a lot of overhead per item, so iteration is a case of iterating linear tables and once completed, searching for the next non-empty bucket which in a sparsely occupied bucket list is very lumpy as cache lines are pulled one by one into the cache. * rehash() under this design firstly locks the special rehash lock, then locks every bucket in the list. It then constructs a new empty bucket list and locks every lock in that. It then swaps the bucket lists, and moves all the items from the old buckets to the new buckets. It finally unlocks all the buckets, and the special rehash lock. This function is hard to write completely race free under all CPU load circumstances without burdening the main API implementations. I may use the concept of keeping around the old just emptied bucket list which is permanently marked to users as "go reload the bucket list" to work around this, and then document that you must not rehash too frequently or else you will race your code. Note that rehashing under this design is completely manual. If it were not, insertion, erasure and item referencing could not be safe. This implies that if you rehash, it is up to you to not be using items by reference at the time. Rehashing is also by definition not concurrent, indeed it is the only function in the design which is not capable of concurrency. Thoughts on this design before I write a final implementation are welcome. I don't plan to complete my implementation past the minimum needed for proposed Boost.AFIO, so if someone else wishes to finish it that and make it part of official Boost that would be great. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

4 6

Fwd: [sync?] "Distributed" value (or something...)
by Klaim - Joël Lamotte 31 Jul '14

31 Jul '14

---------- Forwarded message ---------- From: Klaim - Joël Lamotte <mjklaim(a)gmail.com> Date: Tue, Jul 29, 2014 at 9:44 PM Subject: [boost][sync?] "Distributed" value (or something...) To: Boost users list <boost-users(a)lists.boost.org> I would like to ask Boost developers and users: 1. Am I the only one seeing the following pattern in my concurrent systems code? 2. Is there already some tools available in Boost that does what I describe here and that I am not aware of? 3. If not, would there be any interest in the tool I'm describing here or not? Is it a dumb idea? (I keep getting back to this idea but I'm surprised not finding anything similar online) The pattern that I am seeing occurs when: 1. I have a value or a set of values in an object or several objects which value(s) describe some kind of configuration that several other systems will read and use. 2. The object(s) will be updated only by one thread (at a time). 3. When the object's value is modified, all the systems that rely on the object's value have to be notified so that they update themselves using that value. 4. The systems I am refering to can use different threads for updating and we can assume that they are running concurrently. 5. It is not important that the source object's value is synched with the reader's values at all time, however it is important that the sequence of changes to the source object's value is signaled to the readers exactly in the same order and "as fast as possible", but delay is ok. The simplest solution to solve this is to have a mutex locking the value and the systems checking the value regularly. (that is, using syncrhonized_value for example) However the locking and looping reads might not be necessary if it is done in another way (if I am correct, I still lack experience in the domain). The solution that I am thinking about would look like this. ///////// // The object containing the value used by the other systems. class System_A { distributed_source<int> m_foo_count; // changes to this object will be sent to "readers"... public: System_A() : m_foo_count(42) ... {} //... // thread-safe read-only access distributed_value<int> make_foo_count_reader() const { return distributed_value<int>{ m_foo_count }; } }; ///////// // The different systems acquire a read access to the value. class System_B { distributed_value<int> foo_count; // read-only value reflecting System_A::m_foo_count's value since last update. public: SystemB( const System_A& system_a, ... ) : m_foo_count( system_a.make_foo_count_reader() ) { // setup the initial config... on_foo_count_changed( *m_foo_count ); // trigger some processing when SystemA::foo_count's value have been changed: m_foo_count.on_changed( [this]( int new_value ){ // called after the value have been changed m_work_queue.push( [=]{ // will be executed on the next update cycle for this system... on_foo_count_changed( new_value ); / }); }); } //... private: void update_cycle( const UpdateInfo& info ) { // ... // the following is just to clarify: const int foo_before = *m_foo_count; // the value is not updated yet, but we can still use it m_work_queue.execute(); const int foo_after = *m_foo_count; // the value might have been udpated if( foo_before != foo_after ) { // the value have been updated (though you don't need to do this if you used the callback instead). } //... }; }; //////// // At some point the value is changed by the system owning the source object... System_A::change_foo( int new_value ) { m_work_queue.push( [=]{ // will be executed on the next update cycle of this system... // ... *m_foo_count = new_value; // assign the new value then send the new value to the readers... }); } System_A::something_complex( std::function<void(int&)> some_work ) { m_work_queue.push( [=]{ // ... m_foo_count.update( some_work ); // execute on the current value then send the resulting value to the readers }); } Some notes: a. The names are just invented while I write this email. b. I suppose that the distributed_source might have an interface similar to synchronized_value, anyway I didn't get farther than this idea. c. I see several different ways to implement this idea, some might be more efficient than others but I didn't do any tests yet (by lack of time). Thoughts?

3 7

Documentation appears to have part of boost::mpi::packed_iarchive and boost::mpi::packed_oarchive reversed?
by Dominic Marcello 31 Jul '14

31 Jul '14

Hello It seems like the documentation for boost::mpi::packed_iarchive and boost::mpi::packed_oarchive are reversed. boost::mpi_packed_iarchive uses MPI_Unpack and boost::mpi::packed_oarchive uses MPI_Pack. This can be verified by looking at the packed_iprimitive and packed_oprimitive base classes. The documentation, as well as the packed_iarchive and packed_oarchive header files, however, say the opposite - that packed_oarchive uses MPI_Unpack and packed_iarchive uses MPI_Pack. -Dominic

2 1

Clang on Windows
by Matthew Musto 31 Jul '14

31 Jul '14

I attempted to run Clang 3.4.1 on Win 8.1 with horrible results as evident on the current master regression summary here: http://www.boost.org/development/tests/master/developer/summary.html Besides the fact that boost identifies it as linux clang and there appears to be numerous other issues with the b2 script, has anyone had any success with this? The reason I ask is AMD is using LLVM for their heterogeneous development and I'm trying to see if boost will be something I can count on. Thanks, -Matt

1 0

b2 headers is unreliable on 1.56 on Windows
by Niall Douglas 30 Jul '14

30 Jul '14

Anyone else found that b2 headers in 1.56 randomly misses things to hard link on Windows? If you run it four or five times it eventually hard links everything through monte-carlo, but it is a bit annoying. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

2 2

Lock free condition_variable or using shared_lock
by Kirk Liberty 30 Jul '14

30 Jul '14

I'm not too familiar with the mechanisms which underlie mutexs and conditionals, which is why I'm asking this question. Is it possible to modify condition_variable so that wait() can be called on shared_lock? The use case would be it is acceptable to release all threads waiting on a condition_variable when notify_all() is called. Using a unique_lock, every thread who wakes up will then have to obtain the lock before proceeding; this being a potential bottleneck. An even more performant alternative to using shared_lock would be with a dummy lock never blocks when dummy_lock.lock() is called. These are things I would like to contribute, but I need would need some guidance and a review of feasibility. Thanks for the help, Kirk

3 2

News about proposed Boost.Application library
by Renato Forti 30 Jul '14

30 Jul '14

Hi All, This e-mail is only to inform news about proposed Boost.Application library. We have done many updates on version 0.4.9, see: - MinGW support; - The ‘auto_handler’ class to bind ‘stop’, ‘pause’, ‘continue’ and ‘single instance’ handlers automatically; - New samples for ‘auto_handler’; - Many fix and improvements on ‘Windows Server’ (Service API) mode; - Simplification of ‘handlers’ creation, now application context can by passed on constructor of functor, or user can use global version of it; - Docs are updated to version 0.4.9; - And many others small fixes. To know more, please visit: https://github.com/retf/Boost.Application Article (0.4.8) http://www.codeproject.com/Articles/756866/Build-a-Server-Application-using… (Prize winner in Competition "Best C++ Article of April 2014" ) Thanks

7 20

[trac] Adding new components (Align, Type_Index)
by Michel Morin 30 Jul '14

30 Jul '14

Wouldn't it be better to add newly introduced libraries Align: glenfe (Glen Fernandes) Type_Index: apolukhin (Antony Polukhin) to trac components? Regards, Michel

1 0

[quickbook] disambiguate overloaded function
by Benedek Thaler 30 Jul '14

30 Jul '14

Hi, I'm using quickbook and doxygen to generate my documentation. Suppose I have the following functions: func(int); func(std::string); They have separate reference pages, I'd like to link to them. However, I can't specify which one I mean using quickbooks funcref construct: [funcref func text] // results in: Reference to function 'func' is ambiguous. How can I specify the overload? Thanks, Benedek

1 0