February 2013 - Boost - lists.preview.boost.org

[thread] thread_specific_ptr leaks on VS2012 when using std::async()
by Klaim - Joël Lamotte 05 Mar '13

05 Mar '13

Hi, The title says it all. I think I found a memory leak with the help of boost.log author Andrei Semashev. I was using Boost.Log in Visual Studio 2012 (released one) and tried to use Visual Leak Detector because Visual Studio basic leak detector was reporting leaks. I isolated the leaks to usage of log and switched to VLD to check if the basic detector wasn't giving me false positive. I contacted Andrei to see if he could check it but he don't have VS2012 available so I setup some tests to isolate the problem. In the end of our exchange we concluded it was a false positive (it only leak on end of program). However it is still an annoying noise in the leak report logs, so people not understanding the source of this problem might be interested in this email. I attach the whole email exchange to this email, tell me if leak logs are necessary too. The last version of my test is at the end of the email (first program). Basically, the leak appear only when using std::async() in combination of boost::thread_specific_ptr : - using exclusively boost threading libraries (both with dynamic and static linking) result in no leak - using exclusively std threading libraries but not std::async() result in no leak - using std::async() result in a leak The second test program shows that using Intel Threading Building Blocks, I get leaks only if : - I use std::async() in combination with boost::thread_specific_ptr, as before (I mean I use tbb without a leak, then I add a std::async() call to the same task function and there is a leak) - I don't use std::async() but I don't manage the lifetime of tbb task scheduler We think the threads spawned by both tbb (that is supposed to be a superset of ppl) and std::async() (that is supposed to use ppl/concrt task scheduler) are not ended yet when the program reach the end of main (which seems logical as the task scheduler is then managed as a static global), making boost::thread_specific_ptr not release some data on thread destruction, as it is notified too late and any leak detection tool willl generate noise due to this. A way to fix this is to explicitely end the task scheduler, being tbb or ppl/concrt. I will use tbb task scheduler explicitely initialized/terminated in my project so it will solve the problem on my side. Joel Lamotte ------------------------------------ // this program was used to see the memory leak #define TEST_ASYNC // comment this to make it linear and not leak //#define WITH_BOOST_THREADS #define WITH_STD_ASYNC // uncomment this to leak //#define WITH_BOOST_LOG // boost log also leak if std::async is used #include <boost/thread.hpp> #ifdef WITH_BOOST_THREADS # include <boost/thread/future.hpp> # include <boost/chrono.hpp> # define THIS_THREAD boost::this_thread # define THREAD_LIB boost::thread # define CHRONO_LIB boost::chrono # define MY_FUTURE boost::future #else # include <future> # include <thread> # include <chrono> # define THIS_THREAD std::this_thread # define THREAD_LIB std::thread # define CHRONO_LIB std::chrono # define MY_FUTURE std::future #endif #ifdef WITH_BOOST_LOG # include <boost/log/trivial.hpp> # define MY_LOG BOOST_LOG_TRIVIAL(info) #else # include <iostream> # define MY_LOG std::cout << '\n' #endif #define BASIC_VS_LEAK_DETECTOR 0 #define VISUAL_LEAK_DETECTOR 1 #define LEAK_DETECTOR VISUAL_LEAK_DETECTOR #if LEAK_DETECTOR == BASIC_VS_LEAK_DETECTOR # define _CRTDBG_MAP_ALLOC # include <stdlib.h> # include <crtdbg.h> #else # include "vld.h" #endif struct my_data { char data[1024]; my_data() { MY_LOG << "CONSTRUCTOR my_data - " << THIS_THREAD::get_id(); } ~my_data() { MY_LOG << "DESTRUCTOR my_data - " << THIS_THREAD::get_id(); } }; boost::thread_specific_ptr< my_data > ts_ptr; void my_thread_foo() { ts_ptr.reset(new my_data()); } int work() { for( int i = 0; i < 42; ++i ) { MY_LOG << "THIS IS A MESSAGE - I = " << i; MY_LOG << "ts_ptr = " << ts_ptr.get(); my_thread_foo(); THIS_THREAD::sleep_for( CHRONO_LIB::milliseconds(10) ); } return 0; } int main(int argc, char *argv[]) { #if LEAK_DETECTOR == BASIC_VS_LEAK_DETECTOR // First simple memory leak detection for Visual Studio // use _crtBreakAlloc to put a breakpoint on the provided memory leak id allocation //_crtBreakAlloc = 1149; _CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF ); #endif //int* k = new int[42]; // to check that memory leak detection does work MY_LOG << "Hello, World!"; #ifdef TEST_ASYNC # ifndef WITH_BOOST_THREADS std::vector< std::future<int> > work_in_progress(2); # ifdef WITH_STD_ASYNC for( auto& ftr : work_in_progress ) { ftr = std::async( work ); } # else for( auto& ftr : work_in_progress ) { std::packaged_task< int() > task(work); ftr = task.get_future(); std::thread thread( std::move(task) ); thread.detach(); } # endif for( auto& ftr : work_in_progress ) { ftr.get(); } # else std::vector< boost::unique_future<int> > work_in_progress(42); for( auto& ftr : work_in_progress ) { boost::packaged_task<int> task(work); // boost's packaged_task don't take the full signature as parameter, it takes the return type only ftr = task.get_future(); boost::thread thread( boost::move(task) ); thread.detach(); } for( auto& ftr : work_in_progress ) { ftr.get(); } # endif #else for( int i = 0; i < 42; ++i ) { work(); } #endif MY_LOG << "END"; return EXIT_SUCCESS; } -------------------------------------- // this program tests if the memory leaks appear with Threading Building Blocks #define BASIC_VS_LEAK_DETECTOR 0 #define VISUAL_LEAK_DETECTOR 1 #define LEAK_DETECTOR VISUAL_LEAK_DETECTOR #define WITH_ROOT_TASK //#define WITH_BOOST_LOG //#define WITH_STD_ASYNC // uncomment to leak #ifdef WITH_STD_ASYNC # include <future> #endif #include <thread> #include <atomic> #include <boost/thread.hpp> #include <tbb/tbb.h> #include <tbb/task_scheduler_init.h> #ifdef WITH_BOOST_LOG # include <boost/log/trivial.hpp> # define MY_LOG BOOST_LOG_TRIVIAL(info) #else # include <iostream> # define MY_LOG std::cout << '\n' #endif #if LEAK_DETECTOR == BASIC_VS_LEAK_DETECTOR # define _CRTDBG_MAP_ALLOC # include <stdlib.h> # include <crtdbg.h> #else # include "vld.h" #endif namespace test { struct my_data { char data[1024]; }; boost::thread_specific_ptr< my_data > ts_ptr; void my_thread_foo() { ts_ptr.reset(new my_data()); } class UpdateTask : public tbb::task { public: UpdateTask() : idx( s_idx++ ) { MY_LOG << "[UpdateTask " << idx << " ] " << std::endl; } ~UpdateTask() { MY_LOG << "[/UpdateTask " << idx << " ]" << std::endl; } tbb::task* execute() { MY_LOG << "[" << std::this_thread::get_id() << "] Tick "<< m_count <<std::endl; my_thread_foo(); ++m_count; if( m_count < 100 ) { // make sure this is re-executed this->increment_ref_count(); this->recycle_as_safe_continuation(); } return nullptr; } private: static std::atomic<int> s_idx; std::atomic<int> m_count; const int idx; }; std::atomic<int> UpdateTask::s_idx = 0; class TaskScheduler // using TBB { public: #ifdef WITH_ROOT_TASK TaskScheduler() : m_root_task( *new (tbb::task::allocate_root()) tbb::empty_task ) { m_root_task.increment_ref_count(); } ~TaskScheduler() { m_root_task.destroy(m_root_task); } #endif void register_update_task() { #ifdef WITH_ROOT_TASK m_root_task.increment_ref_count(); UpdateTask& updater = *new(m_root_task.allocate_child()) UpdateTask(); #else UpdateTask& updater = *new(tbb::task::allocate_root()) UpdateTask(); #endif m_task_list.push_back( updater ); } void run_and_wait() { #ifdef WITH_ROOT_TASK m_root_task.spawn( m_task_list ); m_root_task.wait_for_all(); #else tbb::task::spawn_root_and_wait( m_task_list ); #endif } private: tbb::task_list m_task_list; // uncomment the following line to not leak tbb //tbb::task_scheduler_init m_engine_instance; // We explicitly create and destroy the TBB task scheduler to avoid memory leak warning noises. #ifdef WITH_ROOT_TASK tbb::task& m_root_task; #endif }; void tbb_test() { TaskScheduler task_scheduler; for( int i = 0; i < 50; ++i ) { task_scheduler.register_update_task(); } task_scheduler.run_and_wait(); } } int main(int argc, char *argv[]) { #if LEAK_DETECTOR == BASIC_VS_LEAK_DETECTOR // First simple memory leak detection for Visual Studio // use _crtBreakAlloc to put a breakpoint on the provided memory leak id allocation //_crtBreakAlloc = 148; _CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF ); #endif #ifdef WITH_STD_ASYNC std::atomic<bool> finished = false; std::async( [&]{ while( !finished ) { MY_LOG << "POP"; test::my_thread_foo(); std::this_thread::sleep_for( std::chrono::milliseconds(500) ); } } ); #endif test::tbb_test(); #ifdef WITH_STD_ASYNC finished = true; #endif MY_LOG << "\nEND"; std::this_thread::sleep_for( std::chrono::seconds(4) ); return 0; }

3 5

[thread] C++11 once_flag enabled when constexpr is not available
by Andrey Semashev 05 Mar '13

05 Mar '13

Hi, Vicente! I see you applied my patch for call_once but you seem to forgot to disable the C++11 call_once interface when the compiler does not support constexpr. Was that intentional or you just missed it? Anyway, please find the patch attached.

3 3

[thread] boost::thread and thread safety annotations
by Wladimir 04 Mar '13

04 Mar '13

Hello, The Clang C++ compiler project is currently working on a thread safety analyzer. By making use of so-called thread safety annotations within the source code, several problems with synchronization primitives can be found statically. For example: usage of a data structure without acquiring its guarding lock. See here for an overview of the available attributes: http://clang.llvm.org/docs/LanguageExtensions.html#threadsafety It is customary to use these attributes by defining macros, for compatibility: http://www.mail-archive.com/linuxkernelnewbies@googlegroups.com/msg01455.ht… At one time, it was also supposed to end up in G++, but this was abandoned in favor of clang (see http://gcc.gnu.org/wiki/ThreadSafetyAnnotation) But at least the annotations are fairly standardized. I think it would be useful to add these annotations to boost::thread, so that projects using that library can make use of the static checking. As I understand it, is only a matter of marking data structures that serve as lock/scoped_lock (such as boost::mutex, boost::recursive_mutex, ...), and methods that act upon the lock with the appropriate attributes. Any interest in this? Regards, Wladimir

2 1

[thread] barrier::wait() not marked as interruption_point
by Oliver Kowalke 04 Mar '13

04 Mar '13

In the documentation (1.53 beta) barrier::wait() is not marked as interruption-point nor that it will throw thread_interrupted if thread blocked inside barrier::wait() if it gets interrupted. Because barrier::wait() calls condition_variable::wait() and condition_variable::wait() is a interruption-point barrier::wait() should it be too - or do I miss something? Oliver

2 1

[persistent] Library preview: Persistent
by Stefan Strasser 04 Mar '13

04 Mar '13

Hi everyone, I'm thinking about finishing a library that was proposed a couple of years ago. It was pretty far along, and there is documentation available in the sandbox (somewhat outdated): https://svn.boost.org/svn/boost/sandbox/persistent/libs/persistent/doc/html… It is a library with the goal of storing C++ objects on disk, with an interface that is as close as possible to accessing regular C++ objects in memory. A very simple example could look like this: shared_loc<person> p=...; { transaction tx; p->name="..."; p->address="..."; tx.commit(); } It is based on Boost.Serialization, but automatically maintains a consistent state on disk (ACID transactions), which can be updated incrementally. Part of its code was moved to a library named Transact to work alongside other transactional resources (like memory) and is available in the sandbox: https://svn.boost.org/svn/boost/sandbox/transaction/boost/transact/ I'd like to know whether the boost community is still interested in such a library, and I suppose there are quite a few new people here so your comments are welcome as well. Thank You!

6 9

GSOC 2013
by David Bellot 04 Mar '13

04 Mar '13

Only 2 projects have been proposed so far for GSOC 2013: wake up Boost ! https://svn.boost.org/trac/boost/wiki/SoC2013 :-D

7 7

Review managers wanted for augmented data structures
by Vadim Stadnik 03 Mar '13

03 Mar '13

Hi all, At present, neither Boost library nor STL provides containers using augmented data structures. I suggest that the Boost community consider solving the challenging problem – how to extend STL by integrating advanced data structures. Is there an alternative of not integrating advanced data structures into STL? IMO, no, since it is equivalent to “STL must go”. C++ cannot ignore performance benefits provided by these data structures. On the other hand, advanced data structures can be implemented in many programming languages and this will make C++ less competitive. Augmented red-black trees could have been included in the very first version of STL. Why is it difficult and has not been done yet? The major obstacle for the integration is the C++ standard specification of computational complexities of single operations for containers and iterators. Augmented trees are non-standard due to different performance guarantees. At the same these data structures are beneficial for complex algorithms, since they offer a wider set of more efficient operations compared to standard containers. The typical performance improvement for one operation is O(log N) against O(N). For less efficient operations these trees normally provide running time O(log N) against O(1), which is good enough for many applications. The C++ STL will be under pressure as long as there are new data structures that offer better running time for user algorithms. Asymptotically, the main parameter that determines the running time of an algorithm is the total computational cost (to simplify the discussion we omit the effect of locality of reference). By definition, the total computational cost is the sum of costs of single operations multiplied by their frequencies. For a general purpose library, such as STL, it is impossible at the design stage to know frequencies of all container and iterator operations required in all possible user algorithms. A more reasonable and practical approach to STL would be to provide a wide set of interchangeable containers and data structures with different performance guarantees and allow programmers to make decisions about the most suitable containers for specific applications. Relatively small frequencies of inefficient operations do not have significant effect on the total computational cost of an algorithm. Thus, in principle, containers and iterators can provide even inefficient operation in order to support more unified interfaces and to improve the generality. The replacement of interchangeable containers offers the advantage of low development cost solutions to performance bottlenecks that arise due to enhancement requests and/or an increase in the number of elements processed by a user algorithm. Also, I’d like to note that the representation of sequences using augmented trees is a frequent point of confusion and probably the fact that was overlooked in the development of previous versions of STL. The general description of the method of augmenting data structures is given in the section 14.2 of this book T.H. Cormen, C.E. Leiserson, R.L. Rivest and C. Stein, Introduction to Algorithms, 2nd Edition, MIT Press and McGraw-Hill, 2001. The specific application of this method is provided for order statistic trees and interval trees. Unfortunately, there is no example of a sequence that stores unordered elements. Probably for this reason augmented red-black trees are often considered to be an improvement of search trees only. The efficient representation of sequences by the described red-black trees follows from the fact that augmenting by the counter of children nodes supports efficient mapping from non-negative integers into container elements. It works both for ordered and unordered containers with unique and multiple elements or keys. There are two libraries in the Boost review schedule: http://www.boost.org/community/review_schedule.html "STL Extensions" and "Countertree". None of them has as yet a review manager. The order of the reviews may not be significant. It is more important that at least one library pass the review. This would help integrating other advanced data structures. Are there Boost experts who could volunteer to take responsibility of review manager for any of these two libraries? Regards, Vadim Stadnik

2 4

[type traits] is_literal_type ?
by Robert Ramey 02 Mar '13

02 Mar '13

Any chance of a boost implementation for is_literal_type? Robert Ramey

4 6

Git Conversion Schedule
by Sebastian Redl 02 Mar '13

02 Mar '13

Hi, According to the timeline linked from the svn.boost.org frontpage, we're already using Git: https://svn.boost.org/trac/boost/wiki/ModCvtSchedule But I'm pretty sure that's not the case, right? What is the current schedule? Sebastian

3 2

[tweener] How to copy image files in documentation
by Julien Jorge 01 Mar '13

01 Mar '13

Hi, I wrote the documentation of Boost.Tweener in boostbook format. I have references to some images in the documentation but they are not copied in the target html directory when I type bjam html How can I make the image directory to be automatically copied in the target directory of the generated documentation? Since I am not familiar with the build process of Boost yet, I would accept a reason not to do this as an answer :) Regards, Julien Jorge

2 2