December 2012 - Boost - lists.preview.boost.org

[thread] thread_specific_ptr leaks on VS2012 when using std::async()
by Klaim - Joël Lamotte 05 Mar '13

05 Mar '13

Hi, The title says it all. I think I found a memory leak with the help of boost.log author Andrei Semashev. I was using Boost.Log in Visual Studio 2012 (released one) and tried to use Visual Leak Detector because Visual Studio basic leak detector was reporting leaks. I isolated the leaks to usage of log and switched to VLD to check if the basic detector wasn't giving me false positive. I contacted Andrei to see if he could check it but he don't have VS2012 available so I setup some tests to isolate the problem. In the end of our exchange we concluded it was a false positive (it only leak on end of program). However it is still an annoying noise in the leak report logs, so people not understanding the source of this problem might be interested in this email. I attach the whole email exchange to this email, tell me if leak logs are necessary too. The last version of my test is at the end of the email (first program). Basically, the leak appear only when using std::async() in combination of boost::thread_specific_ptr : - using exclusively boost threading libraries (both with dynamic and static linking) result in no leak - using exclusively std threading libraries but not std::async() result in no leak - using std::async() result in a leak The second test program shows that using Intel Threading Building Blocks, I get leaks only if : - I use std::async() in combination with boost::thread_specific_ptr, as before (I mean I use tbb without a leak, then I add a std::async() call to the same task function and there is a leak) - I don't use std::async() but I don't manage the lifetime of tbb task scheduler We think the threads spawned by both tbb (that is supposed to be a superset of ppl) and std::async() (that is supposed to use ppl/concrt task scheduler) are not ended yet when the program reach the end of main (which seems logical as the task scheduler is then managed as a static global), making boost::thread_specific_ptr not release some data on thread destruction, as it is notified too late and any leak detection tool willl generate noise due to this. A way to fix this is to explicitely end the task scheduler, being tbb or ppl/concrt. I will use tbb task scheduler explicitely initialized/terminated in my project so it will solve the problem on my side. Joel Lamotte ------------------------------------ // this program was used to see the memory leak #define TEST_ASYNC // comment this to make it linear and not leak //#define WITH_BOOST_THREADS #define WITH_STD_ASYNC // uncomment this to leak //#define WITH_BOOST_LOG // boost log also leak if std::async is used #include <boost/thread.hpp> #ifdef WITH_BOOST_THREADS # include <boost/thread/future.hpp> # include <boost/chrono.hpp> # define THIS_THREAD boost::this_thread # define THREAD_LIB boost::thread # define CHRONO_LIB boost::chrono # define MY_FUTURE boost::future #else # include <future> # include <thread> # include <chrono> # define THIS_THREAD std::this_thread # define THREAD_LIB std::thread # define CHRONO_LIB std::chrono # define MY_FUTURE std::future #endif #ifdef WITH_BOOST_LOG # include <boost/log/trivial.hpp> # define MY_LOG BOOST_LOG_TRIVIAL(info) #else # include <iostream> # define MY_LOG std::cout << '\n' #endif #define BASIC_VS_LEAK_DETECTOR 0 #define VISUAL_LEAK_DETECTOR 1 #define LEAK_DETECTOR VISUAL_LEAK_DETECTOR #if LEAK_DETECTOR == BASIC_VS_LEAK_DETECTOR # define _CRTDBG_MAP_ALLOC # include <stdlib.h> # include <crtdbg.h> #else # include "vld.h" #endif struct my_data { char data[1024]; my_data() { MY_LOG << "CONSTRUCTOR my_data - " << THIS_THREAD::get_id(); } ~my_data() { MY_LOG << "DESTRUCTOR my_data - " << THIS_THREAD::get_id(); } }; boost::thread_specific_ptr< my_data > ts_ptr; void my_thread_foo() { ts_ptr.reset(new my_data()); } int work() { for( int i = 0; i < 42; ++i ) { MY_LOG << "THIS IS A MESSAGE - I = " << i; MY_LOG << "ts_ptr = " << ts_ptr.get(); my_thread_foo(); THIS_THREAD::sleep_for( CHRONO_LIB::milliseconds(10) ); } return 0; } int main(int argc, char *argv[]) { #if LEAK_DETECTOR == BASIC_VS_LEAK_DETECTOR // First simple memory leak detection for Visual Studio // use _crtBreakAlloc to put a breakpoint on the provided memory leak id allocation //_crtBreakAlloc = 1149; _CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF ); #endif //int* k = new int[42]; // to check that memory leak detection does work MY_LOG << "Hello, World!"; #ifdef TEST_ASYNC # ifndef WITH_BOOST_THREADS std::vector< std::future<int> > work_in_progress(2); # ifdef WITH_STD_ASYNC for( auto& ftr : work_in_progress ) { ftr = std::async( work ); } # else for( auto& ftr : work_in_progress ) { std::packaged_task< int() > task(work); ftr = task.get_future(); std::thread thread( std::move(task) ); thread.detach(); } # endif for( auto& ftr : work_in_progress ) { ftr.get(); } # else std::vector< boost::unique_future<int> > work_in_progress(42); for( auto& ftr : work_in_progress ) { boost::packaged_task<int> task(work); // boost's packaged_task don't take the full signature as parameter, it takes the return type only ftr = task.get_future(); boost::thread thread( boost::move(task) ); thread.detach(); } for( auto& ftr : work_in_progress ) { ftr.get(); } # endif #else for( int i = 0; i < 42; ++i ) { work(); } #endif MY_LOG << "END"; return EXIT_SUCCESS; } -------------------------------------- // this program tests if the memory leaks appear with Threading Building Blocks #define BASIC_VS_LEAK_DETECTOR 0 #define VISUAL_LEAK_DETECTOR 1 #define LEAK_DETECTOR VISUAL_LEAK_DETECTOR #define WITH_ROOT_TASK //#define WITH_BOOST_LOG //#define WITH_STD_ASYNC // uncomment to leak #ifdef WITH_STD_ASYNC # include <future> #endif #include <thread> #include <atomic> #include <boost/thread.hpp> #include <tbb/tbb.h> #include <tbb/task_scheduler_init.h> #ifdef WITH_BOOST_LOG # include <boost/log/trivial.hpp> # define MY_LOG BOOST_LOG_TRIVIAL(info) #else # include <iostream> # define MY_LOG std::cout << '\n' #endif #if LEAK_DETECTOR == BASIC_VS_LEAK_DETECTOR # define _CRTDBG_MAP_ALLOC # include <stdlib.h> # include <crtdbg.h> #else # include "vld.h" #endif namespace test { struct my_data { char data[1024]; }; boost::thread_specific_ptr< my_data > ts_ptr; void my_thread_foo() { ts_ptr.reset(new my_data()); } class UpdateTask : public tbb::task { public: UpdateTask() : idx( s_idx++ ) { MY_LOG << "[UpdateTask " << idx << " ] " << std::endl; } ~UpdateTask() { MY_LOG << "[/UpdateTask " << idx << " ]" << std::endl; } tbb::task* execute() { MY_LOG << "[" << std::this_thread::get_id() << "] Tick "<< m_count <<std::endl; my_thread_foo(); ++m_count; if( m_count < 100 ) { // make sure this is re-executed this->increment_ref_count(); this->recycle_as_safe_continuation(); } return nullptr; } private: static std::atomic<int> s_idx; std::atomic<int> m_count; const int idx; }; std::atomic<int> UpdateTask::s_idx = 0; class TaskScheduler // using TBB { public: #ifdef WITH_ROOT_TASK TaskScheduler() : m_root_task( *new (tbb::task::allocate_root()) tbb::empty_task ) { m_root_task.increment_ref_count(); } ~TaskScheduler() { m_root_task.destroy(m_root_task); } #endif void register_update_task() { #ifdef WITH_ROOT_TASK m_root_task.increment_ref_count(); UpdateTask& updater = *new(m_root_task.allocate_child()) UpdateTask(); #else UpdateTask& updater = *new(tbb::task::allocate_root()) UpdateTask(); #endif m_task_list.push_back( updater ); } void run_and_wait() { #ifdef WITH_ROOT_TASK m_root_task.spawn( m_task_list ); m_root_task.wait_for_all(); #else tbb::task::spawn_root_and_wait( m_task_list ); #endif } private: tbb::task_list m_task_list; // uncomment the following line to not leak tbb //tbb::task_scheduler_init m_engine_instance; // We explicitly create and destroy the TBB task scheduler to avoid memory leak warning noises. #ifdef WITH_ROOT_TASK tbb::task& m_root_task; #endif }; void tbb_test() { TaskScheduler task_scheduler; for( int i = 0; i < 50; ++i ) { task_scheduler.register_update_task(); } task_scheduler.run_and_wait(); } } int main(int argc, char *argv[]) { #if LEAK_DETECTOR == BASIC_VS_LEAK_DETECTOR // First simple memory leak detection for Visual Studio // use _crtBreakAlloc to put a breakpoint on the provided memory leak id allocation //_crtBreakAlloc = 148; _CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF ); #endif #ifdef WITH_STD_ASYNC std::atomic<bool> finished = false; std::async( [&]{ while( !finished ) { MY_LOG << "POP"; test::my_thread_foo(); std::this_thread::sleep_for( std::chrono::milliseconds(500) ); } } ); #endif test::tbb_test(); #ifdef WITH_STD_ASYNC finished = true; #endif MY_LOG << "\nEND"; std::this_thread::sleep_for( std::chrono::seconds(4) ); return 0; }

3 5

[thread] boost::thread and thread safety annotations
by Wladimir 04 Mar '13

04 Mar '13

Hello, The Clang C++ compiler project is currently working on a thread safety analyzer. By making use of so-called thread safety annotations within the source code, several problems with synchronization primitives can be found statically. For example: usage of a data structure without acquiring its guarding lock. See here for an overview of the available attributes: http://clang.llvm.org/docs/LanguageExtensions.html#threadsafety It is customary to use these attributes by defining macros, for compatibility: http://www.mail-archive.com/linuxkernelnewbies@googlegroups.com/msg01455.ht… At one time, it was also supposed to end up in G++, but this was abandoned in favor of clang (see http://gcc.gnu.org/wiki/ThreadSafetyAnnotation) But at least the annotations are fairly standardized. I think it would be useful to add these annotations to boost::thread, so that projects using that library can make use of the static checking. As I understand it, is only a matter of marking data structures that serve as lock/scoped_lock (such as boost::mutex, boost::recursive_mutex, ...), and methods that act upon the lock with the appropriate attributes. Any interest in this? Regards, Wladimir

2 1

Fwd: [Thread] Solution to conflict with MFC?
by Thorsten Ottosen 28 Feb '13

28 Feb '13

Hi Thread maintainers, Anthony speaks about a patch to make this work. Does anyone know where it is? And could it be applied? Thanks -Thorsten -------- Original Meddelelse -------- Subject: [Thread] Solution to conflict with MFC? Date: Tue, 26 Apr 2011 13:28:56 +0200 From: Christian Larsen <contact(a)dword.dk> Reply-To: boost-users(a)lists.boost.org To: boost-users(a)lists.boost.org Newsgroups: gmane.comp.lib.boost.user Hello, I know this has been asked before, but I could not find a good solution. I'm experiencing problems with an application using both MFC and Boost.Thread. Specifically the assertion ASSERT(AfxGetModuleState() != AfxGetAppModuleState()); in AfxCoreInitModule() fails. I link Boost.Thread statically into a DLL in my project. From previous posts to this list and elsewhere, e.g. http://lists.boost.org/boost-users/2009/04/46929.php, I see that the issue is caused by setting _pRawDllMain in tss_pe.cpp. Commenting this line does "solve" the problem, but I would rather not have to hack the Boost source code this way. It seems there is a patch somewhere for this problem (referred to in http://lists.boost.org/boost-users/2009/04/47015.php) but it appears it never made it into the source. Is there any chance of this issue being solved in a better way? Best regards, Christian Larsen

5 9

[Config] N3276 decltype support on VC++ 11
by Michel Morin 28 Feb '13

28 Feb '13

IIRC, Stephan said that VC++ 11 supports N3276 decltype. Shall we "undef" BOOST_NO_CXX11_DECLTYPE_N3276 for VC++ 11? This enables automatic use of decltype-based `boost::result_of`. Trivial patch attached. Regards, Michel

11 79

[multi_index] Move Semantics
by Mark Snelling 27 Feb '13

27 Feb '13

Hi, I've been a user of Boost.MultiIndex for a number of years and think it's a great library. Is there any firm plan to add move semantics? It's been noted under "Future Work" in the docs since Boost 1.32. Regards, Mark

6 10

Problems with building quickbook docs
by Andrey Semashev 21 Feb '13

21 Feb '13

Hi, I'm having problems with building quickbook docs for my library. I have the using quickbook ; as well as doxygen, xsltproc and boostbook in my ~/user-config.jam. When I run: bjam -j 2 release link=static libs/log/doc I get the following error: .../boost-release/tools/build/v2/build/generators.jam:978: in try-one-generator-really *** argument error * rule object(generator)(a)60.run ( project name ? : property-set : sources + ) * called with: ( object(project-target)@1010 filters_reference-xml : object(property-set)@1387 : ) * missing argument sources .../boost-release/tools/build/v2/build/generators.jam:370:see definition of rule 'object(generator)(a)60.run' being called .../boost-release/tools/build/v2/build/generators.jam:1040: in try-one-generator .../boost-release/tools/build/v2/build/generators.jam:1254: in construct-really .../boost-release/tools/build/v2/build/generators.jam:1338: in generators.construct .../boost-release/tools/build/v2/build/targets.jam:1495: in construct .../boost-release/tools/build/v2/build/targets.jam:1298: in object(typed-target)(a)1058.generate .../boost-release/tools/build/v2/build/targets.jam:732: in generate-really .../boost-release/tools/build/v2/build/targets.jam:704: in object(main-target)(a)1164.generate .../boost-release/tools/build/v2/build/targets.jam:848: in targets.generate-from-reference .../boost-release/tools/build/v2/build/targets.jam:1217: in generate-dependencies .../boost-release/tools/build/v2/build/targets.jam:1269: in object(typed-target)(a)1061.generate .../boost-release/tools/build/v2/build/targets.jam:732: in generate-really .../boost-release/tools/build/v2/build/targets.jam:704: in object(main-target)(a)1165.generate .../boost-release/tools/build/v2/build/targets.jam:848: in targets.generate-from-reference .../boost-release/tools/build/v2/build/targets.jam:1217: in generate-dependencies .../boost-release/tools/build/v2/build/targets.jam:1269: in object(typed-target)(a)1062.generate .../boost-release/tools/build/v2/build/targets.jam:732: in generate-really .../boost-release/tools/build/v2/build/targets.jam:704: in object(main-target)(a)1166.generate .../boost-release/tools/build/v2/build/targets.jam:848: in targets.generate-from-reference .../boost-release/tools/build/v2/build/targets.jam:1217: in generate-dependencies .../boost-release/tools/build/v2/build/targets.jam:1269: in object(install-target-class)(a)1065.generate .../boost-release/tools/build/v2/build/targets.jam:732: in generate-really .../boost-release/tools/build/v2/build/targets.jam:704: in object(main-target)(a)1167.generate .../boost-release/tools/build/v2/build/targets.jam:848: in targets.generate-from-reference .../boost-release/tools/build/v2/build/targets.jam:1217: in generate-dependencies .../boost-release/tools/build/v2/build/targets.jam:1399: in basic-target.compute-usage-requirements .../boost-release/tools/build/v2/build/alias.jam:47: in compute-usage-requirements .../boost-release/tools/build/v2/build/targets.jam:1319: in object(alias-target-class)(a)1068.generate .../boost-release/tools/build/v2/build/targets.jam:732: in generate-really .../boost-release/tools/build/v2/build/targets.jam:704: in object(main-target)(a)1168.generate .../boost-release/tools/build/v2/build/targets.jam:258: in object(project-target)(a)1010.generate .../boost-release/tools/build/v2/build-system.jam:736: in load .../boost-release/tools/build/v2/kernel/modules.jam:283: in import .../boost-release/tools/build/v2/kernel/bootstrap.jam:142: in boost-build .../boost-release/boost-build.jam:17: in module scope index.html libs/index.html more/getting_started.html tools/index.html I'm using the release branch. Does anyone know what the problem could be?

2 2

Release Managers: permission to merge small feature to release?
by Marshall Clow 19 Feb '13

19 Feb '13

I have a ticket <https://svn.boost.org/trac/boost/ticket/7652> asking for compile-time checked access to elements of a boost::array. I have implemented a specialization of boost::get (and for c++11, std::get) that does this, and it's been on the trunk and being tested for a couple weeks. This is technically not a bug fix, but new functionality, but I'd like to get it into the next release. So, I'm asking permission to merge to release. The change lists are 82083 (for the functionality), 82084 and 82102 (for the tests), and 82103 (for the docs) Thanks! -- Marshall

5 5

[Lockfree] ringbuffer
by Rich E 17 Feb '13

17 Feb '13

First off, all Boost.Lockfree tests in trunk are passing for me on clang 4.0 on OS X, hurrah! Thank you Tim for working on this for so long... I'm running through the examples now, only with the outdated documentation as a guide (more than a couple years old). I cannot find the ringbuffer container - was it renamed or removed altogether during the review period? I notice spsc_queue acts awefully like a ringbuffer, and it's implementation contains many

2 6

[fixed_point] Request for interest in a binary fixed point library
by Vicente J. Botet Escriba 02 Feb '13

02 Feb '13

Hi, the recent discussion on the MultiplePrecission Arithmetic library has show that some people has its ow fixed point library. Is there an interest in a Boost library having as base http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3352.html? I have started a prototype http://svn.boost.org/svn/boost/sandbox/fixed_point (there is no doc yet but the basic goal is quite close to the N3352 C++ proposal). This prototype should allow you to play a little bit with. If yes, what would you like to be changed, added or improved in n3352? Next follows some design decisions that IMO need to be decided before hand. * Should integers and reals be represented by separated classes? * Should signed and unsigned be represented by separated classes? * Should the library use a specific representation for signed numbers (separated sign, 2-complement? Let the user choose? * Should the library provide arbitrary range and resolution and allocators? * Should the library be open to overflow and rounding or just implement some of the possible policies? and in this case which ones? * Should fixed_point be convertible to/from integer/float/double? * Could the result of an arithmetic operation have more range and resolution than his arguments? * Is there a need for a specific I/O? * is there a need for radix other than 2 (binary)? * Should the library implement the basic functions, or should it imperatively implement the C++11 math functions? Could a first version just forward to the c++11 math functions? * Should the library support just one of the know ways to name a fixed-point, a U(a,b), nQm, ...? Provide some ways to move from one to another? * Could expect the same/better performances respect to hand written code? * What should be the size used by a fixed_point instance? _fast? _least? Should the user be able to decide which approach is better for his needs? * Which should be the namespace? boost? boost/fixed_point? boost/binary_fixed_point? boost/bfp? ... * others you can think of. Please, replay if you are interested, have some experience in the domain, good ideas, .. Best regards, Vicente

6 44

program_options: Support for open-end options
by ST 27 Jan '13

27 Jan '13

Hi, It would be great to implement support for open-end options, something like that: item_1 = 23 item_2 = 45 ... item_N = 465 The idea is to to be able to provide following input: configOptions.add_options() ("item_", value<int>(), "items"); now if add_options() sees a key that ends with a "_" it accepts all options with keys that start with item_, no matter what comes after it, and treat all of them as int . Nesting should also be possible - like this: "item_.subitem_.subsubitem_" (item_3.subitem_FOO.subsubitem_34 should be a valid option key). I can try to implement it, however only if it will have a chance to be merged into the boost library. Whom should I contact regarding this? Thank you, Simone

5 13