- Boost - lists.preview.boost.org

[compute] GPGPU Library - Request For Feedback
by Kyle Lutz 05 Mar '13

05 Mar '13

Hi everyone, A while back I posted a message asking for interest in a GPGPU computing library and the response seemed positive. I've been slowly working on it for the last few months and it has finally reached a usable state. I've made an initial release on GitHub (details below) and would like to get feedback from the community. The Boost Compute library provides a partial implementation of the C++ standard library for GPUs and multi-core CPUs. It includes common containers (vector<T>, flat_set<T>) and standard algorithms (transform, sort, accumulate). It also features a number of extensions including parallel-computing focused algorithms (exclusive_scan, scatter, reduce) along with a number of fancy iterators (transform_iterator, permutation_iterator). The library is built around the OpenCL framework which allows it to be portable across many types of devices (GPUs, CPUs, and accelerator cards) from many different vendors (NVIDIA, Intel, AMD). The source code and documentation are available from the links below. Code: https://github.com/kylelutz/compute Documentation: http://kylelutz.github.com/compute Bug Tracker: https://github.com/kylelutz/compute/issues I've tested the library with GCC 4.7 and Clang 3.3 on both NVIDIA GPUs and Intel CPUs. However, I would not yet consider the library production-ready. Most of my time has been devoted to reaching a solid and well-tested API rather than on performance. Over time this will improve. Feel free to send any questions, comments or feedback. Thanks, Kyle

7 13

[thread] thread_specific_ptr leaks on VS2012 when using std::async()
by Klaim - Joël Lamotte 05 Mar '13

05 Mar '13

Hi, The title says it all. I think I found a memory leak with the help of boost.log author Andrei Semashev. I was using Boost.Log in Visual Studio 2012 (released one) and tried to use Visual Leak Detector because Visual Studio basic leak detector was reporting leaks. I isolated the leaks to usage of log and switched to VLD to check if the basic detector wasn't giving me false positive. I contacted Andrei to see if he could check it but he don't have VS2012 available so I setup some tests to isolate the problem. In the end of our exchange we concluded it was a false positive (it only leak on end of program). However it is still an annoying noise in the leak report logs, so people not understanding the source of this problem might be interested in this email. I attach the whole email exchange to this email, tell me if leak logs are necessary too. The last version of my test is at the end of the email (first program). Basically, the leak appear only when using std::async() in combination of boost::thread_specific_ptr : - using exclusively boost threading libraries (both with dynamic and static linking) result in no leak - using exclusively std threading libraries but not std::async() result in no leak - using std::async() result in a leak The second test program shows that using Intel Threading Building Blocks, I get leaks only if : - I use std::async() in combination with boost::thread_specific_ptr, as before (I mean I use tbb without a leak, then I add a std::async() call to the same task function and there is a leak) - I don't use std::async() but I don't manage the lifetime of tbb task scheduler We think the threads spawned by both tbb (that is supposed to be a superset of ppl) and std::async() (that is supposed to use ppl/concrt task scheduler) are not ended yet when the program reach the end of main (which seems logical as the task scheduler is then managed as a static global), making boost::thread_specific_ptr not release some data on thread destruction, as it is notified too late and any leak detection tool willl generate noise due to this. A way to fix this is to explicitely end the task scheduler, being tbb or ppl/concrt. I will use tbb task scheduler explicitely initialized/terminated in my project so it will solve the problem on my side. Joel Lamotte ------------------------------------ // this program was used to see the memory leak #define TEST_ASYNC // comment this to make it linear and not leak //#define WITH_BOOST_THREADS #define WITH_STD_ASYNC // uncomment this to leak //#define WITH_BOOST_LOG // boost log also leak if std::async is used #include <boost/thread.hpp> #ifdef WITH_BOOST_THREADS # include <boost/thread/future.hpp> # include <boost/chrono.hpp> # define THIS_THREAD boost::this_thread # define THREAD_LIB boost::thread # define CHRONO_LIB boost::chrono # define MY_FUTURE boost::future #else # include <future> # include <thread> # include <chrono> # define THIS_THREAD std::this_thread # define THREAD_LIB std::thread # define CHRONO_LIB std::chrono # define MY_FUTURE std::future #endif #ifdef WITH_BOOST_LOG # include <boost/log/trivial.hpp> # define MY_LOG BOOST_LOG_TRIVIAL(info) #else # include <iostream> # define MY_LOG std::cout << '\n' #endif #define BASIC_VS_LEAK_DETECTOR 0 #define VISUAL_LEAK_DETECTOR 1 #define LEAK_DETECTOR VISUAL_LEAK_DETECTOR #if LEAK_DETECTOR == BASIC_VS_LEAK_DETECTOR # define _CRTDBG_MAP_ALLOC # include <stdlib.h> # include <crtdbg.h> #else # include "vld.h" #endif struct my_data { char data[1024]; my_data() { MY_LOG << "CONSTRUCTOR my_data - " << THIS_THREAD::get_id(); } ~my_data() { MY_LOG << "DESTRUCTOR my_data - " << THIS_THREAD::get_id(); } }; boost::thread_specific_ptr< my_data > ts_ptr; void my_thread_foo() { ts_ptr.reset(new my_data()); } int work() { for( int i = 0; i < 42; ++i ) { MY_LOG << "THIS IS A MESSAGE - I = " << i; MY_LOG << "ts_ptr = " << ts_ptr.get(); my_thread_foo(); THIS_THREAD::sleep_for( CHRONO_LIB::milliseconds(10) ); } return 0; } int main(int argc, char *argv[]) { #if LEAK_DETECTOR == BASIC_VS_LEAK_DETECTOR // First simple memory leak detection for Visual Studio // use _crtBreakAlloc to put a breakpoint on the provided memory leak id allocation //_crtBreakAlloc = 1149; _CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF ); #endif //int* k = new int[42]; // to check that memory leak detection does work MY_LOG << "Hello, World!"; #ifdef TEST_ASYNC # ifndef WITH_BOOST_THREADS std::vector< std::future<int> > work_in_progress(2); # ifdef WITH_STD_ASYNC for( auto& ftr : work_in_progress ) { ftr = std::async( work ); } # else for( auto& ftr : work_in_progress ) { std::packaged_task< int() > task(work); ftr = task.get_future(); std::thread thread( std::move(task) ); thread.detach(); } # endif for( auto& ftr : work_in_progress ) { ftr.get(); } # else std::vector< boost::unique_future<int> > work_in_progress(42); for( auto& ftr : work_in_progress ) { boost::packaged_task<int> task(work); // boost's packaged_task don't take the full signature as parameter, it takes the return type only ftr = task.get_future(); boost::thread thread( boost::move(task) ); thread.detach(); } for( auto& ftr : work_in_progress ) { ftr.get(); } # endif #else for( int i = 0; i < 42; ++i ) { work(); } #endif MY_LOG << "END"; return EXIT_SUCCESS; } -------------------------------------- // this program tests if the memory leaks appear with Threading Building Blocks #define BASIC_VS_LEAK_DETECTOR 0 #define VISUAL_LEAK_DETECTOR 1 #define LEAK_DETECTOR VISUAL_LEAK_DETECTOR #define WITH_ROOT_TASK //#define WITH_BOOST_LOG //#define WITH_STD_ASYNC // uncomment to leak #ifdef WITH_STD_ASYNC # include <future> #endif #include <thread> #include <atomic> #include <boost/thread.hpp> #include <tbb/tbb.h> #include <tbb/task_scheduler_init.h> #ifdef WITH_BOOST_LOG # include <boost/log/trivial.hpp> # define MY_LOG BOOST_LOG_TRIVIAL(info) #else # include <iostream> # define MY_LOG std::cout << '\n' #endif #if LEAK_DETECTOR == BASIC_VS_LEAK_DETECTOR # define _CRTDBG_MAP_ALLOC # include <stdlib.h> # include <crtdbg.h> #else # include "vld.h" #endif namespace test { struct my_data { char data[1024]; }; boost::thread_specific_ptr< my_data > ts_ptr; void my_thread_foo() { ts_ptr.reset(new my_data()); } class UpdateTask : public tbb::task { public: UpdateTask() : idx( s_idx++ ) { MY_LOG << "[UpdateTask " << idx << " ] " << std::endl; } ~UpdateTask() { MY_LOG << "[/UpdateTask " << idx << " ]" << std::endl; } tbb::task* execute() { MY_LOG << "[" << std::this_thread::get_id() << "] Tick "<< m_count <<std::endl; my_thread_foo(); ++m_count; if( m_count < 100 ) { // make sure this is re-executed this->increment_ref_count(); this->recycle_as_safe_continuation(); } return nullptr; } private: static std::atomic<int> s_idx; std::atomic<int> m_count; const int idx; }; std::atomic<int> UpdateTask::s_idx = 0; class TaskScheduler // using TBB { public: #ifdef WITH_ROOT_TASK TaskScheduler() : m_root_task( *new (tbb::task::allocate_root()) tbb::empty_task ) { m_root_task.increment_ref_count(); } ~TaskScheduler() { m_root_task.destroy(m_root_task); } #endif void register_update_task() { #ifdef WITH_ROOT_TASK m_root_task.increment_ref_count(); UpdateTask& updater = *new(m_root_task.allocate_child()) UpdateTask(); #else UpdateTask& updater = *new(tbb::task::allocate_root()) UpdateTask(); #endif m_task_list.push_back( updater ); } void run_and_wait() { #ifdef WITH_ROOT_TASK m_root_task.spawn( m_task_list ); m_root_task.wait_for_all(); #else tbb::task::spawn_root_and_wait( m_task_list ); #endif } private: tbb::task_list m_task_list; // uncomment the following line to not leak tbb //tbb::task_scheduler_init m_engine_instance; // We explicitly create and destroy the TBB task scheduler to avoid memory leak warning noises. #ifdef WITH_ROOT_TASK tbb::task& m_root_task; #endif }; void tbb_test() { TaskScheduler task_scheduler; for( int i = 0; i < 50; ++i ) { task_scheduler.register_update_task(); } task_scheduler.run_and_wait(); } } int main(int argc, char *argv[]) { #if LEAK_DETECTOR == BASIC_VS_LEAK_DETECTOR // First simple memory leak detection for Visual Studio // use _crtBreakAlloc to put a breakpoint on the provided memory leak id allocation //_crtBreakAlloc = 148; _CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF ); #endif #ifdef WITH_STD_ASYNC std::atomic<bool> finished = false; std::async( [&]{ while( !finished ) { MY_LOG << "POP"; test::my_thread_foo(); std::this_thread::sleep_for( std::chrono::milliseconds(500) ); } } ); #endif test::tbb_test(); #ifdef WITH_STD_ASYNC finished = true; #endif MY_LOG << "\nEND"; std::this_thread::sleep_for( std::chrono::seconds(4) ); return 0; }

3 5

[thread] C++11 once_flag enabled when constexpr is not available
by Andrey Semashev 05 Mar '13

05 Mar '13

Hi, Vicente! I see you applied my patch for call_once but you seem to forgot to disable the C++11 call_once interface when the compiler does not support constexpr. Was that intentional or you just missed it? Anyway, please find the patch attached.

3 3

[locale] Composing asymmetric locale for character encoding conversion
by Andrey Semashev 05 Mar '13

05 Mar '13

Hi, Suppose I have a logging application that writes log records in wide (wchar_t, UTF-16) and narrow (char, UTF-8) encodings and I want these logs to be stored in a UTF-16LE encoded file. For simplicity, let's assume that I write log files with std::wofstream. Now, the standard says that the file stream buffer is supposed to convert wide characters to byte sequences using the locale imbued into the buffer. However, it seems that the locale should be the same as the one imbued into the stream (basic_ostream::imbue makes sure of that). What this leads to is that in order to achieve my goal the locale should be able to convert narrow characters of UTF-8 to wide characters of UTF-16 and wide characters of UTF-16 to narrow characters representing byte sequence of UTF16LE. Is it possible to make such an asymmetric locale with Boost.Locale? Or maybe there is another way of doing this? An additional question. Is it possible to to achieve my goal with std::ofstream (as opposed to std::wofstream)? I have a very strong suspicion that the answer is no because the narrow characters will pass on unconverted to the file instead of being translated from UTF-8 to UTF-16LE, but maybe I'm missing something. Thank you.

3 4

[sandbox] [tree_node] Major update
by Cromwell Enage 05 Mar '13

05 Mar '13

Hello, fellow Boosters. In the hope that some of you here are still interested in easily extensible / augmented tree implementations, I've updated my Boost.TreeNode library in the sandbox. It provides adaptors for augmenting counts (a la countertree), heights, and even Boost.Accumulator values (which can solve the use cases mentioned in <http://je4d.blogspot.com/2013/01/boostintrusive-annotated-trees-part-1.html>), as well as concepts against which you can write your own adaptors. For a quick glance, documentation is at <http://svn.boost.org/svn/boost/sandbox/tree_node/libs/tree_node/doc/html/in…>. I'm still looking at the other two proposals. They have performance benchmarks which my library does not currently possess. However, at first glance I see no comparably straightforward way to augment user-defined data (e.g. Boost.Accumulator features) in either library. Feedback appreciated! Cromwell D. Enage

5 6

[coroutine] documentation comment
by Brian Wood 05 Mar '13

05 Mar '13

I wrote about this last month on Boost users but there wasn't a reply. I have a comment about a sentence on this page: http://www.boost.org/doc/libs/1_53_0/libs/coroutine/doc/html/coroutine/intr… "The principal difference between coroutines and routines is that a coroutine enables explicit suspend and resume their progress via additional operations by preserving local state ..." I'm wondering if it would be helpful to change that to this: The principal difference between coroutines and routines is that coroutines enable explicit suspend and resume of their progress via additional operations by preserving local state ... Shalom Brian Wood Ebenezer Enterprises http://webEbenezer.net (651) 251-9384

2 1

[tweener] Preliminary Submission
by Julien Jorge 05 Mar '13

05 Mar '13

Hello, I would like to propose the Boost.Tweener library for preliminary submission. The purpose of the Boost.Tweener library is to provide solutions to interpolate the intermediate values between two given values during a time interval. Various easing functions can be applied to control the computation of the intermediate values. I felt the need of such a library while developing games but I have also seen this need in various animation softwares. The library provides the following functionalities: - Interpolation is done on the value of a given variable or using an user-provided callback function, - The interpolated values are computed using a predefined easing function or any user-provided function respecting the contract, - Several tweeners can easily be executed simultaneously or in a row, - The type of the interpolated values are user defined. Please follow this link to download a Zip of the library: http://sourceforge.net/projects/libclaw/files/boost/boost-tweener-rc1.tar.g… Can you check that the library works as expected or tell me what should be fixed before submitting for review? There are various example programs in the archive. Regards, Julien Jorge

6 15

[thread] boost::thread and thread safety annotations
by Wladimir 04 Mar '13

04 Mar '13

Hello, The Clang C++ compiler project is currently working on a thread safety analyzer. By making use of so-called thread safety annotations within the source code, several problems with synchronization primitives can be found statically. For example: usage of a data structure without acquiring its guarding lock. See here for an overview of the available attributes: http://clang.llvm.org/docs/LanguageExtensions.html#threadsafety It is customary to use these attributes by defining macros, for compatibility: http://www.mail-archive.com/linuxkernelnewbies@googlegroups.com/msg01455.ht… At one time, it was also supposed to end up in G++, but this was abandoned in favor of clang (see http://gcc.gnu.org/wiki/ThreadSafetyAnnotation) But at least the annotations are fairly standardized. I think it would be useful to add these annotations to boost::thread, so that projects using that library can make use of the static checking. As I understand it, is only a matter of marking data structures that serve as lock/scoped_lock (such as boost::mutex, boost::recursive_mutex, ...), and methods that act upon the lock with the appropriate attributes. Any interest in this? Regards, Wladimir

2 1

[thread] barrier::wait() not marked as interruption_point
by Oliver Kowalke 04 Mar '13

04 Mar '13

In the documentation (1.53 beta) barrier::wait() is not marked as interruption-point nor that it will throw thread_interrupted if thread blocked inside barrier::wait() if it gets interrupted. Because barrier::wait() calls condition_variable::wait() and condition_variable::wait() is a interruption-point barrier::wait() should it be too - or do I miss something? Oliver

2 1

Re: [boost] [persistent] Library preview: Persistent
by Stefan Strasser 04 Mar '13

04 Mar '13

Am 04.03.2013 16:52, schrieb Jarrad Waterloo: > So if your library is in need, which it is, than release it, preferably > to boost under the boost license. You can always improve, add on latter. > You may solicit architecture and requirements from the list to increase > its usefulness and minimize rewrites. If you can't provide enough > manpower to complete it or even if you can, than solicit help. If four > people throughout the world can get it done in half the time of one > person than WOO HOO! There are many who would like to contribute from > the beginning, which also can't hurt acceptance. The important thing is > to have a vision, preferably shared, and to release something usable, > useful and working, as quickly as possible before interest dries up and > in a timely manner. > > Thanks for your remarks! Specifically for Persistent: > Have you considered an MVCC [option] where there are no updates from a > users perspective but are rather inserts; This sacrifices hard drive > space for a lock free/low lock database. yes, the entire storage backend is based on MVCC [1]. there currently is no lock-based option. objects are MVCC in memory for concurrency (see section "concurrent transactions" [2]), and on disk to achieve atomic updates. when an update is safely synced to disk the old object versions are discarded. I haven't written much about the backend in the documentation for two reasons: the interface is most important, and I actually don't think it would get used much in the long term. as much as I try to make it safe and efficient, a backend using an industrially proven storage engine like BerkeleyDB or SQLite would almost always be preferred, if the stored data has any value. I've written more about this in a previous reply to Klaim-Joël Lamotte in this thread. Would you like to share anything on the architecture of the interface? > Since you in some ways may be competing against relational databases by > making a object oriented database, maybe equivalent administrative > functionality are in order such as import and export into some textual > representations. > On a related note, ways to access system information such external > id/OIDS, timestamps, version numbers via an API for tooling and maybe > querying purposes > ?File format specifications? this and much more is certainly needed for database applications. but since we don't have the resources to implement an industrial-strength storage engine, my approach was to provide one "black-box" implementation for applications that would have used (boost.)serialization or other files otherwise, and provide resource managers using external databases later on. > > Concerning my interest, why haven't you given this to me two years ago, > how much longer must I wait. [Ha Ha Ha] *pointing to the copyright notice* :p Regards, Stefan [1] http://en.wikipedia.org/wiki/Multiversion_concurrency_control [2] https://svn.boost.org/svn/boost/sandbox/persistent/libs/persistent/doc/html…

2 1