December 2009 - Boost-users - lists.preview.boost.org

Re: [Boost-users] thread_group::interrupt_all is not reliable
by Stonewall Ballard 01 Dec '09

01 Dec '09

On Dec 1, 2009, at 6:05 PM, Roland Bock wrote: > Stonewall Ballard wrote: > [...] >>> In such a case, notify_one would wake less threads than required. But how can that happen with notify_one being called for each and every task? >> Take a look at this: >> <http://www.justsoftwaresolutions.co.uk/threading/implementing-a-thread-safe…> > [...] >> I added the comment to the code in case someone came along later and decided that it was wasteful to notify_all(). > > Hmm. Have you read the section "UPDATE" on the page? To me that says that the requirement for the notify_all has already been taken care of in the presented code (by calling it each time a task is pushed). Ah, I'd missed that. Thanks for pointing it out. It takes several readings to understand all this. >> My code is running in the idle loop of a dialog window, so I had to avoid blocking the producer for more than a moment. Since there could be many more available tasks than threads, a queue was the obvious choice. > > Yes, a blocking dialog would be bad. I am using the threadpool for a merge process which can delegate the processing of chunks to its workers. At the time I wrote it, I thought that blocking would not be problem (and in the intended scenario it isn't), because I can rarely use all available threads due to the sheer size of the chunks. The merger blocks that anyway. > > But after today, I think I will change my pool. Limiting the number of tasks is part of the merge logic, the pool should not do something like that. May as well code a general-purpose pool so that it's debugged by the time you need it for another program. Thanks again for your help. - Stoney -- Stonewall Ballard stoney(a)sb.org http://stoney.sb.org/

1 0

thread_group::interrupt_all is not reliable
by Stonewall Ballard 01 Dec '09

01 Dec '09

I've discovered that under circumstances apparently related to timing and load, sending interrupt_all to a thread_group when all the threads are waiting on a boost::condition_variable leaves one thread waiting about 1/3 of the time. This is with boost 1_40_0 running on Mac OS X 10.6.2, with 32-bit boost libraries. Boost uses the posix thread system here. I boiled my app down to some test code that runs as a command-line app. It's a bit longer than I'd like, but this configuration seems to be necessary to invoke the problem. The test uses a queue to pass "tasks" from the main thread to worker threads, and another queue to pass "results" back to the main thread. The problem is most apparent when all the tasks are finished and the queue empties, so that all the worker threads are waiting on the input queue when the main thread sends interrupt_all. I've looked at the waiting thread in a debugger when this happens, and found that it has been interrupted, but is still waiting on the condition. It looks like it just got missed by the interrupt_all. This is more likely to happen when there are a lot of worker threads (16, or one per core in my testing). The test code is parked at <http://sb.org/ThreadTest.zip>, 20KB. It's an XCode 3.2 project, but the five source files could be readily compiled and run in any Unix environment. I don't see any errors in the code that could cause these failures. There is a work-around, which is to interrupt the waiting thread again. This required a modified version of thread_group so I could do a timed_join_all on it. I welcome any suggestions about what could be wrong here, or ways to simplify the test to make it more suitable for a bug report. - Stoney -- Stonewall Ballard stoney(a)sb.org http://stoney.sb.org/

2 3

[coroutines] Stack corruption bug
by Nat Goodspeed 01 Dec '09

01 Dec '09

My colleague Mark Palange discovered a Boost.Coroutines scenario that can lead to stack corruption. The normal usage for a boost::coroutines::coroutine object is to repeatedly invoke its operator() method. Each entry to this method rebinds the coroutine_impl::m_result pointer to the address of a stack result_slot_type* variable. So far, so good. When a coroutine waits for a boost::coroutines::future object, the original operator() call exits, destroying the stack result_slot_type* variable. When that coroutine is later resumed by calling the future object's callback, coroutine_impl::m_result is *not* rebound: it's left set to the now-destroyed (possibly reused) address. Finally, when the coroutine exits, coroutine_impl::bind_result() blithely assigns through *m_result, thereby overwriting that slot on the stack. Now we're in Undefined Behavior territory. The attached patch file addresses the problem in two ways: 1. Each of the three methods that calls coroutine_impl::bind_result_pointer() (coroutine::call_impl(), coroutine::call_impl_nothrow(), coroutine_impl::run()) now uses an instance of a new helper class coroutine_impl::local_result_slot_ptr. This class contains the result_slot_type* variable and calls bind_result_pointer(&member) when constructed, wrapping the existing logic. But it also calls bind_result_pointer(NULL) when destroyed. This way, an invalid attempt to dereference *m_result is more likely to self-identify than to lead to mysterious, not obviously related side effects. 2. coroutine_impl::bind_result() now avoids the assignment if m_result is NULL. Possibly there are better behaviors for bind_result() on NULL m_result than silently bypassing the assignment, but this works for us, since our typical coroutine is void-valued anyway. If you're using boost::coroutines::coroutine and waiting on a boost::coroutines::future, please apply the attached patch to your copy of the Boost.Coroutines library. Alternatively, update the whole library with the new tarball: http://www.boostpro.com/vault/index.php?action=downloadfile&filename=boost-… diff -rc boost-coroutine/boost/coroutine/coroutine.hpp /Users/nat/linden/viewer-20-topush/libraries/include/boost/coroutine/coroutine.hpp *** boost-coroutine/boost/coroutine/coroutine.hpp Wed Apr 29 14:41:05 2009 --- /Users/nat/linden/viewer-20-topush/libraries/include/boost/coroutine/coroutine.hpp Tue Dec 1 15:25:52 2009 *************** *** 258,265 **** result_type call_impl(arg_slot_type args) { BOOST_ASSERT(m_pimpl); m_pimpl->bind_args(&args); ! result_slot_type * ptr; ! m_pimpl->bind_result_pointer(&ptr); m_pimpl->invoke(); return detail::fix_result<result_slot_traits>(*m_pimpl->result()); --- 261,267 ---- result_type call_impl(arg_slot_type args) { BOOST_ASSERT(m_pimpl); m_pimpl->bind_args(&args); ! BOOST_DEDUCED_TYPENAME impl_type::local_result_slot_ptr ptr(m_pimpl); m_pimpl->invoke(); return detail::fix_result<result_slot_traits>(*m_pimpl->result()); *************** *** 270,277 **** call_impl_nothrow(arg_slot_type args) { BOOST_ASSERT(m_pimpl); m_pimpl->bind_args(&args); ! result_slot_type * ptr; ! m_pimpl->bind_result_pointer(&ptr); if(!m_pimpl->wake_up()) return detail::optional_result<result_type>(); --- 272,278 ---- call_impl_nothrow(arg_slot_type args) { BOOST_ASSERT(m_pimpl); m_pimpl->bind_args(&args); ! BOOST_DEDUCED_TYPENAME impl_type::local_result_slot_ptr ptr(m_pimpl); if(!m_pimpl->wake_up()) return detail::optional_result<result_type>(); diff -rc boost-coroutine/boost/coroutine/detail/coroutine_impl.hpp /Users/nat/linden/viewer-20-topush/libraries/include/boost/coroutine/detail/coroutine_impl.hpp *** boost-coroutine/boost/coroutine/detail/coroutine_impl.hpp Sun Aug 20 13:11:09 2006 --- /Users/nat/linden/viewer-20-topush/libraries/include/boost/coroutine/detail/coroutine_impl.hpp Tue Dec 1 16:04:48 2009 *************** *** 89,95 **** } void bind_result(result_slot_type* res) { ! *m_result = res; } // Another level of indirecition is needed to handle --- 89,98 ---- } void bind_result(result_slot_type* res) { ! // This used to be unconditional. But m_result isn't always valid. ! if (m_result) { ! *m_result = res; ! } } // Another level of indirecition is needed to handle *************** *** 102,120 **** return m_result; } // This function must be called only for void // coroutines. It wakes up the coroutine. // Entering the wait state does not cause this // method to throw. void run() { arg_slot_type void_args; - result_slot_type * ptr = 0; // This dummy binding is required because // do_call expect args() and result() // to return a non NULL result. bind_args(&void_args); ! bind_result_pointer(&ptr); this->wake_up(); } protected: --- 105,157 ---- return m_result; } + /// This helper class packages data/logic originally found inline in + /// coroutine::call_impl() and call_impl_nothrow(), also + /// coroutine_impl::run(). + class local_result_slot_ptr + { + public: + local_result_slot_ptr(pointer pimpl): + m_pimpl(pimpl), + m_ptr(NULL) + { + m_pimpl->bind_result_pointer(&m_ptr); + } + ~local_result_slot_ptr() + { + // In the original use case, a coroutine could only be resumed by + // calling coroutine::operator() again, which would rebind the + // result pointer to a new valid value. But with the introduction + // of futures, it's possible to suspend a coroutine by waiting on + // a future object -- thus destroying the local result_slot_type* + // -- then resume that coroutine by calling the future's callback, + // bypassing coroutine::operator(). This used to leave an old, + // invalid result pointer in effect. Subsequent coroutine exit + // wrote through that pointer, munging a word of stack. Now we + // make a point of setting the bound result pointer NULL when the + // result_slot_type* to which it pointed vanishes, so that any + // attempt to dereference it will at least self-identify -- + // instead of producing arbitrary undefined behavior. + m_pimpl->bind_result_pointer(NULL); + } + + private: + pointer m_pimpl; + result_slot_type* m_ptr; + }; + // This function must be called only for void // coroutines. It wakes up the coroutine. // Entering the wait state does not cause this // method to throw. void run() { arg_slot_type void_args; // This dummy binding is required because // do_call expect args() and result() // to return a non NULL result. bind_args(&void_args); ! local_result_slot_pointer(this); this->wake_up(); } protected: diff -rc boost-coroutine/boost/coroutine/detail/self.hpp /Users/nat/linden/viewer-20-topush/libraries/include/boost/coroutine/detail/self.hpp *** boost-coroutine/boost/coroutine/detail/self.hpp Sun Aug 20 13:11:09 2006 --- /Users/nat/linden/viewer-20-topush/libraries/include/boost/coroutine/detail/self.hpp Wed Aug 5 16:50:17 2009 *************** *** 217,222 **** --- 217,235 ---- BOOST_ASSERT(m_pimpl); return m_pimpl->pending(); } + + /// @c true only if this @c self object was created by the passed @a coroutine + template <typename SomeCoroutine> + bool is_from(const SomeCoroutine& coroutine) const + { + // get_impl() only accepts non-const ref... a mistake, IMO. + return static_cast<void*>(coroutine_accessor::get_impl(const_cast<SomeCoroutine&>(coroutine)).get()) == + static_cast<void*>(m_pimpl); + } + + /// opaque token used to correlate this 'self' with its corresponding coroutine + void* get_id() const { return m_pimpl; } + private: coroutine_self(impl_type * pimpl, detail::init_from_impl_tag) : m_pimpl(pimpl) {}

1 0

Re: [Boost-users] [OT] Interested in good (fast, robust)
by Richard Ulrich 01 Dec '09

01 Dec '09

I'm using json_spirit from http://www.codeproject.com/KB/recipes/JSON_Spirit.aspx for parsing big but simple json files. So far I didn't have any problems with it. It's still based on classic spirit. Don't know if and when it will switch to spirit 2. Rgds Richard On Tue, 2009-12-01 at 12:00 -0500, boost-users-request(a)lists.boost.org wrote: > Date: Tue, 01 Dec 2009 17:38:36 +0100 > From: Sebastian Redl <sebastian.redl(a)getdesigned.at> > To: boost-users(a)lists.boost.org > Subject: Re: [Boost-users] [OT] Interested in good (fast, robust) > boost-based JSON API > Message-ID: <4B15468C.7000309(a)getdesigned.at> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Dominique Devienne wrote: > > Can anyone recommend one such good targeted library up to Boost's > > standards you've had good success with? (with a boost-like > commercial > > friendly license) > > > Boost.PropertyTree has some support for parsing and writing JSON, but > it > doesn't preserve types. > > Sebastian

3 2

[Ubuntu 8.04] Linking Problem in Eclipse with Boost 1.41 Libraries
by Brooks Garrison 01 Dec '09

01 Dec '09

Hello all, My question is concerning the Boost installation process. I am currently working on a machine that has Ubuntu 8.04 LTS. I have downloaded the latest copy of Boost (1.41.0) and am trying to successfully install it. I have followed the procedures for building the libraries such as Filesystem / Regex / Thread / etc. using both the bootstrap, "easy", method and the "build custom binaries" method. I'm using Eclipse as my C++ development environment and I can create programs that use the Boost headers (the ones that do not require compilation - ie. Lambda) just fine. When I try to implement the sample program that incorporates [regex], I am able to compile and link just fine. I've set the include directory to ( /usr/local/boost_1_41_0/prefix/include - prefix was just the name I gave the folder), the library search path to ( /usr/local/boost_1_41_0/prefix/lib ) and the library I wanted to link to, [regex], using (boost_regex - I have also tried boost_regex-gcc##-mt ). As I said though, when I try to run the program, I get the result: ./boost_regex_test: error while loading shared libraries: libboost_regex.so.1.41.0: cannot open shared object file: No such file or directory I have not specified that I was using shared libraries, so I do not understand why it is trying to look for them. If anyone could please clarify why this is happening, I'd be more than greatful. Just as an aside, I also tried downloading and installing Boost from the Synaptic Package Manager that comes with Ubuntu 8.04. I'm able to use all of the libraries when I do this, but I'd rather not because the version of Boost is 1.34.1 (quite archaic) and has cause problems because member variables have evolved and some no longer exist. Thanks again for any and all help.

2 10

lexical_cast and default params
by Noah Roberts 01 Dec '09

01 Dec '09

Running into something very strange with boost::lexical_cast in 1.39. This code works, but notice the f(). If I attempt to get rid of it and call boost::lexical_cast directly in the constructor for test I get this error: 1>d:\dev_workspace\experimental\scratch\scratch\main.cpp(16) : error C2783: 'Target boost::lexical_cast(const Source &)' : could not deduce template argument for 'Target' 1> d:\boostvs39\include\boost\lexical_cast.hpp(1164) : see declaration of 'boost::lexical_cast' This is hard for me to understand since I've given the target type as a template parameter....and the f() version works just fine. It even works if I make it inline as the lexical_cast function is. Any ideas? #include <boost/lexical_cast.hpp> #include <string> #include <iostream> template < typename R, typename T > R f(T const& x) { return boost::lexical_cast<R>(x); } template < typename T > struct test { test(std::string const& x = f<std::string>(T())) : test_string(x) {} std::string test_string; }; int main() { test<int> t; std::cout << t.test_string; std::cout << boost::lexical_cast<std::string>(int()); std::cin.get(); }

1 1

[Serialization - 1.35] Crash while serializing
by Soumen 01 Dec '09

01 Dec '09

Hi, I've following class hierarchy: FilterCondition <---+--- CompositeCondition | +--- ConcreteCondition FilterCondition is ABC. FilterCondition has certain data members which I need to serialize. I already have another class "Table" for which serialization works perfectly. Now, I've added a pointer of type "FilterCondition" as a member to "Table". Say, the member name is "filterCond_". Now, as soon as I add following line in Table::serialize(), I get a crash : ar & filterCond_; This happens even though I make body of FilterCondition::serialize(), CompositeCondition::serialize() and ConcreteCondition::serialize() empty. In Table::serialize(), I've verified that filterCond_ is a valid pointer. I've declared FilterCondition as abstract through BOOST_CLASS_ABSTRACT macro as well. Following is top of stack from gdb: ------------------------------------------------------------------------------------------------ (gdb) where #0 0x0000000000fcfee6 in boost::serialization::detail::extended_type_info_typeid_0::less_than () #1 0x0000000000fcf703 in boost::serialization::extended_type_info::operator< () #2 0x0000000000fcf78b in boost::serialization::extended_type_info::find () #3 0x0000000000fcffea in boost::serialization::detail::extended_type_info_typeid_0::get_derived_extended_type_info () #4 0x000000000136ca0c in boost::serialization::detail::extended_type_info_typeid_1<FilterCondition const>::get_derived_extended_type_info (t=@0x19c6e390) at /remote/arch-tools/boost/qsc-b/1_35_0/include/boost/serialization/extended_type_info_typeid.hpp:92 #5 0x000000000136ca5b in boost::archive::detail::save_pointer_type<boost::archive::binary_oarchive, FilterCondition*>::polymorphic<FilterCondition>::save (ar=@0x7fbfffc410, t=@0x19c6e390, bpos_ptr=0x0) at /remote/arch-tools/boost/qsc-b/1_35_0/include/boost/archive/detail/oserializer.hpp:385 #6 0x000000000136cbed in boost::archive::detail::save_pointer_type<boost::archive::binary_oarchive, FilterCondition*>::save<FilterCondition> (ar=@0x7fbfffc410, t=@0x19c6e390, bpos_ptr=0x0) at /remote/arch-tools/boost/qsc-b/1_35_0/include/boost/archive/detail/oserializer.hpp:434 #7 0x000000000136cc4a in boost::archive::detail::save_pointer_type<boost::archive::binary_oarchive, FilterCondition*>::invoke (ar=@0x7fbfffc410, t=0x19c6e390) at /remote/arch-tools/boost/qsc-b/1_35_0/include/boost/archive/detail/oserializer.hpp:462 #8 0x000000000136cc6c in boost::archive::save<boost::archive::binary_oarchive, FilterCondition*> (ar=@0x7fbfffc410, t=@0x19c6d820) at /remote/arch-tools/boost/qsc-b/1_35_0/include/boost/archive/detail/oserializer.hpp:532 #9 0x000000000136cc96 in boost::archive::detail::common_oarchive<boost::archive::binary_oarchive>::save_override<FilterCondition* const> (this=0x7fbfffc410, t=@0x19c6d820) at /remote/arch-tools/boost/qsc-b/1_35_0/include/boost/archive/detail/common_oarchive.hpp:64 #10 0x000000000136ccbb in boost::archive::array::oarchive<boost::archive::binary_oarchive>::save_override<FilterCondition*> (this=0x7fbfffc410, x=@0x19c6d820, version=0) at /remote/arch-tools/boost/qsc-b/1_35_0/include/boost/archive/array/oarchive.hpp:122 #11 0x000000000136cce3 in boost::archive::basic_binary_oarchive<boost::archive::binary_oarchive>::save_override<FilterCondition*> (this=0x7fbfffc410, t=@0x19c6d820) at /remote/arch-tools/boost/qsc-b/1_35_0/include/boost/archive/basic_binary_oarchive.hpp:64 #12 0x000000000136cd0b in boost::archive::binary_oarchive_impl<boost::archive::binary_oarchive, char, std::char_traits<char> >::save_override<FilterCondition* const> (this=0x7fbfffc410, t=@0x19c6d820) at /remote/arch-tools/boost/qsc-b/1_35_0/include/boost/archive/binary_oarchive_impl.hpp:45 #13 0x000000000136cd38 in boost::archive::detail::interface_oarchive<boost::archive::binary_oarchive>::operator<< <FilterCondition* const> (this=0x7fbfffc410, t=@0x19c6d820) at /remote/arch-tools/boost/qsc-b/1_35_0/include/boost/archive/detail/interface_oarchive.hpp:64 #14 0x000000000136cd69 in boost::archive::detail::interface_oarchive<boost::archive::binary_oarchive>::operator&<FilterCondition*> (this=0x7fbfffc410, t=@0x19c6d820) at /remote/arch-tools/boost/qsc-b/1_35_0/include/boost/archive/detail/interface_oarchive.hpp:72 #15 0x000000000136ce5a in Table::serialize<boost::archive::binary_oarchive> (this=0x19c6d750, ar=@0x7fbfffc410, version=0) at /remote/scratch/gsoumen/hiers/tmp.trunk/tools/code/tables/Table.h:178 #16 0x000000000136ce7f in boost::serialization::access::serialize<boost::archive::binary_oarchive, Table> (ar=@0x7fbfffc410, t=@0x19c6d750, file_version=0) at /remote/arch-tools/boost/qsc-b/1_35_0/include/boost/serialization/access.hpp:109 ------------------------------------------------------------------------------------------------ Is this a bug in serialization library? Regards, ~ Soumen -- View this message in context: http://www.nabble.com/-Serialization---1.35--Crash-while-serializing-tp2553… Sent from the Boost - Users mailing list archive at Nabble.com.

3 3

Boost Program Options: Just curiosity, why double dashes ?
by Mauricio Gomes 01 Dec '09

01 Dec '09

I am studying Boost Program Options library and I am curious, why was it choosen double dashes as the token to identify an option in the command line ? Example: my_program --help Would not a single dash be enough and less verbose ? What is the rationale behind that ? Thanks, -- Mauricio Gomes "Think globally, act locally".

5 4

Mac OSX - gcc crash
by Igor R 01 Dec '09

01 Dec '09

Hello, My question is not focused, but maybe someone had similar experience and could share his conslucions. I'm porting to iPhone (SDK 3.1.2, XCode3.2.1) rather big static library that uses boost very extensively. It compiles and works well under the simulator both in debug and in release modes. But when I try to compile it for the device, gcc4.2 crashes ("exists with code 1"). I found-out that if I do not generate debug symbols, the compilation succeeds. However, then I can't debug the application on iPhone - it doesn't stop on breakpoints and I can't step through the code (if the application would work on the device like it works on simulator, debugging wouldn't be an issue, but it doens't work for some reason). Again, I even don't know *where* the compiler crashes and whether it's related to boost headers or not, but any idea/experience would be greatly appreciated. Thanks.

3 3

serialization library leaks memory in its global object
by gast128 01 Dec '09

01 Dec '09

Dear all, we were transferring to Boost 1.41 but came across some serious issues in Boost.Serialization. The first is already adressed in a previous post, but now the M$ crt reports a memory leak when using Boost.Serialization in combination of pointers. I think it is located in 'basic_iarchive_impl::load_pointer' where an object is allocated and added to a global object ('object_id_vector') but never released. I know that we might end up here in an argument about the defintion of a memory leak, but the fact remains that the crt just reports it on program closure and until now we never had any reports of that type. If we have to think about every report dumped by the crt, it's just unacceptable. It's like a broken window syndrome; in the end u can not distinguish between false and correct reports. Code: #include <sstream> #include <boost\smart_ptr\make_shared.hpp> #include <boost\archive\xml_oarchive.hpp> #include <boost\archive\xml_iarchive.hpp> struct Foo { template <class Archive> void serialize(Archive& /*ar*/, const unsigned int /*version*/) { } }; int main() { #ifdef _DEBUG _CrtSetDbgFlag(_CRTDBG_ALLOC_MEM_DF | _CRTDBG_CHECK_ALWAYS_DF | _CRTDBG_LEAK_CHECK_DF); #endif std::stringstream sstr; boost::shared_ptr<Foo> ptrFoo = boost::make_shared<Foo>(); { boost::archive::xml_oarchive oa(sstr); oa << boost::serialization::make_nvp("v", ptrFoo); } boost::archive::xml_iarchive ia(sstr); ia >> boost::serialization::make_nvp("v", ptrFoo); return 0; }

3 5