Re: [boost] TSAN SEGFAULT in Boost.Beast CI

If these CI tests are running under a container, could you please link to that containerfile? I have noticed that clang’s tsan in particular behaves differently on different operating systems, even with the same compiler, sysroots and stdlib. WL

On Aug 13, 2022, at 10:53 AM, Richard Hodges wrote:



...

On Sat, 13 Aug 2022 at 16:48, William Linkmeyer wrote: That’s interesting. I don’t see any reason why that lock would fail. Clang’s analyzers do come up with false positives sometimes.

If I can get around to it this weekend, I’ll run this under valgrind. If that comes back positive, I’ll run it under a couple different compilers and their default standard libraries; circle, clang, apple clang, msvc. (This tsan looks like it’s using gcc‘s non-extended stdlib.)

Thank you William. Your support is greatly appreciated. For what it's worth, our CI does run tests under valgrind, and they pass. An example run can be found here: https://drone.cpp.al/boostorg/beast/326/11/2

...

But it’ll be a busy weekend, I’m trying to learn Haskell and put together a pet project. So, if anyone has the time, the notes above would be what I’d do.

WL

...

On Aug 12, 2022, at 2:25 PM, Richard Hodges via Boost wrote:

Dear all,

You may have noticed that one of the Beast CI checks fails on the master and develop branches.

The failure is a segfault in thread sanitizer. Running on my local machine I get the stack trace embedded below.

This does not happen if Asio is reverted back to version 1.79.0

Chris has confirmed that the change in Asio that caused this is in asio::spawn.

Note that at this stage there is no reason to suspect that Asio is actually at fault. I believe it to be a false positive. However, I am an absolute noob when it comes to the Dark Arts of thread sanitizer. I have no idea what to blacklist.

This email has two aims:

1. To prevent multiple people wasting their time trying to get to the bottom of this, 2. To encourage some sage TSAN-maester to emerge who may guide me to enlightenment.

Here is the error and stack trace:

testing.capture-output bin.v2/libs/beast/test/beast/http/read.test/gcc-12/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/read.run ====== BEGIN OUTPUT ====== beast.http.read ThreadSanitizer:DEADLYSIGNAL ==1761540==ERROR: ThreadSanitizer: SEGV on unknown address 0x7f6ea78ff000 (pc 0x7f6eaaaba0d0 bp 0x000000000000 sp 0x7f6ea783d910 T1761542) ==1761540==The signal is caused by a READ memory access. #0 __sanitizer::StackDepotBase<__sanitizer::StackDepotNode, 1, 20>::Put(__sanitizer::StackTrace, bool*) <null> (libtsan.so.2+0xba0d0) #1 __tsan::CurrentStackId(__tsan::ThreadState*, unsigned long) <null> (libtsan.so.2+0x8c48f) #2 __sanitizer::DD::MutexInit(__sanitizer::DDCallback*, __sanitizer::DDMutex*) <null> (libtsan.so.2+0xac534) #3 __tsan::DDMutexInit(__tsan::ThreadState*, unsigned long, __tsan::SyncVar*) <null> (libtsan.so.2+0x9a3f8) #4 __tsan::MetaMap::GetSync(__tsan::ThreadState*, unsigned long, unsigned long, bool, bool) <null> (libtsan.so.2+0xa85dc) #5 __tsan_atomic32_fetch_add <null> (libtsan.so.2+0x783e9) #6 __gnu_cxx::__exchange_and_add(int volatile*, int) /usr/include/c++/12/ext/atomicity.h:66 (read+0x41ad46) #7 __gnu_cxx::__exchange_and_add_dispatch(int*, int) /usr/include/c++/12/ext/atomicity.h:101 (read+0x41ad46) #8 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/include/c++/12/bits/shared_ptr_base.h:187 (read+0x41ad46) #9 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/12/bits/shared_ptr_base.h:361 (read+0x40d87c) #10 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/include/c++/12/bits/shared_ptr_base.h:1071 (read+0x41b43c) #11 std::__shared_ptr::~__shared_ptr() /usr/include/c++/12/bits/shared_ptr_base.h:1524 (read+0x424db7) #12 std::shared_ptrboost::asio::detail::strand_executor_service::strand_impl::~shared_ptr() /usr/include/c++/12/bits/shared_ptr.h:175 (read+0x424de3) #13 boost::asio::detail::strand_executor_service::invoker const, void>::~invoker() <null> (read+0x486f4d) #14 boost::asio::detail::executor_op const, void>, boost::asio::detail::recycling_allocator, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) <null> (read+0x49ff36) #15 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) boost/asio/detail/scheduler_operation.hpp:40 (read+0x513742) #16 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) boost/asio/detail/impl/scheduler.ipp:492 (read+0x501be9) #17 boost::asio::detail::scheduler::run(boost::system::error_code&) boost/asio/detail/impl/scheduler.ipp:210 (read+0x5008af) #18 boost::asio::io_context::run() boost/asio/impl/io_context.ipp:63 (read+0x4f54d6) #19 boost::beast::test::enable_yield_to::enable_yield_to(unsigned long)::{lambda()#1}::operator()() const <null> (read+0x41480d) #20 void std::__invoke_impl(std::__invoke_other, boost::beast::test::enable_yield_to::enable_yield_to(unsigned long)::{lambda()#1}&&) <null> (read+0x4bb4bc) #21 std::__invoke_result::type std::__invoke(boost::beast::test::enable_yield_to::enable_yield_to(unsigned long)::{lambda()#1}&&) <null> (read+0x4b8c78) #22 void std::thread::_Invoker >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (read+0x4b5b00) #23 std::thread::_Invoker >::operator()() <null> (read+0x4b1f74) #24 std::thread::_State_impl > >::_M_run() <null> (read+0x4ab02e) #25 execute_native_thread_routine <null> (libstdc++.so.6+0xdbb72) #26 __tsan_thread_start_func <null> (libtsan.so.2+0x393ef) #27 start_thread <null> (libc.so.6+0x8ce2c) #28 clone3 <null> (libc.so.6+0x1121af)

ThreadSanitizer can not provide additional info. SUMMARY: ThreadSanitizer: SEGV (/lib64/libtsan.so.2+0xba0d0) in __sanitizer::StackDepotBase<__sanitizer::StackDepotNode, 1, 20>::Put(__sanitizer::StackTrace, bool*) ==1761540==ABORTING

EXIT STATUS: 66 ====== END OUTPUT ======

-- Richard Hodges hodges.r@gmail.com office: +44 2032 898 513 home: +376 861 195 mobile: +376 380 212

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost