Hi,

Linux, Solaris and Mac OS X have functions backtrace and backtrace_symbols in libc: See: http://linux.die.net/man/3/backtrace

Windows starting from XP has RtlCaptureBackTrace in kernel32.dll: See http://msdn.microsoft.com/en-us/library/ff552119(VS.85).aspx

So the rest is translate symbols, under Unixes I have dladdr and under Windows dbghelp.

One recommendation, decide early on limitations. I recommend to choose from the beginning: - whether it should work in asynchronous interrupts such as signal handlers, - whether it should work when the program is unstable, such as memory fault handlers, and - how good a performance to expect. At least the linux variety of backtrace() has limitations in all the above areas. Getting an occasional stack trace in application execution flow, say debug output or throwing an exception is definitely ok. The function works well enough in many common crash situations, but will fail in somewhat unpleasant ways in certain circumstances. Regards, Lassi

Hi,

Very Good idea:

Limitations of use:

1. No it is not asynchronous interrupts safe 2. It would not likely work when the application is unstable or as serious issues 3. The trace collection is expected to be very fast (as walking on linked list) 4. The trace printing may be quite slow and costly, expected to be used quite rarely.

Good :-) FWIW, backtrace() on recent'ish linux is fairly heavyweight. It's definitely not a frame pointer walk as your 3 above seems to suggest. For example on most x86_64 systems there's no frame pointer chain to walk to begin with, so it's full blown DWARF unwind - which among other things can allocate memory. Recent x86 GLIBC / linux will also do the full unwind on 32-bit, not just a simple frame pointer walk. For a few random stack walks it's very acceptable, but you wouldn't want to call it thousands of times a second on deep stacks with lots of shared libraries. Exactly where the performance becomes an issue you'd have to benchmark and decide for yourself.

Notes:

- Generally trace collection should be interrupt safe, but it is up to the specific implementation of libc backtrace function on specific platform.

I doubt any libc backtrace() which uses unwind info is async safe. The only async-safe unwind library I know is libunwind.

- It is possible to create degraded print tracing in case the process is unstable such, it would be safe to use in signal handlers, but it is unlikely to provide full information. Only pointers data and not symbols.

Of course the very act of printing gets complicated inside signal handlers and/or when the app is unstable. For one, you can't really allocate memory, so things like iostreams, std:strings and other high-level interfaces (including __cxa_demangle) are out. You are pretty much reduced to making system calls only (e.g. unix write()). Calls to dladdr() are not async signal safe, but they are mostly ok in a crash. As I noted, most backtrace() implementations are sufficiently complex to sometimes fail when the application has become unstable. Fortunately the failures will be rare, so the tool can still be useful most of the time. Unfortunately the failure modes can be nasty, either recursive signals (crash handlers doing stack dumps should have a static counter and abandon after N recursive signals), or dead-locks (no fix that I know of, not even any way to know if it's unsafe to call - you'll just be unlucky). Where you draw the line on "useful even if not perfect" is really up to you and why I suggest to specify the limitations above, as you did.

Limitations:

- Under ELF platform - system should be compiled with -rdynamic - Under Windows/MSVC - debug information should be provided. - Function inlining and omitting frame pointers would significantly reduce visible frames of calls. - Static function would not be resolved under ELF platforms.

Bottom line:

- It can't be as powerful as Java's printStackTrace as C/C++ generally has fewer runtime information - It can be very useful with top level catch blocks to see what happen.

Artyom

P.S.: The implementation exists take a look on it and see if it useful enough P.P.S.: I don't think the implementation should include much more then now.

Sure, I did. I'm not really a client for it, just making recommendations from experience. FWIW, I regularly use apps that dump a stack trace on themselves on crash. One of the most annoying failure modes is... the stack printing getting out of control, creating a fork bomb, or dead-locking forever leaving hanging processes on the system. My experience is that it's relatively easy to make a stack dump that works often - and really quite hard to make one that Really Just Works(tm). It's surprisingly annoying to end up with side effects from the stack tracer itself in one run in 100, or 200, or even 1000. It's really up to you where on that continuum you want to position your library; pretty much no matter what you choose it will still be useful for some people :-) Regards, Lassi

On 10/21/2010 06:40 AM, Artyom wrote:

Hi All,

...

One recommendation, decide early on limitations.

I recommend to choose from the beginning: - whether it should work in asynchronous interrupts such as signal handlers, - whether it should work when the program is unstable, such as memory fault handlers, and - how good a performance to expect. Very Good idea:

Limitations of use:

1. No it is not asynchronous interrupts safe 2. It would not likely work when the application is unstable or as serious issues 3. The trace collection is expected to be very fast (as walking on linked list) 4. The trace printing may be quite slow and costly, expected to be used quite rarely.

Will the trace printing function use information provided by debuginfo packages on Linux? Rob

On 10/22/2010 09:51 AM, Artyom wrote:

...

Will the trace printing function use information provided by debuginfo packages on Linux?

No using ELF data - the symbols available in. My understanding is that the software distributed by RPM these days (since circa 2003) default to having all of the symbol information stripped and distributed in a separate debuginfo package.

To wit: nm /bin/ls nm: /bin/ls: no symbols I believe that to get any sort of stack trace out of /bin/ls one would need to first load the symbols from the coreutils-debuginfo package. Rob

On 10/23/2010 03:40 AM, Artyom wrote:

...

...

No using ELF data - the symbols available in. My understanding is that the software distributed by RPM these days (since circa 2003) default to having all of the symbol information stripped and distributed in a separate debuginfo package.

To wit:

nm /bin/ls nm: /bin/ls: no symbols

I believe that to get any sort of stack trace out of /bin/ls one would need to first load the symbols from the coreutils-debuginfo package.

Rob

I'm not talking about debug symbols but rather function entry points defined in any shared object:

$ nm /lib/libc.so.6 nm: /lib/libc.so.6: no symbols

$ nm -D /lib/libc.so.6 ... 00000000000864b0 W wcstod_l 0000000000083770 T wcstof 000000000008ace0 W wcstof_l 0000000000040810 T wcstoimax 0000000000081e80 T wcstok 00000000000836b0 T wcstol 0000000000083bf0 W wcstol_l ...

All you need is to compile executables with -rdynamic

Ah. I understand now. But does this mean that the symbols (functions) that are already resolved within my application will not have symbolic names available for the backtrace? My stripped executables only seem to have the undefined symbolic names present in the executable. Rob

On 10/21/2010 1:10 AM, Artyom wrote:

...

My experience is that the stack is difficult to follow in debug builds and close to impossible in release builds. Highly optimized code sometimes discards the notion of a stack altogether in certain cases.

I'm curious how you're going to accomplish this feat and make it portable

too?

...

If you are successful, this would definitely be a valuable tool to have.

Because I'm not trying to reinvent the wheel:

Linux, Solaris and Mac OS X have functions backtrace and backtrace_symbols in libc: See: http://linux.die.net/man/3/backtrace

Windows starting from XP has RtlCaptureBackTrace in kernel32.dll: See http://msdn.microsoft.com/en-us/library/ff552119(VS.85).aspx

So the rest is translate symbols, under Unixes I have dladdr and under Windows dbghelp.

Of course for inlined functions or for functions with omitted frame pointer you'll not see their frame it the trace, but for rest of it, it should work.

Artyom

This functionality would definitely be appreciated. Beware though, that gcc folks made a controversial decision to set "no frame pointer" flag as a default on 64bit POSIX. That also means that backtrace() will be either very inefficient (in my tests it was more than 100x slower than when the frame pointers are included) or unstable. It actually crashes sometimes. That makes backtrace() pretty much useless for the code that was compiled with the default flags. (which is most of the shared system libraries). And even if the user adds (-with-frame-pointer), the start of the trace might be in the system libraries, so to make backtrace() work in that case you have to somehow limit it only to the user's code. Scary. Andy.

Hi,

...

or unstable. It actually crashes sometimes. [...] Scary.

Can you give me more information about it? Do you have any references? Because if backtrace crashes I would likely just not use this feature and this library would not be actually created.

As I mentioned before, there are uses for which backtrace() is perfectly valid choice and will result in a fine user experience. There are limitations and you need to decide if your library is fine with those limitations. It will almost certainly be useful to some set of users - and almost certainly will not work reliably for some users. Such is life :-) We have applications which perform up to about 250k stack walks a second, with average stack depth of 33 and maybe 700 loaded shared libraries. Max stack depth is maybe 200-300. We'd do million walks per second if we found a library capable of doing it. So fairly heavy use. I can't really quote a lot of specific examples on how linux' backtrace() fails, mostly because we moved to something else fairly early on as linux' backtrace() wasn't at all viable - for our use. From what I recall the main limitations on backtrace() were that a) it could be very slow on a large binary, meaning a single stack trace could exceed 5ms on a reasonably modern x86_64 system; b) it can call malloc so unsafe to use inside malloc itself; c) it makes other calls which are unsafe when inside an asynchronous signal handler (result: deadlocks and crashes); d) for the same reasons, it's not re-entrant _in the same thread_, so problematic if you are unlucky enough to get nested signals and try to call backtrace() again while one was already running (NB: in the same thread!). About 10% of our x86_64 stack walks hit incorrect or inaccurate unwind info or various creative corner cases on 64-bit RHEL5-derived system. The issues split into maybe 5-10 different categories. I wouldn't know which ones backtrace() is protected from, but my experience seemed to indicate it probably is largely unprotected. Which is fine - as long as you don't trigger a stack walk that requires those corner cases to be handled! If you limit yourself to calling backtrace() only inside the program flow, not in signal handlers, not in instrumentation the compiler wasn't aware of, and not from the global constructors / destructors, and you don't require high performance, it should work just fine for you. Note that backtrace() implementation has varied significantly over time and platforms. If you are doing benchmarks, make sure you test enough variety of systems. Something old like RHEL4 or RHEL5 may do something different than newer systems like say a recent Ubuntu. I don't know much about backtrace() on non-linux systems so can't speak for them. Regards, Lassi

I am very interested, but when I attempted to link your code I got the following error:

Undefined symbols: "boost::stack_trace::write_symbols(void* const*, int, std::basic_ostream<char, std::char_traits<char> >&)", referenced from: boost::backtrace::trace(std::basic_ostream<char, std::char_traits<char>

...

&) const in libfl_scheduler-1.a(scheduler.cpp.o)

What is the platform/compiler do you try to compile on? Because each ifdef provide write_symbols. Are you sure you compile and link your code correctly?

In the code you provided I saw backtrace.hpp and backtrace.cpp, neither one provides an implementation for boost::track_trace::write_symbols, what am I missing?

How can this be? It implemented in backtrace.cpp at lines - line 123 for Linux/Solaris/Mac using GCC - line 174 for Linux/Solaris/Mac using other compilers - line 253 for MSVC/Windows - line 286 for other generic trace Artyom