[outcome] Inline GDB pretty printing support
Dear Boost, The biggest new feature to land into Outcome since it went ABI stable is inline GDB pretty printing support, whereby we embed into all binaries using Outcome the Python script to display the state of `result<>/outcome<>`. As Outcome uses a layered inheritance technique to match triviality of member functions from its configured types, having to click down through the layers to discover the state was always frustrating. MSVC has long shipped with a visualiser to eliminate this, so now support for GDB matches MSVC support. The relevant implementation file can be found at: https://github.com/boostorg/outcome/blob/master/include/boost/outcome/outcom... I would appreciate any testing and reporting of any issues or bugs before the 1.86 bugfix window closes. It should "just work" once you have enabled inline script loading in your `.gdbinit`. Thanks, Niall
On Mon, Jul 1, 2024 at 7:45 AM Niall Douglas via Boost < boost@lists.boost.org> wrote:
inline GDB pretty printing support, whereby we embed into all binaries using Outcome the Python script to display the state of `result<>/outcome<>`.
Huh...I don't quite understand what this means. Although from the source file it looks like you are embedding Python code in the library? Can you please explain this a little better, or provide some links to reading materials where this is explained? Visualizer support is an ongoing area of interest for my libraries. Thanks
On 01/07/2024 15:48, Vinnie Falco wrote:
On Mon, Jul 1, 2024 at 7:45 AM Niall Douglas via Boost
mailto:boost@lists.boost.org> wrote: inline GDB pretty printing support, whereby we embed into all binaries using Outcome the Python script to display the state of `result<>/outcome<>`.
Huh...I don't quite understand what this means. Although from the source file it looks like you are embedding Python code in the library? Can you please explain this a little better, or provide some links to reading materials where this is explained? Visualizer support is an ongoing area of interest for my libraries.
Sure! As you already know, you can write a Python helper which GDB loads to better render types in GDB. This is a great facility. Most docs online say you have to manually patch in visualisers to your `.gdbinit`, and indeed that is one technique and that works well. It is kinda bothersome though, and tedious and it's easy to forget you've done it and then the code gets changed and the visualiser doesn't and then it gets annoying. A less well known technique I've had in the Outcome todo list since I first wrote Outcome is you can embed the Python script inside a magical ELF section which causes it to appear in all binaries which include your library. This magical ELF section will be autoloaded by GDB if your `.gdbinit` has whitelisted the paths where your binaries are. If you haven't whitelisted it, you get a useful diagnostic from GDB hinting you should enable the whitelist. You can of course whitelist all paths in your `.gdbinit`, and then all auto loaded inline GDB visualisers "just work". This is very convenient. In terms of public docs about this, there is a terse section in the GDB docs at https://sourceware.org/gdb/current/onlinedocs/gdb.html/dotdebug_005fgdb_005f.... What had blocked me until now was the appropriate asm incantation to get the sections to auto merge in a header only library and for it to "just work". I finally created the time to get this working right a few weeks ago out of frustration with working with Outcome in GDB, and I'm very glad it's done. My current day job has me mainly working on a C codebase. Many don't realise Outcome has a C interface, due to C being C it is highly type erased so having a GDB visualiser which prints the contents of the state is very very useful in my current day job right now. Feel free to lift ideas from my linked file into your code. I agree more inline embedded visualisers in Boost would be a great help. Niall
пн, 1 июл. 2024 г. в 17:45, Niall Douglas via Boost
Dear Boost,
The biggest new feature to land into Outcome since it went ABI stable is inline GDB pretty printing support, whereby we embed into all binaries using Outcome the Python script to display the state of `result<>/outcome<>`. As Outcome uses a layered inheritance technique to match triviality of member functions from its configured types, having to click down through the layers to discover the state was always frustrating. MSVC has long shipped with a visualiser to eliminate this, so now support for GDB matches MSVC support.
The relevant implementation file can be found at:
https://github.com/boostorg/outcome/blob/master/include/boost/outcome/outcom...
This is a very neat trick. I have a WIP branch with GDB printers for JSON (https://github.com/boostorg/json/pull/733). The branch hasn't been merged yet mostly because I couldn't decide on the best way to deploy the script. And this trick trivializes that problem
On 01/07/2024 16:16, Дмитрий Архипов via Boost wrote:
пн, 1 июл. 2024 г. в 17:45, Niall Douglas via Boost
: Dear Boost,
The biggest new feature to land into Outcome since it went ABI stable is inline GDB pretty printing support, whereby we embed into all binaries using Outcome the Python script to display the state of `result<>/outcome<>`. As Outcome uses a layered inheritance technique to match triviality of member functions from its configured types, having to click down through the layers to discover the state was always frustrating. MSVC has long shipped with a visualiser to eliminate this, so now support for GDB matches MSVC support.
The relevant implementation file can be found at:
https://github.com/boostorg/outcome/blob/master/include/boost/outcome/outcom...
This is a very neat trick. I have a WIP branch with GDB printers for JSON (https://github.com/boostorg/json/pull/733). The branch hasn't been merged yet mostly because I couldn't decide on the best way to deploy the script. And this trick trivializes that problem
Sorry it's several years late. I had known this was possible for many years, the lack of it had to become painful enough to my work productivity that I could justify using work time to fix it. When the current contract ends I am hoping to reduce how much I work for money in order to get more time to do other useful stuff I've backlogged. If I can afford to do that when the time comes, I shall. Obviously if anybody finds any ways my solution above could be improved before the 1.86 window closes, I am all ears. Niall
1 juillet 2024 à 16:45 "Niall Douglas via Boost"
Dear Boost,
The biggest new feature to land into Outcome since it went ABI stable is inline GDB pretty printing support, whereby we embed into all binaries using Outcome the Python script to display the state of `result<>/outcome<>`. As Outcome uses a layered inheritance technique to match triviality of member functions from its configured types, having to click down through the layers to discover the state was always frustrating. MSVC has long shipped with a visualiser to eliminate this, so now support for GDB matches MSVC support.
The relevant implementation file can be found at:
https://github.com/boostorg/outcome/blob/master/include/boost/outcome/outcom...
That is a really neat trick. I've been playing around with cmake post build step to achieve similar results, but this is a much nicer solution for library code. Many thanks for sharing this. Regards, Julien
Am 01.07.24 um 16:45 schrieb Niall Douglas via Boost:
The relevant implementation file can be found at:
https://github.com/boostorg/outcome/blob/master/include/boost/outcome/outcom...
__asm__(".pushsection \".debug_gdb_scripts\", \"MS\",@progbits,1\n" ".byte 4 /* Python Text */\n" ".ascii \"gdb.inlined-script\\n\"\n" ".ascii \"import gdb.printing\\n\"\n" ".ascii \"import os\\n\"\n");
That is really neat indeed! Only thing bothering me is the abundant quoting and quotes. Maybe it looks a bit better with raw string literals. I.e. instead of: this equivalent
__asm__(R"py(.pushsection ".debug_gdb_scripts", "MS",@progbits,1 .byte 4 /* Python Text */ .ascii "gdb.inlined-script\n" .ascii "import gdb.printing\n" .ascii "import os\n" )py"); It looses the indentation though (otherwise it would be contained in the binary. I also don't see a way to avoid the '.ascii "' and '\n"' boilerplate and just paste the python script but I think it more closely resembles it.
Alex
On Tue, 2 Jul 2024, Alexander Grund via Boost wrote:
Am 01.07.24 um 16:45 schrieb Niall Douglas via Boost:
The relevant implementation file can be found at:
https://github.com/boostorg/outcome/blob/master/include/boost/outcome/outcom...
That is really neat indeed!
Only thing bothering me is the abundant quoting and quotes.
There is an outcome_gdb.py next to it, so I assume the .h file is generated automatically, you don't need to work on it directly. -- Marc Glisse
On 7/1/24 17:45, Niall Douglas via Boost wrote:
Dear Boost,
The biggest new feature to land into Outcome since it went ABI stable is inline GDB pretty printing support, whereby we embed into all binaries using Outcome the Python script to display the state of `result<>/outcome<>`. As Outcome uses a layered inheritance technique to match triviality of member functions from its configured types, having to click down through the layers to discover the state was always frustrating. MSVC has long shipped with a visualiser to eliminate this, so now support for GDB matches MSVC support.
The relevant implementation file can be found at:
https://github.com/boostorg/outcome/blob/master/include/boost/outcome/outcom...
I would appreciate any testing and reporting of any issues or bugs before the 1.86 bugfix window closes. It should "just work" once you have enabled inline script loading in your `.gdbinit`.
Is there a benefit to embedding python comments into the script? Since the embedded script itself presumably won't be read by a human, the comments are just unnecessary bloat to the binary size. You could preserve the comments in C++ though.
On 7/2/24 13:52, Andrey Semashev wrote:
On 7/1/24 17:45, Niall Douglas via Boost wrote:
Dear Boost,
The biggest new feature to land into Outcome since it went ABI stable is inline GDB pretty printing support, whereby we embed into all binaries using Outcome the Python script to display the state of `result<>/outcome<>`. As Outcome uses a layered inheritance technique to match triviality of member functions from its configured types, having to click down through the layers to discover the state was always frustrating. MSVC has long shipped with a visualiser to eliminate this, so now support for GDB matches MSVC support.
The relevant implementation file can be found at:
https://github.com/boostorg/outcome/blob/master/include/boost/outcome/outcom...
I would appreciate any testing and reporting of any issues or bugs before the 1.86 bugfix window closes. It should "just work" once you have enabled inline script loading in your `.gdbinit`.
Is there a benefit to embedding python comments into the script? Since the embedded script itself presumably won't be read by a human, the comments are just unnecessary bloat to the binary size. You could preserve the comments in C++ though.
Also, on the topic of binary size, is this script embedded into the final binary only once, even if the header is included in multiple TUs?
2 juillet 2024 à 12:58 "Andrey Semashev via Boost"
Also, on the topic of binary size, is this script embedded into the final binary only once, even if the header is included in multiple TUs?
That (binary size) was also one of my concern, i checked the following : * the whole section is correctly stripped when stripping debug symbols (and correctly preserved in the external debug symbols file) * multiple sections works correctly * the section is indeed only added once in the final binary (for what it's worth, tests done with gcc x64, gcc cross-compiling to arm32 and clang x64, no issues with any of these configurations) Regards, Julien
On 02/07/2024 13:09, Julien Blanc via Boost wrote:
2 juillet 2024 à 12:58 "Andrey Semashev via Boost"
a écrit: Also, on the topic of binary size, is this script embedded into the final binary only once, even if the header is included in multiple TUs?
That (binary size) was also one of my concern, i checked the following :
* the whole section is correctly stripped when stripping debug symbols (and correctly preserved in the external debug symbols file) * multiple sections works correctly * the section is indeed only added once in the final binary
(for what it's worth, tests done with gcc x64, gcc cross-compiling to arm32 and clang x64, no issues with any of these configurations)
Note that linkers don't have to honour the merge section request if they don't want to. Furthermore, some may ignore merging sections of null terminated strings, like these. The merging algorithm which GNU ld seems to use is stick all the sections into a hash table keyed by the hash of the section contents, thus eliminating all exact duplicates. It then emits an unpredictable sequence of appended strings. You therefore need to be careful that any Python you write can be safely #included with any other Python in any arbitrary order. If there is any difference at all, that causes a copy to be emitted, so if you ever change the string and a user mixes binary versions, you'll get two of the visualisers or more. You need to write your visualiser to cope with that, too. I didn't test other linkers apart from ld well, though from first glance GNU gold appears to do the right thing. I didn't try other linkers. Niall
Great stuff Niall. It just so happens that someone has been collecting pretty-printers for various Boost libraries for some time; you can find them here: https://github.com/ruediger/Boost-Pretty-Printer Creating a formal framework for them to be included with Boost itself would be pretty cool. Jeff
On Wed, Aug 7, 2024 at 4:04 PM Jeff Trull via Boost
Creating a formal framework for them to be included with Boost itself would be pretty cool.
We could add a section "Debugger Visualizers" to the Contributor's Guide. At first it can just list some resources, and over time we can add best practices and examples for how authors and maintainers may add support to their libraries. Feel free to add to this open issue: https://github.com/boostorg/website-v2-docs/issues/278 Thanks
On 08/08/2024 17:56, Vinnie Falco via Boost wrote:
On Wed, Aug 7, 2024 at 4:04 PM Jeff Trull via Boost
wrote: Creating a formal framework for them to be included with Boost itself would be pretty cool.
We could add a section "Debugger Visualizers" to the Contributor's Guide. At first it can just list some resources, and over time we can add best practices and examples for how authors and maintainers may add support to their libraries.
I would much rather that the maintainer of the library inlines the visualisers for their library, and then maintains them over time as their library changes. To make this work well, what we really need and don't have is a visualiser testing framework i.e. unit tests will fail if your visualiser stops matching your C++. I fell foul of this exact problem: https://github.com/ned14/outcome/issues/301 I suspect that another copy of the visualiser overrode the inline one, and so I didn't see this failure until it was too late (master is now locked). What would be super great (hint: C++ Alliance funded) is if Boost had convenient tooling which in both b2 and cmake invoked the gdb/MSVC debugger during unit tests and checked that a corpus of library use snippets do produce the correct visualisations in the debugger. It's a chunky enough bit of work, highly unlikely to be done in volunteer time as it'll be tedious and boring and annoying. Niall
When I wrote the Natvis visualizers for Unordered, I was looking into
automated unit testing and couldn't figure out a way to get that to work.
In the end I settled on a test file with a specific location to break, then
check the debugger window manually. If there's anything out there in terms
of automated unit testing for Natvis, I want to know.
Similarly, I'm working on pretty-printers for Unordered at the moment, and
I would love some automated testing there as well. (Incidentally, I may
need to reach out privately for some help on the inline asm stuff.)
Mainly, I'm sending this message to also express interest in a visualizer
test framework. I think visualizers are a very important part of library
authorship, and it would encourage more people to write visualizers if the
testing was more rigorous and easier to hook into.
On Thu, Aug 8, 2024, 12:36 PM Niall Douglas via Boost
On 08/08/2024 17:56, Vinnie Falco via Boost wrote:
On Wed, Aug 7, 2024 at 4:04 PM Jeff Trull via Boost < boost@lists.boost.org> wrote:
Creating a formal framework for them to be included with Boost itself would be pretty cool.
We could add a section "Debugger Visualizers" to the Contributor's Guide. At first it can just list some resources, and over time we can add best practices and examples for how authors and maintainers may add support to their libraries.
I would much rather that the maintainer of the library inlines the visualisers for their library, and then maintains them over time as their library changes.
To make this work well, what we really need and don't have is a visualiser testing framework i.e. unit tests will fail if your visualiser stops matching your C++.
I fell foul of this exact problem:
https://github.com/ned14/outcome/issues/301
I suspect that another copy of the visualiser overrode the inline one, and so I didn't see this failure until it was too late (master is now locked).
What would be super great (hint: C++ Alliance funded) is if Boost had convenient tooling which in both b2 and cmake invoked the gdb/MSVC debugger during unit tests and checked that a corpus of library use snippets do produce the correct visualisations in the debugger.
It's a chunky enough bit of work, highly unlikely to be done in volunteer time as it'll be tedious and boring and annoying.
Niall
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Thu, Aug 8, 2024 at 1:09 PM Braden Ganetsky via Boost
If there's anything out there in terms of automated unit testing for Natvis, I want to know.
I'm not aware of such (haven't looked). But I did write some basic GDB testing for my <debugging> wg21 implementation using specially constructed code and awk regex matching.. https://github.com/grafikrobot/debugging/tree/main/tests - that might inspire someone to apply it for visualizers. -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supone Nada -- Robot Dreams - http://robot-dreams.net
On Thu, Aug 8, 2024 at 11:30 AM René Ferdinand Rivera Morell via Boost < boost@lists.boost.org> wrote:
https://github.com/grafikrobot/debugging/tree/main/tests - that might inspire someone to apply it for visualizers.
Please comment in this issue :) https://github.com/boostorg/website-v2-docs/issues/278 Thanks
On Thu, Aug 8, 2024 at 10:36 AM Niall Douglas via Boost < boost@lists.boost.org> wrote:
I would much rather...
Yes, authors would still be responsible for the visualizers. My point is that the Contributor's Guide is a good place to offer knowledge and rationale for implementing visualizers to help authors along. Thanks
пт, 9 авг. 2024 г. в 03:36, Niall Douglas via Boost
On Wed, Aug 7, 2024 at 4:04 PM Jeff Trull via Boost
wrote: I would much rather that the maintainer of the library inlines the visualisers for their library, and then maintains them over time as On 08/08/2024 17:56, Vinnie Falco via Boost wrote: their library changes.
To make this work well, what we really need and don't have is a visualiser testing framework i.e. unit tests will fail if your visualiser stops matching your C++.
In my WIP GDB pretty printer branch I have the basis for GDB pretty printers tests. The "framework" consists of 1) an executable that creates objects to print and has labels at which to print (https://github.com/boostorg/json/pull/733/files#diff-61c3f2894b149b936ad8386...), 2) a Python script that makes GDB jump to labels and print expressions (https://github.com/boostorg/json/pull/733/files#diff-61c3f2894b149b936ad8386...), 3) b2 script to make that work from b2 (https://github.com/boostorg/json/pull/733/files#diff-2cde2bed13578c1217b5047...). This in theory could be turned into something more reusable.
participants (10)
-
Alexander Grund
-
Andrey Semashev
-
Braden Ganetsky
-
Jeff Trull
-
Julien Blanc
-
Marc Glisse
-
Niall Douglas
-
René Ferdinand Rivera Morell
-
Vinnie Falco
-
Дмитрий Архипов