New subject: A question for folks here

10 Jul 2024

      Hi all,

My name is Kenneth Reitz, and I got involved with Boost when Vinnie Falco asked me to join the C++ Alliance to work on the new boost website. 

You might recognize my name from the Python community, where I’m known for making Requests, Pipenv, and other tools. 

It’s nice to meet everyone!

—
Kenneth Reitz
...
1. A question for folks here (David Sankel)
  2. Re: A secure string library? (Andrey Semashev)
  3. Re: A secure string library? (Andrey Semashev)
  4. Re: A question for folks here (Richard Hodges)
  5. Re: A secure string library? (Daniela Engert)
  6. Re: A secure string library? (Andrey Semashev)
  7. Re: A secure string library? (Christian Mazakas)
  8. Re: A question for folks here (Matt Borland)
  9. Re: A question for folks here (Christopher Kormanyos)
 10. Re: A question for folks here (Louis Tatta)
----------------------------------------------------------------------
Message: 1
Date: Wed, 10 Jul 2024 10:47:50 -0400
From: David Sankel 
To: Boost 
Subject: [boost] A question for folks here
Message-ID:

Content-Type: text/plain; charset="UTF-8"
For those who got involved in Boost within the last couple years, how did
you hear about boost? What attracted you to it?
For those who have been around for a while, what keeps you here? Why do you
stay engaged?
------------------------------
Message: 2
Date: Wed, 10 Jul 2024 18:26:20 +0300
From: Andrey Semashev 
To: boost@lists.boost.org
Subject: Re: [boost] A secure string library?
Message-ID: <6265cabd-c0b9-405e-98f4-8a47683e5472@gmail.com>
Content-Type: text/plain; charset=UTF-8
On 7/10/24 14:59, Peter Dimov via Boost wrote:
...
Ruben Perez wrote:
...
There is OPENSSL_Cleanse, which does a similar job. I found it surprisingly
simple - I wonder if this is just enough or there are any corner cases where this
doesn't work:
https://github.com/openssl/openssl/blob/b544047c99c4a7413f793afe82ab1c165f85...
That's an interesting approach. I can't offhand think of a reason why it wouldn't
work.
The compiler might convert OPENSSL_cleanse to something like this:
void OPENSSL_cleanse(void *ptr, size_t len)
 {
   memset_t func = memset_func;
   if (func != memset)
     func(ptr, 0, len);
   else
     memset(ptr, 0, len);
 }
The purpose here is that, if memset_func is actually memset most of the
time, it can further optimize the call to memset, including to
completely remove it, depending on the call context. The
(well-predictable) branch is typically cheaper than an indirect call. I
think I've seen compilers do something along those lines as a result of
call devirtualization, especially with IPO and LTCG.
I'm not saying that's what actually happens in OpenSSL, just that
something like this is possible. I think, a dummy asm statement is more
reliable and more efficient.
void secure_cleanse(void *ptr, size_t len)
 {
   memset(ptr, 0, len); // a normal memset, optimizations are welcome
   __asm__ __volatile__ ("" : : "r" (ptr), "r" (len) : "memory");
 }
You can even make that function inline and it'll work, and in an optimal
way, too.
------------------------------
Message: 3
Date: Wed, 10 Jul 2024 18:37:00 +0300
From: Andrey Semashev 
To: boost@lists.boost.org
Subject: Re: [boost] A secure string library?
Message-ID: <57326885-13ab-4a73-ac5a-5aa6357e799c@gmail.com>
Content-Type: text/plain; charset=UTF-8
On 7/10/24 18:26, Andrey Semashev wrote:
...
On 7/10/24 14:59, Peter Dimov via Boost wrote:
...
Ruben Perez wrote:
...
There is OPENSSL_Cleanse, which does a similar job. I found it surprisingly
simple - I wonder if this is just enough or there are any corner cases where this
doesn't work:
https://github.com/openssl/openssl/blob/b544047c99c4a7413f793afe82ab1c165f85...
That's an interesting approach. I can't offhand think of a reason why it wouldn't
work.
The compiler might convert OPENSSL_cleanse to something like this:
void OPENSSL_cleanse(void *ptr, size_t len)
 {
   memset_t func = memset_func;
   if (func != memset)
     func(ptr, 0, len);
   else
     memset(ptr, 0, len);
 }
The purpose here is that, if memset_func is actually memset most of the
time, it can further optimize the call to memset, including to
completely remove it, depending on the call context. The
(well-predictable) branch is typically cheaper than an indirect call. I
think I've seen compilers do something along those lines as a result of
call devirtualization, especially with IPO and LTCG.
I'm not saying that's what actually happens in OpenSSL, just that
something like this is possible. I think, a dummy asm statement is more
reliable and more efficient.
void secure_cleanse(void *ptr, size_t len)
 {
   memset(ptr, 0, len); // a normal memset, optimizations are welcome
   __asm__ __volatile__ ("" : : "r" (ptr), "r" (len) : "memory");
 }
You can even make that function inline and it'll work, and in an optimal
way, too.
And for compilers that don't support __asm__, I think you could replace
it with:
std::atomic_signal_fence(std::memory_order::acq_rel);
------------------------------
Message: 4
Date: Wed, 10 Jul 2024 18:01:03 +0200
From: Richard Hodges 
To: boost@lists.boost.org
Subject: Re: [boost] A question for folks here
Message-ID:

Content-Type: text/plain; charset="UTF-8"
On Wed, 10 Jul 2024 at 16:48, David Sankel via Boost 
wrote:
...
For those who got involved in Boost within the last couple years, how did
you hear about boost? What attracted you to it?
I've been using boost as an essential force multiplier of C++ since version
1.35 when Asio appeared, courtesy of Chris Kohlhoff.
More recently in 2017, I needed an asynchronous websocket library for C++
and lo-and-behold, boost had one, courtesy of Vinnie Falco.
...
For those who have been around for a while, what keeps you here? Why do you
stay engaged?
After finishing a contract in which I used Boost.Beast et.al. in production
code, I took up an offer by the C++ Alliance to become a maintainer of it.
A bit of a "Victor Kiam" moment, if you will.
While working for the C++ Alliance I was also involved in the design and
first cut of Boost.Json and had some input into Boost.Url, another two very
keenly needed libraries sponsored by the C++ Alliance.
I am no longer working for the C++ Alliance, but maintain a warm
relationship with the organisation.
I am currently using Boost.Cobalt in a financial trading software suite.
I stay engaged because:
- I don't want to have to switch languages - C++ is perfectly fine for me.
- Boost polyfills all the essential functionality that the children running
WG21 continue to refuse to provide.
- It's convenient to have one library that provides (almost) every tool I
need.
My very great thanks to the original authors and contributors, and more
recently to the C++ Alliance, which as far as I can see is the only major
sponsor of Boost.
Without this organisation's involvement, I believe the libraries would have
been allowed to fall into disrepair a long time ago.
I don't think I have met more than one team in the financial industry which
uses C++ but which does not use Boost.
On a separate note, I was deeply disappointed by your recent public
outbursts, David.
Sincerely,
Richard Hodges
A C++phile
ex maintainer of Boost.Beast
contributor to Boost.Json, Boost.Url
Advisor to Boost.MySQL, Boost.Cobalt
...
_______________________________________________
Unsubscribe & other changes:
http://lists.boost.org/mailman/listinfo.cgi/boost
------------------------------
Message: 5
Date: Wed, 10 Jul 2024 18:13:41 +0200
From: Daniela Engert 
To: boost@lists.boost.org
Subject: Re: [boost] A secure string library?
Message-ID: <98140e71-ef1e-431a-a089-e24f56e390bd@ngrt.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Am 10.07.2024 um 17:37 schrieb Andrey Semashev via Boost:
...
On 7/10/24 18:26, Andrey Semashev wrote:
...
On 7/10/24 14:59, Peter Dimov via Boost wrote:
...
Ruben Perez wrote:
...
There is OPENSSL_Cleanse, which does a similar job. I found it surprisingly
simple - I wonder if this is just enough or there are any corner cases where this
doesn't work:
https://github.com/openssl/openssl/blob/b544047c99c4a7413f793afe82ab1c165f85...
That's an interesting approach. I can't offhand think of a reason why it wouldn't
work.
The compiler might convert OPENSSL_cleanse to something like this:
void OPENSSL_cleanse(void *ptr, size_t len)
  {
    memset_t func = memset_func;
    if (func != memset)
      func(ptr, 0, len);
    else
      memset(ptr, 0, len);
  }
The purpose here is that, if memset_func is actually memset most of the
time, it can further optimize the call to memset, including to
completely remove it, depending on the call context. The
(well-predictable) branch is typically cheaper than an indirect call. I
think I've seen compilers do something along those lines as a result of
call devirtualization, especially with IPO and LTCG.
I'm not saying that's what actually happens in OpenSSL, just that
something like this is possible. I think, a dummy asm statement is more
reliable and more efficient.
void secure_cleanse(void *ptr, size_t len)
  {
    memset(ptr, 0, len); // a normal memset, optimizations are welcome
    __asm__ __volatile__ ("" : : "r" (ptr), "r" (len) : "memory");
  }
You can even make that function inline and it'll work, and in an optimal
way, too.
And for compilers that don't support __asm__, I think you could replace
it with:
std::atomic_signal_fence(std::memory_order::acq_rel);
I think Peter is actually right in his assessment:
While 'memset_func' is TU-local, it is also 'volatile'. This implies 
every look at it might reveal a different content. Therefore the 
necessity for non-constness and dynamic initialization. We humans see 
that the variable can't change its value other than in the 
initialization. The compiler can't reason that because said properties 
are the only ones known during the compilation of 'OPENSSL_cleanse()'. 
It can't perform unbounded look-ahead until the end of the TU like we do.
Dani
-- 
PGP/GPG: 2CCB 3ECB 0954 5CD3 B0DB 6AA0 BA03 56A1 2C4638C5
------------------------------
Message: 6
Date: Wed, 10 Jul 2024 19:24:48 +0300
From: Andrey Semashev 
To: boost@lists.boost.org
Subject: Re: [boost] A secure string library?
Message-ID: <905dd5f7-5d29-48ab-87aa-6aa0a23e60c6@gmail.com>
Content-Type: text/plain; charset=UTF-8
On 7/10/24 19:13, Daniela Engert via Boost wrote:
...
Am 10.07.2024 um 17:37 schrieb Andrey Semashev via Boost:
...
On 7/10/24 18:26, Andrey Semashev wrote:
...
On 7/10/24 14:59, Peter Dimov via Boost wrote:
...
Ruben Perez wrote:
...
There is OPENSSL_Cleanse, which does a similar job. I found it
surprisingly
simple - I wonder if this is just enough or there are any corner
cases where this
doesn't work:
https://github.com/openssl/openssl/blob/b544047c99c4a7413f793afe82ab1c165f85...
That's an interesting approach. I can't offhand think of a reason
why it wouldn't
work.
The compiler might convert OPENSSL_cleanse to something like this:
?? void OPENSSL_cleanse(void *ptr, size_t len)
?? {
???? memset_t func = memset_func;
???? if (func != memset)
?????? func(ptr, 0, len);
???? else
?????? memset(ptr, 0, len);
?? }
The purpose here is that, if memset_func is actually memset most of the
time, it can further optimize the call to memset, including to
completely remove it, depending on the call context. The
(well-predictable) branch is typically cheaper than an indirect call. I
think I've seen compilers do something along those lines as a result of
call devirtualization, especially with IPO and LTCG.
I'm not saying that's what actually happens in OpenSSL, just that
something like this is possible. I think, a dummy asm statement is more
reliable and more efficient.
?? void secure_cleanse(void *ptr, size_t len)
?? {
???? memset(ptr, 0, len); // a normal memset, optimizations are welcome
???? __asm__ __volatile__ ("" : : "r" (ptr), "r" (len) : "memory");
?? }
You can even make that function inline and it'll work, and in an optimal
way, too.
And for compilers that don't support __asm__, I think you could replace
it with:
?? std::atomic_signal_fence(std::memory_order::acq_rel);
I think Peter is actually right in his assessment:
While 'memset_func' is TU-local, it is also 'volatile'. This implies
every look at it might reveal a different content. Therefore the
necessity for non-constness and dynamic initialization. We humans see
that the variable can't change its value other than in the
initialization. The compiler can't reason that because said properties
are the only ones known during the compilation of 'OPENSSL_cleanse()'.
It can't perform unbounded look-ahead until the end of the TU like we do.
With volatile, the compiler is not allowed to optimize away or reorder
loads and stores of the variable. There's no restriction on what the
compiler is allowed to do with the loaded value.
------------------------------
Message: 7
Date: Wed, 10 Jul 2024 10:10:43 -0700
From: Christian Mazakas 
To: boost@lists.boost.org
Subject: Re: [boost] A secure string library?
Message-ID:

Content-Type: text/plain; charset="UTF-8"
...
Andrey Semashev via Boost:
With volatile, the compiler is not allowed to optimize away or reorder
loads and stores of the variable. There's no restriction on what the
compiler is allowed to do with the loaded value.
It could insert a branch against `memset` but that seems really only like a
theoretical
optimization.
This is an example where the volatile preserves the secure_erase() call:
https://godbolt.org/z/854jfarb9
Kind of neat trying to get the compiler to elide this stuff. You were
right, Andrey, you really need
LTO and LTCG and the like to get this kind of behavior. Super neat stuff.
Idk, seems like whatever OpenSSL is doing is Sound Enough in practice.
I'd be curious to know if it's actually sound for a compiler to make that
kind of optimization or at least
what are the limits on how far a compiler can actually go to ensure
soundness. If you know those limits,
you can probably make this pattern work reliably.
- Christian
------------------------------
Message: 8
Date: Wed, 10 Jul 2024 17:57:57 +0000
From: Matt Borland 
To: boost@lists.boost.org
Subject: Re: [boost] A question for folks here
Message-ID:

Content-Type: text/plain; charset="utf-8"
...
For those who got involved in Boost within the last couple years, how did
you hear about boost?
I heard about it from a university professor as a freshman undergraduate student in 2011. Since C++11 was just coming about, and the speed of updating toolchains in academia is slow so we were stuck in the C++03 days on old RHEL systems.
...
What attracted you to it?
I was studying aeronautical engineering so having pre-made libraries like Boost.Math, Boost.Multiprecision, and ODEint were useful to me. I could focus on the problem at hand rather than having to roll my own special functions, solvers, big numbers, constants, etc.
...
For those who have been around for a while, what keeps you here? Why do you stay engaged?
I started to get involved with development during the COVID lockdowns because like many people I found myself with quite a bit of extra free time. What hooked me was learning from and working with extremely competent programmers and mathematicians like John Maddock, Chris Kormanyos, and Nick Thompson. I was hired by the C++ Alliance at the beginning of 2023 and they have sponsored my development since. This allows me to work on new (https://github.com/cppalliance/decimal), and existing libraries (charconv, math, multiprecision, random, etc) in a way that I was never able to on a volunteer basis with a full time job. The community is large, knowledgeable, and helpful so I continue to enjoy working on Boost.
I think it would be beneficial to the developer community for the Boost Foundation board members to also answer these questions. Frankly some even require introductions with their answers. There are members of the board like Peter, Glen, and Zach that are Titans of Boost, and those like Kristen who I had not heard of until her post about wanting to cut ties with the C++ Alliance. Having unknown members of the Boost Foundation runs counter to their mission statement whereby the board intends to achieve its mission by "fostering community engagement, [and] nurturing leaders..."[1].
Matt
[1] https://sites.google.com/boost.org/boost-foundation/home

Re: [boost] A question for folks here

me＠kennethreitz.org

Peter Turcan

tags

participants (2)