December 2024 - Boost - lists.preview.boost.org

asio::strand: Order of execution
by Georg Gast 11 Feb '25

11 Feb '25

Hi, If I use a asio::strand and post() there multiple function objects, are they executed in the same order or is the order of execution not specified? I think to use it as a work queue but I need them to be processed in sequence... Thx Georg

5 9

[review] [sqlite]
by Richard Hodges 04 Feb '25

04 Feb '25

Dear All, Surprise! The Boost formal review of the Boost SQLITE library starts *TODAY*, taking place from November 13th, 2024 to November 22nd, 2024 (inclusive). I apologise profusely for springing this on you without prior warning. The error is entirely mine. I am extending the period by one day to compensate. The library is authored by Klemens Morgenstern (@klemens-morgenstern in the CppLang slack). Documentation: https://klemens.dev/sqlite/ <https://anarthal.github.io/mysql/index.html> Source: https://github.com/klemens-morgenstern/sqlite <https://github.com/anarthal/mysql/> >From the documentation: boost.sqlite is a simple to use C++ sqlite library. It provides a modern interface using facilities like error_code, views (e.g. for blobs) and the ability to use boost.describe or boost.pfr for parameterised queries. Supported features include: - typed queries - prepared statements - json support - custom functions (scalar, aggregate, windows) - event hooks - virtual tables SQLite provides an excellent C-API, so this library does not attempt to hide, but to augment it. Please provide in your review information you think is valuable to explain your choice to ACCEPT or REJECT including SQLITE as a Boost library. Please be explicit about your decision (ACCEPT or REJECT). Some other questions you might want to consider answering: - Will the library bring additional out-of-the-box utility to Boost? - What is your evaluation of the implementation? - What is your evaluation of the documentation? - Will the choice of API abstraction model ease the development of software that must talk to a SQLITE database? - Are there any immediate improvements that could be made after acceptance, if acceptance should happen? - Did you try to use the library? With which compiler(s)? Did you have any problems? - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? - Are you knowledgeable about the problem domain? More information about the Boost Formal Review Process can be found at: http://www.boost.org/community/reviews.html The review is open to anyone who is prepared to put in the work of evaluating and reviewing the library. Prior experience in contributing to Boost reviews is not a requirement. Thank you for your efforts in the Boost community. They are very much appreciated. Richard Hodges - review manager of the proposed Boost.SQLITE library Klemens is often available on CppLang Slack and of course by email should you require any clarification not covered by the documentation, as am I.

17 61

Interest for C++20 modules support of boost officially
by Chuanqi Xu 29 Jan '25

29 Jan '25

Hi Boost Developers, I am Chuanqi Xu, the maintainer for modules & serialization in clang and also a fan of C++20 modules. The post itself was motivated by a discussion (https://cpplang.slack.com/archives/C27KZLB0X/p1734335425348809 <https://cpplang.slack.com/archives/C27KZLB0X/p1734335425348809 >). You don’t need to read it if you’re not interested. I’ll try to make the post itself self contained. The original story comes from a high quality experiment for C++20 modules in boost (https://anarthal.github.io/cppblog/modules2 <https://anarthal.github.io/cppblog/modules2 >). The blog reads pretty good to me. However, anarthal (the author) told me later that the process of supporting modules in boost got suspended since the authors (or the maintainers) don’t want it since no one is going to use them. And if I want it, I need to send a mail here. Then this is the beginning of the story. First of first, I want to emphasize that “no one is going to use them” is not true. We’ve already used C++20 modules at larger scale internally for roughly a year. For boost (and other third party libraries), we choose to wrap a module ourselves as a workaround (with some compiler extensions to ease the refacotoring, but we don’t like it. We still want standard behavior). We didn’t reach out since we feel boost will support C++20 modules later or sooner so we don’t need to do anything specially. (Maybe we were wrong. We didn’t understand the ecosystem enough). Until yesterday, I just knew that boost didn’t know such needs. Before I wrote this email, I wondered a lot of words to emphasize that our need are a good example and the users of closed source are important users too. But just now, I searched the use of boost in modules in GitHub. I just realized I can show it (the importance of C++20 modules support for boost) much easily with the open source world. I searched `boost path:*.cppm` in GitHub and got the following list: https://github.com/infiniflow/infinity/blob/fa922afd58c973a9a317d2c09ad8cea… <https://github.com/infiniflow/infinity/blob/fa922afd58c973a9a317d2c09ad8cea… > https://github.com/ossia/score/blob/0ce6f62a97651a1248a92b3e13ce6ede10341d1… <https://github.com/ossia/score/blob/0ce6f62a97651a1248a92b3e13ce6ede10341d1… > https://github.com/RichardLuo0/make-dot-cpp/blob/e2ef10fd43060a2b56fc8f87f1… <https://github.com/RichardLuo0/make-dot-cpp/blob/e2ef10fd43060a2b56fc8f87f1… > https://github.com/slaakko/cmajor-mod/blob/32e6eb8dc099b395fa751340de96f478… <https://github.com/slaakko/cmajor-mod/blob/32e6eb8dc099b395fa751340de96f478… > https://github.com/df-com/dragonfly/blob/9dfcbc1153f292eb8581f1df7459600cc3… <https://github.com/df-com/dragonfly/blob/9dfcbc1153f292eb8581f1df7459600cc3… > https://github.com/nofe1248/Helium/blob/92ec4452b8c05e6d86c972f843196a4392a… <https://github.com/nofe1248/Helium/blob/92ec4452b8c05e6d86c972f843196a4392a… > https://github.com/LagrangeDev/liblagrange/blob/10e3b48f8d47fd65de00b14cfed… <https://github.com/LagrangeDev/liblagrange/blob/10e3b48f8d47fd65de00b14cfed… > https://github.com/yudaichen/feature-db/blob/e94ade969af682ed5dd25d70bf282b… <https://github.com/yudaichen/feature-db/blob/e94ade969af682ed5dd25d70bf282b… > This is not a complete list even for my search only and I believe there are other patterns to use boost with modules now (e.g., at least `*.cppm` is not the defacto suffix). And I also believe, from my experience as a C++ user, if boost provide modules, I may try to use it. But if boost didn’t provide it, maybe I won’t ask for it if I don’t want modules super bad. Since modules doesn’t add any new functionality actually. So most users may not ask for it explicitly. But if boost provides it, people may give it a try. The above text are the body of the post. I hope it can show it is needed. — Following of are some technical discussion for the extra maintainance burden by introducing modules. The conclusion is, I think, the burden is pretty low. Since modules are not complex to users. Different from other big features in C++, modules are pretty easy to understand. The complexity of modules are majorly on tools and the ecosystem. For programmers who don’t get involved to much to the ecosystem (or infrastructure), modules should be pretty easy to understand. For authors of libraries, after boost introduce modules, you only need to do 2 extra things: - In headers, add a macro (assume its name as BOOST_MODULE_EXPORT, which will be ‘export’ conditionally) to the entities that need to be exported. - In headers, include standard headers and other dependent headers conditionally so that we don’t include anything if the headers are in a module unit. e.g., https://github.com/boostorg/pfr/blob/f09e6aeae9d050897fff72b93d5f5e866cc5e1… <https://github.com/boostorg/pfr/blob/f09e6aeae9d050897fff72b93d5f5e866cc5e1… > then that’s roughly all you need for maintaining it at source level. For building and distributing related things, anarthal told me that the boost will handle it globally so the individual arthors don’t need to care about it. — In slack, there are some other discussion about module names and linking problems (https://cpplang.slack.com/archives/C27KZLB0X/p1734345716728609 <https://cpplang.slack.com/archives/C27KZLB0X/p1734345716728609 >). But I feel they are implementation details. I may not talk too much for that as a user at least now. And also there are opinions for configuration flags. These are the jobs of build systems (cmake is working on it). I think boost don’t need to worry for that. Similarly, at least libc++ has similar problems but libc++ still released std module. — Boost has a special position in the C++’s ecosystem. The most simple and impressive answer I’ve heard for the question “when can we start to use modules” is, “when boost makes it”. Thanks, Chuanqi

15 85

GSoC 2025 Projects
by Matt Borland 24 Jan '25

24 Jan '25

All, In the modules thread and on Slack, I asked if Boost had a current plan for participation in GSoC 2025. I haven't received any responses yet, so I'll assume there isn't a plan. I'm wondering if anyone has: a) Projects they would like to propose for GSoC, and b) An interest in mentoring. Here are some ideas that I think could be beneficial and approachable: 1) A proof of concept for a Boost module, including distribution and installation processes, and potentially adding support to select libraries. This project could have easy access to the Clang and MSVC teams via their mentor it seems. 2) Bi-variate interpolation has been requested before in Math. ACM TOMS algorithms 751 and 761 could fit the bill, but their references are written in F77. A faithful translation and implementation of those papers would be a good start. 3) Chris Kormanyos and I are working on a cryptographic module that we intend to have certified under FIPS 140-3 and then propose to Boost. An extension of this could be a TLS project where a student provides a proof of concept with a generic file transfer server or similar that works securely with OpenSSL, and then also with our new module. I can mentor at least one of these projects. If anyone has past applications for the organization, please send a link or copies. We have until mid-Feb to apply as an organization, which I can do if no one else volunteers. Best, Matt

5 8

Adding optional<T&> find like method to associative containers in boost.
by Ivan Matek 28 Dec '24

28 Dec '24

Hi everybody, I presume somebody thought of this before but for some reason it never happened, but just in case it was never discussed I wonder if people here believe adding an nicer version of find(with different name obviously) to associative containers in boost would be beneficial? Since boost::optional allows references this would work. I presume naming is what most people will have strong feeling about. I personally prefer ofind/oat since I love short names, Barry suggested different names in his blog <https://brevzin.github.io/c++/2023/05/23/map-api/>, e.g. try_at. boost::optional did not get all the updates C++23 std::optional got, e.g. transform that make this usage compose nicely, but those could be added to boost::optional.

13 80

Formal review request for Hash2
by Peter Dimov 23 Dec '24

23 Dec '24

I'd like to request a formal review for the Hash2 library https://github.com/pdimov/hash2 https://pdimov.github.io/hash2/ (by Christian Mazakas and myself.) The library implements an extensible framework for implementing hashing algorithms that can support user-defined types. Its structure is largely based on the paper "Types don’t know #" by Howard Hinnant, Vinnie Falco and John Bytheway. https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3980.html We're looking for a review manager. Comments and questions are welcome; you don't need to wait for the review.

14 75

[hash2] Review Report
by Matt Borland 21 Dec '24

21 Dec '24

Dear all, Again, thank you to those who participated in the review of `Hash2` by Peter Dimov and Christian Mazakas from 7 to 15 December 2024. Bottom line up front: The library is ACCEPTED. By and large, the discussion around the library was positive, with reviewers appreciating that it implements the "Types don't know #" proposal. I think some discussion was perhaps too narrowly focused on providing hash algorithms, as that is not the over-arching goal of the library. Below is a summary of the main discussion points and recommendations from both reviewers and me. Discussion and Recommendations: 1) Support for `std::span`. Based on the discussion, it seems like the use of `std::span` has moved from theoretical to being quite popular. Adding an API for span would in principle be OK behind the requisite feature test macros, but it will add to the conceptual requirements which I'm not convinced is beneficial. Unless Peter and Christian intend to move to C++20, and they don't, I think it would be unreasonable at this time to require spans behind macros in the implementation to allow warning-free builds with Clang's `-Wunsafe-buffer-usage` and whatever memory safety profiles turn out to be. If this library was targeting C++20, or intended to be certified by a standards organization, this would be an entirely different conversation. 2) C++ language standard requirement. As I kind of expected, there was limited support for keeping the library at C++11. The authors should feel free to use C++14 and drop support for any non-conforming compilers before initial release. 3) Documentation. Peter Turcan provided a great deal of feedback on the documentation which I expect the authors to handle. There were also quite a few calls to re-arrange the documentation to put working examples and pitfalls discussed during review first, and details of concepts later. The single vs. multi-page documentation debate will never have a clear winner, so I defer to doc writer's preference on that one. The authors have a pretty good sense of the edits they need to make, so I won't belabor the point. 4) Security. Secure construction was included in the conditional acceptance criteria of two reviews - for example, default construction and allowing too-short keys in SipHash were considered harmful. There were also concerns that the algorithms provided would be considered cryptographically secure. Since the library does not intend to be a cryptographic provider (as that is not the real point of the library), I think an annotation at the top of the docs stating as much should be added. 5) Relationship to `ContainerHash`. Several reviewers noted that since there is no explicit Hash1, the relationship with `ContainerHash` should be explained. Peter currently maintains `ContainerHash`, so I think it would be useful to add similar documentation there as well. This could help avoid the situation we have now where `MP11` fully replaces the functionality of `MPL`, but that's not obvious to the casual outside observer. 6) The concept of HashAlgorithm. There was much discussion around the concept of HashAlgorithm and how it deviates from existing practice. If Peter and Christian don't want to be the sole providers of hash algorithms for their library, they need to provide: - A way to use existing implementations and/or examples in the documentation (this was part of the acceptance criteria for all but one conditional acceptance). This can be accomplished by adding Guidance and an example of an adapter. - An FAQ section addressing questions like "Why does the library impose additional requirements such as result extension on hash algorithms which go beyond the cryptographic specifications in corresponding RFCs?" Boost.Multiprecision already implements this approach to allow unknown arithmetic backends to be used. Final Verdict: I ACCEPT the library. The library was clearly well designed, and the authors provided clear justification for implementation decisions when questioned (e.g., `tag_invoke` for functions as customization points). Peter and Christian are both responsive enough that I expect they will address the above list without needing the forcing function of a conditional accept followed by a review in the future to ensure completeness. In fact Christian already showed me a draft of an OpenSSL adaptor. If this library was coming from an outsider, my opinion would be different. For transparency the final tally of reviews received was ACCEPT: 9, REJECT: 3, CONDITIONAL ACCEPT: 5. Thanks again to all that participated and congratulations to Peter and Christian. Happy Holidays, Matt

3 2

Query on std::variant support
by Murali Kishore 20 Dec '24

20 Dec '24

Hi, Want to check is std::variant allocating memory from shared memory using boost interprocess allocators. EX: std::variant<int, float> v, w; // Define types A and B that we will store in the variant struct A { int x; A(int val) : x(val) {} }; struct B { std::string str; B(const std::string& s) : str(s) {} }; // Define the variant type using MyVariant = std::variant<A, B>; guide me for the above examples to create custom allocators. -- Regards, Murali Kishore

2 1

Pretty printer module
by Дмитрий Архипов 20 Dec '24

20 Dec '24

Recently Niall Douglas posted about his idea to embed GDB pretty printers into binaries using a header. At the time I was thinking about a good way to deploy pretty printers for JSON that facilitates their autoloading. After Niall's post I experimented a little bit more, and I came to the conclusion that indeed nothing beats embedding via a C++ header in terms of convenience. At the same time I was also thinking about testing that pretty printers do in fact work. Given that pretty printers (usually) are implemented as a Python module, it initially looked like one could just run a Python program and import GDB Python modules. But actually you cannot. It turns out, though, you can run a Python script with GDB as the interpreter. After several iterations I arrived at this setup: https://github.com/boostorg/json/blob/develop/test/printers.cpp. A script converts this file into a Python script that uses GDB to break on certain lines and check if a particular expression produces a particular output when printed. While doing all this I thought that most of this work can be useful to other libraries. So, I moved the reusable parts into a separate directory (https://github.com/boostorg/json/tree/develop/pretty_printers) and added both a b2 module, and a CMake module. For B2. Importing: https://github.com/boostorg/json/blob/develop/build.jam#L8-L9 https://github.com/boostorg/json/blob/develop/build/Jamfile#L13 (Re)generating the header: https://github.com/boostorg/json/blob/develop/build/Jamfile#L88-L95 Testing: https://github.com/boostorg/json/blob/develop/test/Jamfile#L96-L101 For CMake: Importing: https://github.com/boostorg/json/blob/develop/CMakeLists.txt#L54-L55 (Re)generating the header: https://github.com/boostorg/json/blob/develop/CMakeLists.txt#L56-L63 Testing: https://github.com/boostorg/json/blob/develop/test/CMakeLists.txt#L80-L88 My questions to the community: do you find this useful? Would you use it in your Boost or non-Boost library? If you do, the module should be moved to tools rather than stay in libs/json.

12 24

Re: [boost] Interest for C++20 modules support of boost officially
by Ruben Perez 19 Dec '24

19 Dec '24

On Thu, 19 Dec 2024, 15:44 John Maddock, <jz.maddock(a)googlemail.com> wrote: > > > If I understood the CMake developers and Chuanqi correctly (please let > > me know if I didn't), you do install a library with the generated > > object files, but you do not install the BMIs, as they're like > > pre-compiled headers. > > > > Let me explain how I understood it with two concrete examples. Take > > Boost.Url as an example of a compiled library, and Boost.Regex as an > > example of a header-only library. > > > > 1. Boost.Url today installs: > > * libboost_url.a, containing function definitions. > > * Headers, containing declarations. > > 2. Boost.Url in the module world would install: > > * libboost_url.a, containing the same function definitions as above. > > * A boost_url.cppm file, akin to what headers are today. > > CMake would build a BMI from this file when a user needs to > > import boost.url. > > In practice, this cppm file will likely be implemented in terms of > the > > header files that we have today. > > 3. Boost.Regex today installs: > > * Headers, containing declarations and definitions. > > 4. Boost.Regex in the module world would install: > > * A libboost_regex.a object, containing module initializers [1]. > > For instance, if you have a singleton defining a Boost.System > > error category, > > initialization code for the singleton would be emitted here. > > From the docs, it seems this object gets created even if no > > initialization code is required. > > * A boost_regex.cppm file, containing the bulk of the library, akin to > > what we provide using headers today. > > > > My intention is doing some testing to see how much trouble would this > > new libboost_regex.a library cause. > > I'm not quite convinced it's that simple, at it's most basic, consider > an inline function that is part of the module interface, if I compile > the module's object library for release, and then do a debug build of > the bmi and my project what happens? Even if clang makes that example > work out in msvc land where almost every option creates binary > compatibility issues, I see users compiling the object library from > source on a per-project basis. > I'd expect the inline function to be part of the BMI (which gets built by the user), and not part of the object file, thus not creating trouble. Of course, I could be wrong, so I'll try it. Regards, Ruben.

1 0

Boost 1.87.0 Release Report
by Vinnie Falco 17 Dec '24

17 Dec '24

We are proud to publish the first Boost Release Report, for 1.87.0. This report is designed to showcase and credit the work of the many volunteers who contribute to the libraries and other things in Boost, published as an official release three times a year. https://downloads.cppalliance.org/reports/boost_1_87_0_release_report.pdf Thanks

4 8

[hash2] Review of Hash2
by Antony Polukhin 16 Dec '24

16 Dec '24

> - What is your evaluation of the design? The design is good, but there is a major issue with security/safety in default constructors of the hash algorithms. ## HashDOS issue The issue is related to HashDOS attacks. The default seed must not be 0. Because with 0 the attacker could easily find strings of the same size that result in the same trailing bits of the hash result. As a result on those strings all the operations with unordered containers degrade to O(N) for each string, giving O(N*N) complexity for a set of strings. The issue affects any parsing where the result is stored into an unordered container: HTTP headers, HTTP cookies, JSON keys, XML... Our in-house experiments show that a single laptop can easily put down a few hundreds of servers just by sending the same precomputed HTTP request. The servers start to use 100% CPU and do not respond to other requests. The problem is so critical, that for some time we write HasDOS tests to make sure that our data structures are not affected: https://github.com/userver-framework/userver/blob/develop/universal/src/htt… With some math knowledge the attack becomes even more simple see https://orlp.net/blog/breaking-hash-functions/ Quick fix for the HashDOS is to generate a random number at the program start use it to initialize all the hashes on construction. Is still leaves a theoretical possibility of attack if there's only one instance of the application or the attacker generates strings set for each instance and sends them in one go. We were not patient enough to generate the exploit, which does not actually mean that it is not possible. A more mature fix is to generate a random seed for each request or container. We were using that approach for quite some time https://github.com/userver-framework/userver/blob/d09b427b42fc80dcd66b8161d… . We also checked some other popular frameworks for the HasDOS issue. Too many of them were affected (we reported to the authors, but not all of them reacted to the problem). All of the above leads to the following conclusion: developers are not aware of that critical issue or do not take the issue seriously. To not provoke vulnerabilities the hashing must be safe by default. ## tag_invoke is dreadful tag_invoke is over complicated. It may make sense if there is a need to specialize the behavior not just for the user defined type, but also to specialize behavior of the user defined type depending on the hashing algorithm and/or flavor. Such functionality does not seem to be required here. So instead of ``` template<class Hash, class Flavor> friend void tag_invoke( boost::hash2::hash_append_tag const&, Hash& h, Flavor const& f, X const& v ) { boost::hash2::hash_append(h, f, v.a); boost::hash2::hash_append(h, f, v.b); } ``` how about using a simpler approach: ``` template<class Visitor> friend constexpr void hash_visit(Visitor visitor, X const& v) { visitor(v.a); visitor(v.b); } ``` where Visitor could be ``` [&h, f](const auto& value) { boost::hash2::hash_append(h, f, value); } ``` ## std::hash and hash_value I'd appreciate functionality to reuse boost::hash2 functionality with std::hash and hash_value. It would be ideal to have something like: ``` template <class T> requires boost::hash2::supports_v<T> struct std::hash { std::size_t operator()(const T& x) const { boost::hash2::default_hasher h{boost::hash2::on_program_start_seed}; boost::hash2::hash_append(h, {}, x); return boost::hash2::get_integral_result<std::size_t>( h.result() ); } } template <class T> requires boost::hash2::supports_v<T> std::size_t hash_value(const T& x) { // put into Boost.Hash return std::hash<T>{}(x); } ``` So that the user just needs to provide boost::hash2 customization for its type, and after that it works with the standard library hashing and old boost hash. > - What is your evaluation of the implementation? Very good. > - What is your evaluation of the documentation? Good. > - What is your evaluation of the potential usefulness of the library? Do you already use it in industry? Not used it. The library is very useful because it researches the constexpr hashing, proposes hash combining and attempts to solve the other std::hash issues. > - Did you try to use the library? With which compiler(s)? Did you have any problems? Not used it. > - How much effort did you put into your evaluation? A quick reading of sources and docs. > - Are you knowledgeable about the problem domain? Yes, but not an expert. > Ensure to explicitly include with your review: ACCEPT, REJECT, or CONDITIONAL ACCEPT (with acceptance conditions). CONDITIONAL ACCEPT. The behavior of the default constructor must be secure. `tag_invoke` seems to be an overkill.

4 5

[hash2] Formal Review Begins
by Matt Borland 16 Dec '24

16 Dec '24

Dear all, The review of Hash2 by Peter Dimov and Christian Mazakas begins today Saturday, December 7th through Sunday December 15th, 2024. Code: https://github.com/pdimov/hash2 Docs: https://pdimov.github.io/hash2/ Please provide feedback on the following general topics: - What is your evaluation of the design? - What is your evaluation of the implementation? - What is your evaluation of the documentation? - What is your evaluation of the potential usefulness of the library? Do you already use it in industry? - Did you try to use the library? With which compiler(s)? Did you have any problems? - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? - Are you knowledgeable about the problem domain? Ensure to explicitly include with your review: ACCEPT, REJECT, or CONDITIONAL ACCEPT (with acceptance conditions). There's already been a flurry of conversation so I thank you all for that. If you have any questions during this process feel free to reach out to me. Additionally, if you would like to submit your review privately, which I will anonymize for the review report, you may send it directly to me at: matt(a)mattborland.com. Matt Borland Review Manager and C++Alliance Staff Engineer

20 94

[hash2] Review
by Julien Blanc 13 Dec '24

13 Dec '24

his is my review of the proposed boost::hash2 library # What is your evaluation of the design? The base idea, separating the hash algorithm from the actual hashing of objects, is a brilliant one. The library consist on two parts: * a collection of hashing algorithm, some of them of crypto quality * a c++ objects hashing facility Given this design, i would not have expected from the library to provide a thorough collection of hash algorithm. But it's a good think to get them. I would have expected, however, to see some guidance about how to integrate an external hashing library. Actually it is unfortunate that this part is missing, because it seems a lot more harder than it should be. Several design decisions could be improved in my opinion: * the requirements on the HashAlgorithm concept are too strict, and have no rationale in the documentation * copy constructibility and copy assignment are unusual across current hash implementations, as it has been noted by other reviewers. I can understand a rationale for copy construct, copy assignment looks very weird to me. The only rationale i can think of is resetting the internal state to a defaulted state, which could be a specific operation. * creation with seed (either as uint64 or as an array) as well as default construction is required. It seems odd. The documentation, for example, strongly advice agains using siphash without a seed. I would prefer that it is a compile time error to use siphash without a seed / with an incorrectly sized seed. I've no problem adding a key/seed to algorithms that do not mandate them, though. But, since the library itself does not create the HashAlgorithm themselve (there is one notable exception to that that), it should not put additional requirements beyond what it uses. The HashAlgorithm concept could be: struct HashAlgorithm { using result_type = /*integral or array-like*/; static constexpr int block_size = /*...*/; // optional void update( void const* data, std::size_t n ); result_type result(); }; Interestingly enough, most of object hashing works if you provide it with such an algorithm (not copyable and not default constructible). I failed to see where default construction is used outside tests, for example. * the usage of void const* + size in interface is controversial, as several people would prefer a span<>. There are good reasons for not using span, and this is not something that should be an issue, as this should only impact HashAlgorithm writers. It is in my opinion an unfortunate design choice that this internal interface is leaked to the end user. The functionality is already provided by hash_append_range, which could be the advertised way of hashing bytes via that library (ie, when mixing hashing of object and hashing of bytes, eg when implementing custom hashing for a type). * constexpr use mandates usage of unsigned char const*. So it could be the primary supported interface over void const*, to avoid interface duplication. * the requirement on result extension is also uncommon. While i understand the rationale, it should be noted that it is likely to cause some language interoperability issues, as it is not necessarily specified by the algorithm. Implementations in other languages are unlikely to provide them, making the feature a niche one, unless the underlying algorithm specification provides it. * most of the functionality is provided by the hash_append function, and the hash_append_xxx helpers. This is a neat design. * The Flavor template argument allows to customize hashing of objects, to get platform-independants hashes. This is a good idea, but it comes with some problems: * the flavor is repeated over each hash_append call. I don't think it is a realistic use case that the flavor changes over time. If i were to integrate the library in our code base, the first thing i'll do is write a wrapper to get rid of that flavor argument at every call site. * the default flavor uses std::uint64_t for size. It seems odd. If i'm not asking for a specific flavor, i'm not asking the hash to be consistent accross different systems. So std::size_t should be used there instead. This looked to me like a pay for what you don't use situation, so i ran a quick near worst-case benchmark on our arm platform (hashing a million times a std::vector of the three integers 1-2-3), to find out that it makes a the code using size_t runs 15% faster. # What is your evaluation of the implementation? The implementation looks good. Code is clean, very readable. A minor remark in hash_append_unordered_range : m should be declared as std::size_t (or std::uint64_t). The current typename std::iterator_traits<It>::difference_type m = 0; is wrong. I'm not at ease with hash_append_unordered_range. The fact that it reduces the hash to 64 bits, simply making a sum of the element hashes, and then hashing that, does not look right. If the goal is simply to avoid unintended collisions, it's probably fine. If the goal is to preserve against an attacker mindfully trying to provoke collisions, it may do the trick,although intuitively xoring the whole results would be better at that job than summing truncated results. If it's to preserve cryptographic properties of the hash function, then it looks wrong. Another remark is about size serialization. The library relies on std::tuple_size to identify container-like structs, but not include the size in the hash (or is it because of is_contiguously_hashable?). This leads to unintuitive results, such as: std::array<int, 4> a{1,2,3,4}; std::span<int, 4> s(a); Hash h1,h2,h3,h4; hash_append(h1, {}, a); hash_append_range(h2, {}, begin(a), end(a)); assert(h1.result() == h2.result()); // TRUE hash_append(h3, {}, s); hash_append_range(h4, {}, begin(s), end(s)); assert(h3.result() == h4.result()); // FAILS I'm not sure there's a general solution to this issue, though. But since std::array models a container, not including its size in the digest may not be the wisest choice (the doc talks about plain arrays, not about std::array, so i don't know if it's a bug or a feature) # What is your evaluation of the documentation? The order of the documentation is odd. I would expect it to first explain me how i use the library with the provided algorithm to hash c++ data or bytes, but it first goes into deep details about how for example how result() can be called multiple times, or what requirements are set on custom hash algorithms. A description of the scope of the library would be welcomed. It both provides hashing for use by containers, and cryptographic quality hashing. These two seems at first unrelated, and the library lies in a not clearly defined in-between. It has been explain during the review, and the arguments given are convincing, and should find their place in the documentation. The motivation for machine architecture independant hashes is lacking. Usually, the way to achieve this is to use a domain specific canonical form, because endianness or word size is unlikely to be the only variable at stake here. But overall, the documentation has all the relevant information. A tutorial / quick start would be a nice addition. I dislike the "single page style" documentation, and hope the tooling can be improved in this regard. # What is your evaluation of the potential usefulness of the library? Do you already use it in industry? I have a need for an easy to use sha1, that i can easily compile for embedded targets. Having that in boost actually lowers the number of external dependancies, so yes this is definitely useful. I currently don't have a need for the object hashing facility, but would definitely use the library it if the needs arise. # Did you try to use the library? With which compiler(s)? Did you have any problems? I tried with several flavors of gcc, cross-compiling for bare-metal cortex-m0 targets or embedded arm32 linux, as well as standard debian x64. Everything worked. # How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I did not review the algorithm implementations. I spent around 15 hours on studying the library doc and internals. # Are you knowledgeable about the problem domain? No specific knowledge of the domain. # What is your final recommendation I would recommend to CONDITIONALLY ACCEPT the library into boost, the conditions being: * demonstrate the usage of an existing external hash library into the boost::hash2 framework * remove default construction and too short key construction for algorithms where this is actually harmful (such as siphash). I don't have an issue with providing constructors with keys for algorithms that do not mandate it. But boost code should encourage correct usage of the algorithms it propose, not allow bad usage in the name of genericity. The following improvements should imho be done, but i don't base my conditional acceptance on them. Actually some of them can be done on top of the proposed library: * advertise hash_append_range as the recommended way to add untyped data to the hash algorithm (or define an additional hash_append_bytes to fulfill this role). * adds a tutorial / introduction in the doc, move the whole hashing byte sequences item after the hashing c++ objects item (if previous point is adressed, part of it would have to be included in the hashing c++ objects item). * makes copy construction requirement on hash algorithm a requirement only if hash_append_unordered_range is used. Hashing unordeder ranges is an open problem, some creative other solutions may be found, and i see no reasons why they would involve duplicating the algorithm state. Actually, if the hash is default constructible, then it is not needed. You need either one of default or copy constructible, not both. * use std::size_t instead of std::uint64_t for the default_flavor, as this is faster on some platforms. * provide wrappers that avoid repeating the flavor over and over. It is much more convenient for the user to write once using MyHash = hash2::FlavoredHash<hash2::default_flavor> and then MyHash::append(X) than repeating over and over the parameter. Finally, i want to thank Peter for developing and proposing this library for boost, and Matt for managing the review. Regards, Julien

3 3

Ruben's Hash2 review
by Ruben Perez 13 Dec '24

13 Dec '24

Hi all, This is my review of the proposed Boost.Hash2. Thanks Peter and Chris for submitting the library, and Matt for managing the review. - What is your evaluation of the design? Very good. This is how hashing C++ objects should work, as it properly separates concerns between types and hash functions. I think usability could be improved if the template class hash shown as an example was included as part of the library and introduced much earlier in the documentation. In general, the HashAlgorithm concept makes sense. I'm not convinced about the HashAlgorithm::result() interface, for two reasons: * It makes integration of third-party hashes more difficult, as already noted by other reviewers. * The naming suggests an accessor, rather than a modifier. I'd consider splitting it into two functions, maybe finalize_block() and result(). But I can also see the benefits of the approach taken. I miss some higher-level features when working with untyped byte sequences. While this doesn't seem the main value proposition of the library, it's listed as a valid use case. See below for more info. These are nice-to-haves rather than requirements, though. I was initially surprised to find out that the library was header-only, as I didn't know that compile-time hashing was something worth it. I was initially worried about compile-times. I built one of the examples with compile-time profiling (-ftime-trace) and the library is pretty lightweight, so I have no problems with the header-only design. Have you considered somehow supporting hashing for objects describable via Boost.Pfr? I had it requested by some users in MySQL even when I had built-in Boost.Describe support, so it may get requested here, too. - What is your evaluation of the implementation? The parts where I've peaked in are good. Some comments would have helped me, but in general I've been able to follow. The test suite looks pretty comprehensive, too. Some fuzz tests for the cryptographic algorithms would be appreciated (I can help if required). As other reviewers have commented, the algorithm for hashing unordered ranges can use some improvements to make it more secure. I'm no security expert and can't judge how important this is, but I'm pretty sure the author will solve this promptly. The implementation of cryptographic functions is currently slower than solutions like OpenSSL [4], but the author has manifested the possibility of closing this gap. It would be nice having HashAlgorithm modelled as a C++20 concept, although the required macro machinery required to perform the checking only if concepts are supported (like BOOST_ASIO_COMPLETION_TOKEN_FOR does) may be too much effort for the benefit it brings. Regarding Boost integration, a build.jam should likely be added, so the library integrates in the new modular Boost infrastructure (although I've been able to use the library fine without it). This can be done after acceptance. As a matter of curiosity: which compilers do not support __builtin_is_constant_evaluated? (as they will always get the slow memcpy implementation, even at runtime). - What is your evaluation of the documentation? The documentation is good, and explains rationales pretty well, but the order could be improved: * Most users will likely want to use the existing algorithms with their unordered containers, so this should be explained first. * The next block of users will likely want to use the existing cryptographic hash algorithms to untyped byte sequences, so this should be next. * The rest can be left as is. I miss the motivation for the flavor construct in the docs. If I understood correctly, it's only relevant if you plan to somehow serialize your hashes. Maybe an example could help. The library is also suitable for use cases that combine the C++ object hashing and untyped byte sequence hashing parts, as explained by the author here: https://lists.boost.org/Archives/boost/2024/12/258765.php. Such cases are the rationale behind having cryptographic functions within the same umbrella as non-cryptographic ones. I'd suggest adding these to the docs. In general, I dislike single-page docs (which unfortunately seem pretty popular in Boost), as they're difficult to search. I also miss linking (e.g. hash_append should contain a link to its description in the reference). It would also be beneficial to explain the library's relationship with boost::hash / Boost.ContainerHash. In general, making it more approachable for non-expert users would be good. - What is your evaluation of the potential usefulness of the library? Do you already use it in industry? Robust, extensible C++ object hashing is very useful. My immediate use case is related to hashing untyped byte sequences, though. MySQL/MariaDB authentication involve a challenge-response protocol that involves hashing the user-supplied password with a server-supplied challenge, using either SHA1 (sigh) or SHA256 [1]. I'm currently using OpenSSL, as I use it for TLS, too. This library would allow me to make the OpenSSL dependency optional (which I've been requested). I'm also planning on writing a Postgres protocol library. It will likely need to support SCRAM-SHA256 authentication, similar to what I've described. This library could be handy for this. - Did you try to use the library? With which compiler(s)? Did you have any problems? I've re-written the password hashing functionality in Boost.MySQL to use the proposed library [2]. Everything has been very smooth. The test suite passes for all the compilers I support [3], which is really good. I'd have liked some utilities on top of HashAlgorithm to make hashing byte sequences safer and easier. These utilities would *not* imply any change in the HashAlgorithm interface: * Adding a contiguous range of bytes to an algorithm currently supports only a pointer + size pair. This is the right choice for HashAlgorithm, but having a helper function on top of it would be great, as proposed by other reviewers. This would make my code go from: // password is a string_view hash2::sha1_160 hasher1; hasher1.update(password.data(), password.size()); auto password_sha1 = hasher1.result(); To: hash2::sha1_160 hasher1; hash2::hash_add_bytes(hasher1, password); auto password_sha1 = hasher1.result(); * Adding a function that computes the hash of an entire byte sequence, as a range, could be useful. OpenSSL has this as SHA1() and SHA256(). This would make my code go from: hash2::sha1_160 hasher1; hasher1.update(password.data(), password.size()); auto password_sha1 = hasher1.result(); To: auto password_sha1 = hash2::compute_hash(hash2::sha1_160(), password); These are just nice to have and trivially implemented in user-land, if required. The code is already good as is today. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I have dedicated around 8 hours to study the documentation, build the library and examples, integrate it into Boost.MySQL, update my CIs to run my tests with the library, and write this review. - Are you knowledgeable about the problem domain? I am a mere user of hash algorithms, as I usually write networking code. I don't know about their internals. I haven't authored associative containers, either. - Affiliation disclosure I am currently affiliated with the C++ Alliance. - Final decision My recommendation is to CONDITIONALLY ACCEPT the library into Boost. My only acceptance condition is adding the hash template class developed in the examples to the library. The benefit of having the library in Boost is big and overcomes any concerns I might have. Many thanks to both Peter and Matt. Kind regards, Ruben. [1] https://dev.mysql.com/doc/dev/mysql-server/8.4.3/page_caching_sha2_authenti… [2] https://github.com/boostorg/mysql/pull/389 [3] https://github.com/boostorg/mysql/blob/c438f26731e36c2db6457705ec5dbb9f7657… [4] https://lists.boost.org/Archives/boost/2024/12/258687.php

4 4

Re: [boost] [hash2] My Review
by Vinnie Falco 12 Dec '24

12 Dec '24

On Sat, Dec 7, 2024 at 5:47 AM Matt Borland via Boost <boost(a)lists.boost.org> wrote: > The review of Hash2 by Peter Dimov and Christian Mazakas begins today > Saturday, December 7th through Sunday December 15th, 2024. > > Code: https://github.com/pdimov/hash2 > Docs: https://pdimov.github.io/hash2/ Disclaimer: I am affiliated with The C++ Alliance and I am also a co-author of N3980 (the paper referenced by the library). This is my review of Hash2. Spoiler alert, my suggestion is to: ACCEPT While most of my review is dedicated to discussing some concerns I have about some aspects of the library, it is clear that Boost as a collection is better with Hash2 than without it. Also, Peter is going to do what Peter is going to do so why fight it. The uncontroversial parts are stated first, and after lies my assessment of some questionable design choices. > - What is your evaluation of the implementation? > I looked over some of the implementation to better understand how certain things work. The tests pass on CI and the test code looks comprehensive. Peter is not in the habit of producing anything less than high-quality implementations and this library is no exception. I don't know which parts are from Peter and which are from Christian and the quality appears high throughout. > - What is your evaluation of the documentation? > I think the documentation could be better in places. The named requirements exposition is too informal for my taste and talks about things that are not strictly relevant, such as zero-padding. The library alludes to cryptographic usage yet glosses over some details which surfaced during mailing list discussion (more on that below). It would be nice if there was some exposition up front explaining which groups of users the library is for, and how each group uses the library. > - What is your evaluation of the potential usefulness > of the library? Do you already use it in industry? > Not only is the library useful, but the C++ Standard is poorer for not having a modern hash solution. Competing approaches were developed by Google and N3980, then promptly abandoned. > - Did you try to use the library? With which compiler(s)? Did > you have any problems? > I tried to build a Visual Studio Solution from the CMakeLists.txt after cloning hash2 into my superproject, with this command line: cmake -G "Visual Studio 17 2022" -A x64 -B bin64 -DCMAKE_TOOLCHAIN_FILE="C:/vcpkg/scripts/buildsystems/vcpkg.cmake" -DVCPKG_CHAINLOAD_TOOLCHAIN_FILE="C:/users/vinnie/src/toolchains/msvc.cmake" It started downloading things from the internet (FetchContent I guess) and then failed. I was not able to generate a solution. I asked Peter about this and he said it was not supported. It would be nice if it was. > - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? > I focused primarily on the ways in which the library differs from the original paper N3980. I looked at the relevant documentation, interfaces, and implementation, and engaged the author on the mailing list. - Are you knowledgeable about the problem domain? Fairly knowledgeable. - What is your evaluation of the design? This is where I have concerns. The audience of users for Hash2 can be divided into roughly these groups: 1. Authors of types who just want to make objects of their type hashable without having to learn about hash algorithms cryptography, and users who want higher quality results when working with unordered containers. Hash2 is basically perfect for these two groups of users (which often overlap) although the documentation could be improved to speak more directly to them. 2. Users who wish to calculate a cryptographic digest for objects of Hash2-enabled types, or for untyped binary data. Hash2 is also almost perfect for these use cases, and as before the documentation could be more explicit about these cases. 3. Users who wish to adapt existing code to meet the HashAlgorithm requirements from Hash2. This is where I believe the library has some design problems. An unfortunate consequence of being right almost all the time, is that it becomes more unlikely to recognize when you are wrong. I will assume that the reader is at least somewhat familiar with N3980 and some of the arguments I made about the HashAlgorithm named requirements on the mailing list. Hash2 attempts to improve the quality of life for users by mandating additional requirements for the HashAlgorithm concept which may be difficult or impossible to meet for some existing libraries. For example, it requires that every HashAlgorithm support: * Seeded construction from std::uint64_t * Keyed construction from binary data * Copy construction * Digest extension from repeated calls to result() * Interleaving of calls to update() and result() * ...Future stuff that the author alludes to These issues above are already discussed extensively on the mailing list and I can't be bothered to spend hours putting it all together into a beautiful post the way that some of the other reviewers have done (Andrey Semashev and Samuel Neves come to mind). The burdensome additional named requirements of Hash2's HashAlgorithm ensure that no casual users will be authoring them. Or worse, they will get them wrong and the results will not have the same guarantees as the underlying algorithms. Right now there are approximately 1,649,127 implementations of cryptographic and non-cryptographic hash algorithms. I think it might be more prudent for Hash2 to offer a system of traits for identifying which algorithms are capable of what, such as CopyConstruction, digest extension, and so on, and give the user tools which only work in safe ways given the traits of a hash algorithm. This would make it easier to write HashAlgorithms. And as a consequence we might get many more of them rather than fewer. Realistically I don't expect the library to change, and a proliferation of user-created HashAlgorithms is probably conditional on the Boost library collection being much more popular than it currently is. This is partly why I have decided to just accept the library. And I could be wrong, and Peter's unwavering insistence that users get HashAlgorithms which all offer the same quality of service might turn out to be better. Time will tell. Thanks

2 1

[hash2] Difference from paper: HashAlgorithm interface vs. function object
by Andrey Semashev 12 Dec '24

12 Dec '24

Hi, The HashAlgorithm interface defined in Boost.Hash2 is notably different from the one described in N3980 (https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3980.html) Specifically, the paper defines HashAlgorithm as a function object, whose operator() is equivalent to Boost.Hash2's HashAlgorithm::update method. The function object also provides a conversion operator to result_type, which is equivalent to Boost.Hash2's HashAlgorithm::result method. Is this discrepancy intentional and are there plans (on the paper authors' or the library authors' side) to eliminate it? Personally, I don't think a function object is the right interface for a hash algorithm. Update and finalize are two distinct operations provided by hash algorithms and both should be performed explicitly by the user. I don't think a conversion operator is the right name for finalize. Conversion operators are normally used when one type can meaningfully "impersonate" another, and this is not the case here. The function object is also misusing result_type, as its operator() doesn't return it. The paper discusses that the fact that a hash algorithm is a function object can be used to wrap it in std::function, which can be useful with pimpl idiom and for the purpose of type erasure. I don't think this will actually work, at least not as simple as the paper describes it, because std::function won't allow to extract the computed hash digest (i.e. you won't be able to call the type conversion operator on the wrapped function object). I think, the user will have to write some wrapper classes anyway to retain access to the wrapped hash algorithm. However, this is still an interesting use case. Does Boost.Hash2 intend to provide utilities to address it? For example, Boost.Hash2 could provide a hash-algorithm-agnostic interface for feeding an algorithm with input data, and a wrapper for an external algorithm object that would implement this interface. Something along these lines: class polymorphic_hash_algorithm { public: polymorphic_hash_algorithm( polymorphic_hash_algorithm const&) = delete; polymorphic_hash_algorithm& operator=( polymorphic_hash_algorithm const&) = delete; virtual void update(const void* data, size_t size) = 0; protected: polymorphic_hash_algorithm() = default; ~polymorphic_hash_algorithm() = default; }; template< typename Hash > class polymorphic_hash_algorithm_wrapper : public polymorphic_hash_algorithm { public: using hash_type = Hash; using result_type = typename hash_type::result_type; static constexpr size_t block_size = hash_type::block_size; private: Hash& h; public: explicit polymorphic_hash_algorithm_wrapper(hash_type& h) noexcept : h(h) {} void update(const void* data, size_t size) override { h.update(data, size); } result_type result() { return h.result(); } }; This way, polymorphic_hash_algorithm could be used to abstract away the hash algorithm type while still be compatible with hash_append and friends. As a side note, this would be another use case where the hash algorithm passed to hash_append is non-copyable. And while copyability could be added, most likely it wouldn't be free.

6 11

The Master branch is now open for changes
by Marshall Clow 12 Dec '24

12 Dec '24

Thanks to everyone for their patience. The next release cycle will begin in late February. — The release managers.

1 0

[hash2] Question about cryptographic guarantees of hash_append_unordered_range
by Ivan Matek 12 Dec '24

12 Dec '24

Hi everyone, Implementation uses uint64_t in which it sums into hashes of particular elements(each computed with "fresh" hasher so ordering does not matter), then it finally hashes that sum and range size into original passed hasher. I see no problem with logic this since it cleverly exploits the fact that uint64_t overflow is defined and that sum is commutative. But I wonder if this preserves the cryptographic quality of original hash. For example if we use sha2 it seems to me that finding collisions becomes easier since sum is "only" 64 bits, and length is "only" 64 bits(or 32 on some systems), and may be even less if attacker knows unordered range was not super long, e.g. number of items was less than 10000. Does somebody know more about this particular domain and could point me to some resource to learn more? I know it was in the original proposal, but it is possible original proposal was not focusing only on cryptographically secure algorithms. I was unable to google any guarantees, maybe because idk what keywords to use. :)

9 17

[release] Boost 1.87.0
by Marshall Clow 12 Dec '24

12 Dec '24

Release 1.87.0 of the Boost C++ Libraries is now available. These open-source libraries work well with the C++ Standard Library, and are usable across a broad spectrum of applications. The Boost license encourages both commercial and non-commercial use. This release contains two new libraries: * Boost.Parser: A framework for building parsers, from Zach Laine. The release also contains numerous enhancements and bug fixes for existing libraries. For details, including download links, see <http://www.boost.org/users/news/version_1.87.0> You can also download directly from: <https://boostorg.jfrog.io/artifactory/main/release/1.87.0/> To install this release on your system, see <http://www.boost.org/doc/libs/release/more/getting_started/index.html> Thanks to everyone who participated in this release. -- The Boost Release Team Marshall Clow, Glen Fernandes, Ion Gaztañaga

2 2

[Hash2] Current list of items brought up during review
by Peter Dimov 11 Dec '24

11 Dec '24

This is what I've scribbled so far as a list of to-do items: - add hash_append_range taking a range - add a "provider" argument (providing the hash_append API) to tag_invoke - example of hashing boost::json::value - maybe change argument of tag_invoke to take the value by pointer, rather than by reference - maybe expose has_tag_invoke to users so they can specialize - make byte sequence seed constructor take `void const*` as well - change get_integral_result to take a Hash, instead of h.result(), so that it can invoke result() more than once - a user-accessible way to test whether a hash algorithm conforms to the requirements - add a variadic hash_append overload - hash_append_unordered_range: use more than 64 bits for intermediate hashes (uint64_t[4]); improve somehow https://lists.boost.org/Archives/boost/2024/12/258643.php - acknowledge https://github.com/HowardHinnant/hash_append - in documentation, focus more on the std::hash case, rather than on the md5sum case - in FNV-1a byte seed constructor, include the seed size - change block_size to std::size_t

2 1

Re: [boost] [Hash2] Result extension
by Peter Dimov 11 Dec '24

11 Dec '24

Vinnie Falco wrote: > Sigh.. copy/paste typo. > > /** Return a seeded HashAlgorithm > */ > template< class HashAlgorithm > > HashAlgorithm make_seeded( > unsigned char const* seed, std::size_t n ) > { > if constexpr(std::is_constructible<HashAlgorithm, > unsigned char const*, std::size_t>) > return HashAlgorithm(seed, n); > else > { > HashAlgorithm h; > hash_append(h, seed, n); You can in principle do something like that (except fix it to compile) but (much) better practices would be - hash.update( seed, n ) - encode `n` as 64 bit little endian in 8 bytes - hash.update these bytes - hash.update( "\x80", 1 ) - hash.update( "\x00", 1 ) - hash.update with as many zeroes as needed to reach a multiple of HashAlgorithm::block_size That's, incidentally, exactly what `update(p, n); result();` does. (What an amazing coincidence.) Note that reaching a multiple of block_size is important, because it ensures that the secret key you passed as the seed is not left in the internal buffer of the hash algorithm, visible in memory dumps. The test https://github.com/pdimov/hash2/blob/develop/test/plaintext_leak.cpp tries to catch these mistakes. That's why this line https://github.com/pdimov/hash2/blob/7a25f8518692b657e9272884519519fbaca2ec… is needed, for example. > return h; > } > } > > Thanks >

4 6

Github Actions CI updates - e.g. Node 20
by Alexander Grund 10 Dec '24

10 Dec '24

Dear maintainers, I made a bunch of PRs to fix the Github Actions CI for almost 40 repositories. Opening a PR in GitHub is manual work while I was able to apply similar changes over all those repositories using a script which commits to a single branch (and hence PR) per repo. Hence the PRs contain a few related changes that in combination allow the CI to pass again. To avoid repeating this in each PR I wanted to shortly explain the changes here: 1. macos-12 runners are dropped by github and jobs won't start anymore. at least macos-13 needs to be used 2. Ubuntu 23.* doesn't work anymore as e.g. some APT repositories don't exist anymore. Ubuntu 24.04 can be used instead 3. clang 17 (and partly 16) don't work in C++23/2b mode with the stdlibc++ (12) installed on ubuntu 24.04, clang 18+ and/or the other C++ standards can still be used 4. Support for Node 16 was recently fully dropped by Github Actions so the @v3 actions must be upgraded to @v4, which require Node 20. The workaround using $ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION has no effect anymore 5. The node 20 version available by default conflicts with the glibc version in the Ubuntu 16/18 container images used for testing older compilers. I addressed all of them with last one being the major one: Installing a compatible version inside a custom image to be used instead doesn't work because GHA mounts the node 20 installation from the host os. The solution I implemented downloads an unofficial node 20 built against the correct/older glibc and creates 2 mount points such that we can put that node onto the host runner and mount that instead of what github gives us using the idea from https://github.com/dixyes/ghactionsplay/blob/bf889ee2fed8dcd72cc8e32bfd4789… To avoid overloading the unofficial-builds.nodejs.org server we use a copy hosted at boost.io Download and extraction requires curl and xz-utils to be installed which I added where required. Additionally I noticed additional failures due to use of C++03 where the dependencies require C++11, or of C++0x using GCC < 4.9 where e.g. C++11 `alignof`, required by the Boost.SmartPtr dependency, is not available. For (only) those I removed the C++03 and/or GCC 4.7/4.8 jobs (which cannot succeed). I think combining those changes into a single PR avoids a lot of additional manual work by me and allows to verify that the changes are sufficient to make CI pass again. Keeping e.g. C++03 builds, that cannot succeed due to dependencies, makes verifying the success of the Node 20 fix harder/less clear If anyone prefers to separate those changes into multiple PRs please mention that in the relevant PR for me to keep track of, or cherry-pick the individual changes. However it might be sufficient to check the status of the CI for the individual commits to compare the state (success/failure) for the jobs fixed or removed by the next commit. I realize maintaining the CI is a PITA and hope this helps. Regards, Alex

6 8

[Release] Boost 1.87.7 Release Candidate 1 is available
by Marshall Clow 09 Dec '24

09 Dec '24

The first release candidates for the 1.87.0 release are now available at: <https://boostorg.jfrog.io/artifactory/main/release/1.87.0/source/> The SHA256 checksums are as follows: f55c340aa49763b1925ccf02b2e83f35fdcf634c9d5164a2acb87540173c741d boost_1_87_0_rc1.tar.gz af57be25cb4c4f4b413ed692fe378affb4352ea50fbe294a11ef548f4d527d89 boost_1_87_0_rc1.tar.bz2 f367a218d27b23b568956bacf56d08d2c1d9647211337136be8cac5cb2ff0828 boost_1_87_0_rc1.zip db6b7d77a2286ebe79d4aa7252c08b051d271fd8b013b6b18bc7a966552c1139 boost_1_87_0_rc1.7z As always, the release managers would appreciate it if you download the candidate of your choice and give building it a try. Please report both success and failure, and anything else that is noteworthy. -- The Release managers

8 11

[Hash2] Result extension
by Peter Dimov 09 Dec '24

09 Dec '24

Vinnie Falco asked me the following on Slack: > I would ask, what is the motivating use-case for calling result > twice? This is not explained in the docs and no examples are > given. In fact, the one example given says "not to do this" Calling result() twice (or more times) provides result extension; the ability to extract variable number of bits from a hash algorithm, instead of a fixed size value (e.g. 64 bit.) This is in fact stated in the docs here https://pdimov.github.io/hash2/doc/html/hash2.html#hashing_bytes_result > Note that result is non-const, because it changes the internal > state. It’s allowed for result to be called more than once; > subsequent calls perform the state finalization again and as a > result produce a pseudorandom sequence of result_type values. > This can be used to effectively extend the output of the hash > function. For example, a 256 bit result can be obtained from a > hash algorithm whose result_type is 64 bit, by calling result four > times. and there is an example of doing that here https://pdimov.github.io/hash2/doc/html/hash2.html#example_result_extension All hash algorithms are required to support result extension, because (in my opinion) this is extremely useful functionality that is easy - even trivial - to provide, but is often withheld either by accident or in some cases, even deliberately. Hash algorithms typically have a "finalization" phase that pads the message, mixes the length, scrambles the internal state in a more thorough manner than in `update`, and then derives a hash value from that state. (The hash value is often shorter than the total amount of state.) If this "finalization" phase is performed more than once, one naturally gets the mandated `result()` behavior. Falco continues: > I pointed out in the post I already made that the quality of > digest from calling result twice is dependent on the hash > algorithm, and there is no way the library can provide > assurances on the quality That's of course correct, but it also applies to the quality of calling `result()` only once; it's naturally dependent on the implementation of the hash algorithm. What's important here is that it's not possible to provide an extended result of better quality from the outside; the hash algorithm is in the best place to provide it because it has access to more bits of internal state than it lets out. This requirement effectively mandates that all _hash algorithms_ be _extendable-output hash functions_: https://en.wikipedia.org/wiki/Extendable-output_function Note that this is not the only innovation that the proposed hash algorithm concept involves. All hash algorithms are required to support seeding from uint64_t and from an arbitrary sequence of bytes, which makes them effectively _keyed hash functions_ (or _message authentication codes_). Also note that the requirement that one can interleave calls to `update` and `result` arbitrarily makes it possible to implement byte sequence seeding (for algorithms that don't already support it) in the following manner: Hash::Hash( unsigned char const* p, size_t n ): Hash() { if( n != 0 ) { update( p, n ); result(); } } Subsequent `update` calls now start from an initial internal state that has incorporated the contents of [p, p+n), and that has been "finalized" (scrambled thoroughly) such that the result is not equivalent to just prepending the seed to the message (as would have happened if the result() call has been omitted.)

4 14

[hash2] Hash Algorithm Requirements versus concepts
by Andrzej Krzemienski 08 Dec '24

08 Dec '24

Hi Everyone, It strikes me that the description of Hash Algorithm Requirements does not use C++ concepts. Even though the library is C++11-compatible, it could use a concept in the documentation. The goal of the concepts was to serve a case like this, so I wonder if this is just an omission from the authors, or if this is an indication that concepts do not deliver their promise. There is the optional `block_size` but this could be modeled by two concepts HashAlgorithm and BlockHashAlgorithm. The type of `block_size` looks like a candidate for another concept HashAlgorithmResult. Regards, &rzej;

1 0

[hash2][review] Early review (due to holidays)
by Claudio DeSouza 07 Dec '24

07 Dec '24

Hello. My review is the summary of a few discussions I've had with Peter and Christian regarding the API design being employed for hash2. I'm aware hash2 is inspired by one of Hinnat's proposals to standardisation that didn't come to fruition. I would like first to commend the authors for their work on this project. Having good libraries for this type of task is important, especially when the big names like OpenSSL provide not-so-great interfaces. Having said that, I have a few thoughts on what this library may be missing. *Spanification* There's a significant change in mood with regards to safe library design taking place in the industry. With clang, we now have -Wunsafe-buffer-usage, and this warning has been continuously improved recently. For reference https://clang.llvm.org/docs/SafeBuffers.html When I was going through this library's code I was left with a frustrating thought that boost should be at the forefront of this type of innovation: safe, and intuitive library design. Each function that takes a pair of void*/size_t should be a span<uint_8>. The model of how to approach these areas is a solved issue at this point. A robust span type with intuitive extensions would have matching constructors for all sorts of contiguous ranges, including native arrays, which would make using this library much simpler. It would also afford the necessary compile-time validations that you get with static extents in a span. At the current state, hash2 relies on a lot of pinky-promise guarantees that we tend to live with in C++, but that in fact can be avoided without much (or any) overhead. *Digest type* Due to limitations to how much of std::array is constexpr in different C++ versions, this library offers its own digest type. However, this just wraps a stack array. I feel that rather than returning a digest that is stack allocated, it would have been better for algorithms to take a fixed size span where the digest can be written to. I say this because in many cases a stack allocated array will be fine. However a mutable fixed span would provide more flexibility about the storage, for instance, a vector. There's also the matter of hardening (https://libcxx.llvm.org/Hardening.html) which one loses by using this particular digest type, rather than storage provided by a standard type. But I would also like to mention that sha3 has a variable digest size, so again, another reason why an out param mutable span would be a more flexible choice on the whole. *Boost Identity* I have been reluctant to write a review because it seems unfair to demand of hash2 to provide robust spanified interfaces, when no other library before was required to do so. However, at the same time, considering that this type of code tends to be used in safety-sensitive contexts, I fear this library may fail to achieve its potential, when a lot of its interfaces are unsafe. if I had to bring this library to any project I'm working on, I would have to wrap it into safe spanified interfaces, to hide away all these pointer/length pairs around, before I could use it in my code. However, I understand that to produce a spanified design, boost would have first to introduce a mature span implementation + extensions that can be leveraged for this type of code design. *Security* I think for things like HMAC, there is a question of allocators that zero-out memory, of secure obfuscated memory content in case of crash dumps, etc, that this library has not been attentive to, so I thought it was worth mentioning. *In conclusion* I'm here to support the authors. I don't feel like voting either accept or reject, because I think Boost as a whole has not provided the mission critical answer to that. I want this library to succeed. I just wish it was a bit more ambitious in its goal on the library-design space. Claudio.

13 107

[hash2] HashAlgorithm's named requirements for result()
by Vinnie Falco 07 Dec '24

07 Dec '24

On Fri, Dec 6, 2024 at 4:24 PM Ivan Matek via Boost <boost(a)lists.boost.org> wrote: > Goal was to "encourage" users in 2 ways to call result() only once. First > way is that move signals that value is "tainted", second is that clang-tidy > can detect double moves sometimes. > I propose to change HashAlgorithm requirements as follows: --- HashAlgorithm result() is renamed to result_type finalize(); The current documentation for finalize [1] is moved elsewhere and replaced with the following text: This function shall return the final hash value of the input message, where the input message is defined by the ordered sequence of bytes provided in all prior calls to update(). The behavior of subsequent calls to finalize() is undefined unless specified by the HashAlgorithm. --- Rationale: The name "finalize" is closer to established practice and is more suggestive of the typical state mutation: https://github.com/openssl/openssl/blob/5fce85ec52a826d53665552b50e67f86c92… The existing Hash2 documentation for result() is too specific and suggests operations which may not be relevant such as for FNV-1a or other byte-oriented algorithms. It would be better to state only the mandatory requirements and leave the rest to the implementation. The existing Hash2 documentation suggests the possibility of calling finalize() twice or more in a row, yet this can only be considered safe, secure, or otherwise generally following best practices on a case by case basis depending on the HashAlgorithm. The HashAlgorithm concept is first and foremost designed for use with the Hash named requirement (std::hash and related). Making calls to a generic finalize() undefined does not hinder this use-case. There has been no research into whether HashAlgorithm should be held up as the concept for how ALL hash algorithms should be modeled. Therefore the named requirements for HashAlgorithm's finalize() function should only go as far as needed to support the Hash use-case, and we should leave the rest of its behavior up to the implementer. In particular I think there is danger here: template< typename HashAlgorithm > auto double_finalize( HashAlgorithm& h ) { h.finalize(); return h.finalize(); } This is dangerous because the usage in generic contexts results in an unpredictable quality of result. We should use the more strict definition I provided above for now, and only in the future loosen the definition if there is evidence that doing so yields a net benefit. It is always easier to go from strict to loose. And going from loose to strict after the fact is difficult and often impossible without breaking things. It is better if double_finalize is undefined. Users who want random numbers or whatever, can do it with a specific implementation of HashAlgorithm which offers additional guarantees for finalize(). [1] https://pdimov.github.io/hash2/doc/html/hash2.html#hashing_bytes_result

2 2

Any interest in a faster "any" type alternative?
by Marcus Tomlinson 06 Dec '24

06 Dec '24

I’m aware that the current implementation of any intentionally avoids copy assignments and uses only copy construction for all value updates. I believe this is due to the fact that copy assignment isn’t always available for all types - understandable. My proposal is what if we had a separate “fast-any” type that does utilise copy assignment when types match. In my own usage of this new type, I’ve found massive (roughly 2x) performance improvement on consecutive assignments of the same type. I’d like to gauge whether there would be any interest in adding such a type to the Boost library. Source code: https://github.com/cross-platform/fast_any Regards, Marcus Tomlinson

2 2

[release] The Master branch is now closed for ALL changes
by Marshall Clow 05 Dec '24

05 Dec '24

Please do not commit to the master branch. — Marshall

1 0

[release] Master branch is now open for bug fixes
by Ion Gaztañaga 04 Dec '24

04 Dec '24

Hi, The master branch is now open for bug fixes, test and documentation changes, but please follow the policy described in: https://github.com/boostorg/wiki/wiki/Releases%3A-Beta-Merge-Policy The next deadline: On December 4th, master closes for all changes. As always, the calendar is at https://www.boost.org/development/ — The release managers

9 22

[release] Reminder: Master Branch closes for all changes TOMORROW December 4th
by Marshall Clow 03 Dec '24

03 Dec '24

For the 1.87.0 release. The calendar is here: https://www.boost.org/development/index.html — Marshall P.S. If you have change notes, don’t forget to add them to the website.

1 0

Boost 1.87-beta1 still has sha1 failure in boost.compute
by Benjamin Redelings 02 Dec '24

02 Dec '24

Hi, It looks like boost 1.87-beta1 still has the compile failures from version 1.86 in boost::compute::detail::sha1. I see that there are patches from Aug 21, 2024 to address this in the `develop` branch of `boost.compute`. Are these likely to make it into boost 1.87? -BenRI

3 3