September 2020 - Boost - lists.preview.boost.org

CMake Announcement from Boost Steering Committee
by Jon Kalb 13 Aug '24

13 Aug '24

The libraries produced by the Boost community have had a greater impact on the way that the C++ community writes code than any other library implementation. The focus of the Boost community will always be on the libraries, but it is undeniable that we are dependent on and often limited by the infrastructure of our trade. Years ago, the move to Git was contentious; yet, it was required to improve development. In a similar vein our build system has become an impediment for many developers and users, existing and prospective. Therefore, we, the Steering Committee, announce to the Boost community our desire and intent to move Boost’s build system to CMake for users and developers alike. We are soliciting comments and proposals from the community to guide the process and the goals. Our desire is that the community can come to consensus by the end of the calendar year with a vision of supporting users and developers. We understand that the actual process of achieving the end goal will be long, time consuming, and not without its share of conflict. We also recognize that the Boost community is comprised of bright and passionate individuals that are eager to get involved and help get work done. The members of the Steering Committee have been encouraged by the discussions and activity surrounding CMake on the mailing lists over the years and know that many people have voiced visions. We hope that each of you rejoins the discussion to support this initiative and contributes to the common goal of improving Boost’s integration into the broader C++ ecosystem. For the Boost Steering Committee, Jon Kalb, Chair jonkalb(a)boost.org

49 199

Boost.Text is looking for a (mini-)review manager
by Zach Laine 07 Dec '22

07 Dec '22

I've made all the recommended changes to Boost.Text that came up in the review. Glen, the previous review manager, recommended a mini review of Boost.Text once I'd made those changes. If you'd like to manage the mini-review, please let me know. For everyone, including potential reviewers and review managers, here are the major changes: * All the specific review feedback issues have been addressed. * The string layer is now gone. * The text layer got a major re-work. It is now a set of template-based adaptors (basic_text<>, vs. a text typedef. etc.), which lets everyone get what they want. You can change the underlying storage type (including whatever allocator you prefer), select UTF-8 or UTF-16, and change the normalization form. Glen recommended I drop this layer, but the library is way less useful without this stuff, and these changes address all the concerns raised about the text layer in the review. The updated docs can be found here: https://tzlaine.github.io/text/doc/html/index.html Oh yeah, and I added concept constraints to the whole library when you build in C++20 mode. Zach

17 46

[review][JSON] Review of JSON starts today: Sept 14 - Sept 23
by Pranam Lashkari 09 Aug '22

09 Aug '22

Boost formal review of Vinnie Falco and Krystian Stasiowski's library JSON starts today and will run for 10 days ending on 23 Sept 2020. Both of these authors have already developed a couple of libraries which are accepted in Boost(boost beast and Static String) This library focuses on a common and popular use-case for JSON. It provides a container to hold parsed and serialised JSON types. It provides more flexibility and better benchmark performance than its competitors. JSON highlights the following features in the documentation: - Fast compilation - Require only C++11 - Fast streaming parser and serializer - Easy and safe API with allocator support - Constant-time key lookup for objects - Options to allow non-standard JSON - Compile without Boost, define BOOST_JSON_STANDALONE - Optional header-only, without linking to a library (a point I would like to add in highlight: it has cool Jason logo 😝) To quickly understand and get the flavour of the library take a look at "Quick Look" <http://master.json.cpp.al/json/usage/quick_look.html> You can find the source code to be reviewed here: <https://github.com/CPPAlliance/json/tree/master> You can find the latest documentation here: <http://master.json.cpp.al/> Benchmarks are also given in the document which can be found here: <http://master.json.cpp.al/json/benchmarks.html> Some people have also given the early reviews, the thread can be found here: <https://lists.boost.org/Archives/boost/2020/09/249745.php> Please provide in your review information you think is valuable to understand your choice to ACCEPT or REJECT including JSON as a Boost library. Please be explicit about your decision (ACCEPT or REJECT). Some other questions you might want to consider answering: - What is your evaluation of the design? - What is your evaluation of the implementation? - What is your evaluation of the documentation? - What is your evaluation of the potential usefulness of the library? - Did you try to use the library? With which compiler(s)? Did you have any problems? - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? - Are you knowledgeable about the problem domain? More information about the Boost Formal Review Process can be found here: <http://www.boost.org/community/reviews.html> Thank you for your effort in the Boost community. -- Thank you, Pranam Lashkari, https://lpranam.github.io/

21 73

Adding my sorting algorithm, ska_sort, to boost, attempt 2
by Malte Skarupke 24 Nov '21

24 Nov '21

Hi everyone, I started a discussion about this in October, but at the time the maintainer of the sort library was busy, then I was busy, so now it's January... I wrote a generalized version of radix sort a couple years ago and gave a talk about it here: https://www.youtube.com/watch?v=zqs87a_7zxw I would like to get a new version of that sorting algorithm into boost. It's faster than what I showed several years ago. It's also even more general and has a new interface that should be easier to extend. I uploaded the code here: https://github.com/skarupke/sort/tree/master The documentation is also already updated with new benchmark results. Let me know what's required to get this into boost. Would you like this in the form of a pull request? One of the big things that probably needs review is the interface, since radix sorting works differently than comparison based sorting. For most normal uses you just need to provide a sort key, but more complicated cases can get more complicated. I tried very hard to make everything easy, but I'd also be curious if others can come up with better ways of doing the interface. Thanks, Malte -- Malte Skarupke malteskarupke(a)fastmail.fm

8 10

[FPR] Review of FPR starts Today: Sept 28 - Oct 7
by Benedek Thaler 16 Oct '20

16 Oct '20

The Boost formal review of the FPR (Flat Precise Reflection, ex Magic Get, ex PODs Flat Reflection) starts Today, taking place from September 28, 2020 to October 7, 2020. The library is authored by Antony Polukhin, author of Boost.DLL, Stacktrace, Type Index libraries. Documentation: http://apolukhin.github.io/magic_get/ Source: https://github.com/apolukhin/magic_get The library is meant for accessing structure elements by index and providing other std::tuple like methods for user defined types without any macro or boilerplate code. The FPR documentation summarizes the out-of-the-box added functionality for aggregate initializable structures: - comparison operators - heterogeneous comparators - hash - stream operators - access to members by index - member reflections - methods for cooperation with std::tuple - methods to visit each field of the structure If the description isn't immediately obvious for you, here's a motivating example: // requires: C++14 #include <iostream> #include <string> #include "boost/pfr/precise.hpp" struct some_person { std::string name; unsigned birth_year; }; int main() { some_person val{"Edgar Allan Poe", 1809}; std::cout << boost::pfr::get<0>(val) // No macro! << " was born in " << boost::pfr::get<1>(val); // Works with any aggregate initializables! } Please provide in your review information you think is valuable to understand your choice to ACCEPT or REJECT including FPR as a Boost library. Please be explicit about your decision (ACCEPT or REJECT). Some other questions you might want to consider answering: - What is your evaluation of the design? - What is your evaluation of the implementation? - What is your evaluation of the documentation? - What is your evaluation of the potential usefulness of the library? - Did you try to use the library? With which compiler(s)? Did you have any problems? - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? - Are you knowledgeable about the problem domain? More information about the Boost Formal Review Process can be found at: http://www.boost.org/community/reviews.html Thank you for your effort in the Boost community. Benedek - review manager of the FPR library

8 12

traits
by Gero Peterhoff 03 Oct '20

03 Oct '20

Hello Peter, I've designed a couple of traits here. Is that right so far so that they can be included in boost? But there are these problems: 1) char8_t is still missing in your is_unsigned 2) my implementations of - is_native; if the type exists natively on the platform - is_vectorisable; if the type can be vectorized on the platform (simd) - is_integer are still incomplete, as I cannot find out whether the corresponding types are available; e.g. on some platforms there is int56_t, but no int8_t (__CHAR_BIT__ > 8), etc. The specializations of is_logic and is_rational should better be done in the libs. If a concept is created, it is stored in boost::mata so that there are no naming conflicts. Similar to math::constants there is also a string::constants; maybe also makes sense. regards Gero

2 3

new type traits
by Gero Peterhoff 03 Oct '20

03 Oct '20

Hello Developers, I would like to retrofit some type-traits in boost, but I have read https://pdimov.github.io/articles/phasing_out_cxx03.html. Now I don't know how to do it best, as the boost-type-traits may be removed. Would the following procedure make sense? We get the std-traits and concepts in a namespace boost::meta, eg file boost/type_traits/meta.hpp: #ifndef BOOST_META_HPP #define BOOST_META_HPP #include <type_traits> #if defined(__cpp_concepts) #include <concepts> #endif namespace boost::meta { template <typename Type> using is_integral = std::is_integral<Type>; template <typename Type> inline constexpr bool is_integral_v = std::is_integral_v<Type>; template <typename Type> using is_pointer = std::is_pointer<Type>; template <typename Type> inline constexpr bool is_pointer_v = std::is_pointer_v<Type>; ... all other #if defined(__cpp_concepts) template <typename Type> concept integral = std::integral<Type>; template <typename Type> concept pointer = is_pointer_v<Type>; ... #endif } // boost::meta #endif // BOOST_META_HPP Then you could possibly map this in boost/type_traits.hpp to boost:: (if that makes sense): #ifndef BOOST_TYPE_TRAITS_HPP #define BOOST_TYPE_TRAITS_HPP #include <boost/type_traits/meta.hpp> remove boost implementations namespace boost { using namespace boost::meta; } // boost #endif // BOOST_TYPE_TRAITS_HPP In this way, new traits can be added in boost/type_traits/meta.hpp that are based on the std-traits, e.g. is_logic namespace boost::meta { template <typename Type> struct is_logic : std::false_type{}; template <typename Type> inline constexpr bool is_logic_v = is_logic<Type>::value; template <> struct is_logic<bool> : std::true_type{}; #if defined(__cpp_concepts) template <typename Type> concept logic = is_logic_v<Type>; #endif } // boost::meta and in this case add in boost/logic/tribool.hpp namespace boost::meta { template <> struct is_logic<boost::tribool> : std::true_type{}; } // boost::meta Or do you have a better idea? regards Gero

6 15

[PFR] The user's perspective
by Andrzej Krzemienski 02 Oct '20

02 Oct '20

Hi Everyone, I am now trying to study the PFR library, and wanted to share some thoughts and see what others think. First, thanks Antony for writing and sharing this library. I have watched your video on it, read the docs, and I find it amazing that you were able to implement such reflection in C++. This is in the spirit of the good old Boost libraries that just implemented magic. Now, to the critique. From looking at the docs and the videos, I inferred (perhaps incorrectly) that the library was created in this order: "We can do type introspection for aggregates, so let's see to what extent we can provide an interface of a reflection library." The extent is quite amazing (e.g., I really appreciate having pfr::less<> out of the box), but to the end user, I think, the library will still appear as incomplete. That is, when my type is an aggregate it will work, but when I need to add a convenience constructor it will immediately break, and I will have to switch to another library, like Peter's Describe ( https://github.com/pdimov/describe/tree/master) With Describe, I will have to type more, but I have a guarantee of a uniform experience both for aggregates and for other types. And given that I have to use Describe at some point there may be no point in also keeping PFR and forcing on myself and my colleagues two different interfaces. Describe is not a perfect solution either, because for aggregates I have to declare the mapping, which is unnecessary as PFR has demonstrated. My ideal solution would be if these two libraries (PFR and Describe) would be merged into one, so that: 1. They provide the same interface 2. A user can easily instruct the library: "this is an aggregate: figure out the mapping yourself". I am sorry if it sounds arrogant: people are devoting their private time to write and share a cool library, and all they hear back is "do more." But I thought it would also be unfair not to give this feedback. I am also not sure if the goals of the two libraries are compatible (e.g., Describe is also interested in giving names of the members) . I would be curious to see what the authors think about it. Regards, &rzej;

5 6

Cache Size
by Matt Borland 01 Oct '20

01 Oct '20

A recurring discussion in Boost.Math has been needing a cross-platform way to query the L1D cache size of a machine. I have most of a solution in a [personal repo](https://github.com/mborland/cache). It did not make sense to us to have this in Boost.Math because it would be useful outside of that scope, but we did not know where it should go. Is there an existing library where this would fit? If necessary I can roll it into a library submission that exposes the functionality of CPUID. Matt

3 3

[review][json] My review
by Gavin Lambert 30 Sep '20

30 Sep '20

- Do you think the library should be accepted as a Boost library? Yes, my vote is to ACCEPT this library. - What is your evaluation of the design? Much has been debated recently on the merits of push parsers vs. pull parsers. Boost.JSON provides both a push parser (SAX-like interface) and a DOM parser (which is somewhat pull-like but I don't think is technically either kind). It does not provide a "real" pull parser, which are sometimes more efficient for deserialisation into custom types; the SAX parser can somewhat do that but is not composable, which makes it harder to use in practice, unless I'm missing something (this is usually why I end up never using SAX parsers). If so, I'd like to see a more complete example where the handler delegates part of the parsing to a class that in turn delegates part to another class (as with reading typed sub-values). I don't think this is currently possible. So I think I can understand some of the debate around this point; while I don't think it's strictly necessary that this particular library implement that, if it doesn't then another library will end up having potentially competing types and the natural Boost.Json name will already be taken. Personally, however, I don't think this is sufficient reason to block acceptance of this library. Direct pull parsing doesn't mesh well with JSON, due to the requirement to tolerate reordered keys, so any non-defective parser would have to have something DOM-like under the hood anyway. (Some kind of cache, if not necessarily a traditional DOM.) As such it should be sufficient to use json::value itself as part of a pull-like parsing interface. Regarding the SAX parser specifically, I dislike the *_part methods in the callback interface. I feel that these should be removed and only the final form retained, called only when the complete text of the value has been determined. I don't feel it is ever sensible to call back with a partially-parsed value (unless I am misunderstanding what these are for, but there is little explanation in the docs, and I wasn't able to get them called at all in my test). I also don't see the point of supplying a size argument to on_key* and on_string, as this can be read from the string_view argument instead. The various number methods don't pass this separately. Similarly, having a bool return as well as an error_code output argument seems peculiar; one or the other should be sufficient, as is it just invites inconsistency. parser.write() is incremental but basic_parser.write() is not, which seems odd. I assume this means that parser is storing an incrementally populated buffer of the full document rather than _really_ incrementally parsing. parse/parser do not appear to support consuming only part of the supplied input as a complete JSON value (returning how much was consumed), which is useful for streams where multiple objects are concatenated without explicit separation. Regarding other aspects of design, I haven't drilled down in great detail but the basic interface seems reasonably sound, other than the concerns raised above. I have mixed feelings on the initializer-list construction method -- I like that there is some method of doing this, and it seems like idiomatic C++ (particularly regarding how key-value pairs appear), but also very non-JSON -- an array of objects ends up having three nested braces. But I don't have any specific suggestion for improvement. - What is your evaluation of the implementation? It seems unfortunate (though understandable, due to heap allocation) that you cannot declare a json::value / json::object as constexpr, for cases where you do want some kind of constant reference object (e.g. a schema description). You can somewhat work around this by declaring the JSON as a constexpr string_view and later parsing it to a non-constexpr json::value, but that seems a bit awkward (and potentially bad-perf-prone if someone parses it multiple times). Otherwise I didn't really look into the implementation at all. - What is your evaluation of the documentation? Overall, good, if perhaps a bit sparse on some aspects. One pet peeve however is that most of the examples appear to be written as if "using namespace boost::json" is in effect, which I feel both encourages bad practice and makes it harder to distinguish which types are provided by the library vs. exist only for the example. This is especially confusing in the examples for tag_invoke, but not only there -- another example of confusion are the aliases for types present in both boost and std namespaces. I strongly recommend going through and changing these to explicitly use a json:: namespace prefix on all library types (including the boost/std aliases). Brevity is useful, but not at the expense of clarity. The "Using Numbers" section of the docs appears to be entirely blank. "serializer can be used to incrementally serialize a value incrementally" (redundant words are redundant) (On a related note, an example use case for serializer is listed as "ensure a fixed amount of work is performed in each cycle", but the only means of control is the length of the provided buffer. I guess this is equating work to number of characters output, which seems dubious. But I have no specific objections.) - What is your evaluation of the potential usefulness of the library? The library seems well suited for JSON as inter-process/network data interchange, other than perhaps the omission of partial input parsing. It seems less suited for JSON as configuration file -- while it can parse such files (via common extensions such as ignoring comments and trailing commas) it lacks means to pretty-print and to round-trip the comments, which would be essential if writing back a config file intended for human use. However while supporting this use case would be nice, I don't consider it mandatory. - Did you try to use the library? With what compiler? Did you have any problems? I tried it out with MSVC 2017 in standalone header-only mode, though only with a few small examples. The basic_parser examples do not compile out of the box -- with the suggested json.hpp and src.hpp includes there are link errors with basic_parser symbols. Additionally including basic_parser.hpp resolves these but starts complaining about missing max_{key,string,array,object}_size members of the handler type, which are not documented. After these are supplied then it works as expected. I don't see why these should be needed, however. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? Roughly four hours over a few days reviewing docs and trying some small experiments. - Are you knowledgeable about the problem domain? I have some apps that use JSON as both interchange and configuration storage formats, although most of these do not use C++. So, "somewhat".

9 23

Review Manager needed for Boost.Lambda2
by Peter Dimov 30 Sep '20

30 Sep '20

Greetings, Some time ago I wrote a blog post, https://pdimov.github.io/blog/2020/07/22/a-c14-lambda-library/, which described how a simple lambda library can be implemented in ~50 lines using the facilities already available in C++14's <functional> standard header. I turned this into a mini-library, https://github.com/pdimov/lambda2, and I'd like to submit that to Boost. I'm looking for a review manager. The library is trivial, so managing the review should be as easy as theoretically possible for a Boost review. Thanks in advance.

7 20

new macros
by Gero Peterhoff 28 Sep '20

28 Sep '20

Hello Developer, can you please provide these 2 macros: - BOOST_HAS_LIB_INT128 - BOOST_NO_INT8_T and there is another bug in boost::is_unsigned with char8_t. See https://godbolt.org/z/o8z1aq thx Gero

1 0

[review][JSON] json::value as a vocabulary type
by Andrzej Krzemienski 26 Sep '20

26 Sep '20

Hi Everyone, I have heard this claim a number of times: json::value is suitable for a vocabulary type. I am not sure what that actually means: no templates? Guarantee that layout or mangled symbol will never change? Anything else? My understanding of a "vocabulary type" is that it should be usable (not necessarily with maximum efficiency) for *any* usage. In the case of JSON that would mean that I should be able to represent any value that corresponds to a valid JSON when converted to text. I do not think that json::value can claim that without the ability to serialize arbitrarily big numbers. I understand that the goal of the library is to address the most common cases, and big numbers do not fall into this category. I am just saying that the name "vocabulary type" may not be accurate here. Regards, &rzej;

11 25

Changing virtual memory amount in Appveyor and Travis CI tests
by Edward Diener 25 Sep '20

25 Sep '20

Is it possible through .yml file for Appveyor and Travis CI to change the amount of virtual memory when running a CI test. I am running CI tests for Boost PP which heavily use the preprocessor, and some of the extended tests I am running fail on both Appveyor and Travis CI with out of memory conditions while on my own local computer, with about 32 GB of ram, the tests succeed. I think in the real world most users now have at least 32 GB of ram or more, while these virtual environments in Appveyor and/or Travis CI probably run their tests with much less overall virtual memory than an average user would have.

4 12

Re: [boost] Endorsement, review manager sought for Boost.Describe
by Peter Dimov 24 Sep '20

24 Sep '20

> Vinícius dos Santos Oliveira wrore: > > Hi Peter, > > > > could you consider using "compile-time strings" such as Hana's? I don't > > have much use for a const char* in TMP algorithms. > > The descriptor strings are "compile-time". This, for instance, works. ... Here's how you can get a Hana string: #include <boost/describe.hpp> #include <boost/mp11.hpp> #include <boost/hana.hpp> #include <cstddef> using namespace boost::describe; using namespace boost::mp11; enum E { v1 }; BOOST_DESCRIBE_ENUM(E, v1) constexpr std::size_t strlen_( char const * s ) { std::size_t n = 0; while( *s++ ) ++n; return n; } template<class D, std::size_t... I> constexpr auto to_string_impl( index_sequence<I...> ) { return boost::hana::string_c<D::name[I]...>; } template<class D> constexpr auto to_string() { return to_string_impl<D>( make_index_sequence<strlen_(D::name)>() ); } using D = mp_first< describe_enumerators<E> >; constexpr auto st = to_string<D>(); #include <typeinfo> #include <cstdio> int main() { std::printf( "%s\n", typeid(st).name() ); }

1 0

Endorsement, review manager sought for Boost.Describe
by Peter Dimov 24 Sep '20

24 Sep '20

I would like to submit a C++14 reflection library, Describe, to Boost and I'm looking for endorsements and a review manager. The library is at https://github.com/pdimov/describe and the documentation is at https://pdimov.github.io/describe/doc/html/describe.html. Some examples of what the library enables are given at https://pdimov.github.io/describe/doc/html/describe.html#examples. The purpose of Describe is to establish a standard way of annotating (describing) user-defined types (enums, structs and classes), so that types can be described once, and then their descriptions can be used from other libraries. At the moment, the practice is for each library to invent its own ad-hoc annotation/description mechanism.

7 9

[nowide] cxx11_moveable_fstreams target
by Sean Farrow 24 Sep '20

24 Sep '20

Hi All, By way of context to the following query, I am trying to update the VCPKG boost port to v1.74. I noticed that the nowide library now has a jam file that has the following import statement: import ../../config/checks/config : requires ; Due to the way VCPKG works, I'm copying the config directory to where it is expected and then changing the import statement. But I then get a message that it can't find the cxx11_moveable_fstreams target. Does the nowide build process rely on any other parts of boost config? If not, where is the cxx11_moveable_fstreams defined? I should say that I'm new to the boost build system, so any help is apreciated. Thanks, Sean.

5 9

[review][JSON] The library name...
by Andrzej Krzemienski 24 Sep '20

24 Sep '20

Hi Everyone, Looking at the reviews of Boost.JSON, a number of them fall into a distinctive pattern: reviewers are saying "you did not implement important parts of JSON parsing/serializing", and the authors' reply is "this was not the design goal". My hypothesis is that this is caused by the library name. Simple "Boost.JSON" makes a promise (at least I read it this way) "this is Boost's response to JSON-related problems". And the library does not fulfil that promise. In a sense, the name misguides the potential users who need an answer, sometimes in less than 10 seconds, to the question, "is this library going to address my need?". Maybe the library would encounter a wider consensus if it was called Boost.JSON_Value, or Boost.JSON_DOM. And if the first introductory sentence in the documentation read, "this library provides a type with Regular interface that can represent JSON contents, and provides a convenient way to serialize and then deserialize it fast and possibly in chunks." Alternatively, maybe we can consider the reviewed library as a candidate for a sub-library in a more general module of JSON. Much like Boost.Math aggregates a number of math-related components, we could treat this library as one of a number of components in a more general Boost.JSON_Toolkit library. Regards, &rzej;

1 0

Proposal for adding C++ level to the meta/libraries.json
by Edward Diener 24 Sep '20

24 Sep '20

Since we are reviewing Vinnie Falco's JSON library I thought that this would be a good time to present a proposal to add information to each Boost library's meta/libraries.json file regarding the level of C++ standard compliance for that library. This could then be added to each library's visual documentation so that end-users would instantly know the C++ standard level they would need to use in order to use a Boost library. My proposal is to add 3 fields whose data would be the same as the cxxstd allowed values from Boost Build, using the first value of each choice ( currently 03, 11, 14, 17, 20 ). standard = minimum C++ level for the library extended = same functionality in the library as the minimum C++ level but with extended use given higher C++ levels required = new functionality in the library above the minimum C++ level which requires given higher C++ levels The 'standard' field would be a single value, while the 'extended' or 'required' fields could be more than one comma separated value. If you don't like the names for 'standard', 'extended', or 'required' you can bikeshed the name, although I think the 'standard' name is pretty well apparent. The idea is to provide this information in the meta/libraries.json field for each library, so that the end-user of the library can immediately know the usability of the library with appropriate C++ standard levels of compilation. I realize that the line between the 'extended' and 'required' entries, as I have described them, could be blurred. The 'standard' field remains the most important. I have long felt that the end-user should not have to do any investigation, in code or in documentation, just to determine if a library is usable for his C++ standard level of compilation.

12 27

[review][JSON] Ruthgers' review
by Ruthger Dijt 23 Sep '20

23 Sep '20

> Do you think the library should be accepted as a Boost library? It will be a good addition to Boost if this library gets ACCEPTed > What is your evaluation of the design? It works, is straightforward and the interface feels good to use, the only pitfall is getting the right type you want; number_cast took me a second to realize it existed and the custom string class is something the will most likely be converted alot to std::string in an average users' code. > What is your evaluation of the implementation? As a casual user I cannot comment on this, but as I used it in stand alone / header only mode without problem I feel that i should be alright under the hood > What is your evaluation of the documentation? Could be better, especially in the examples department; it could use how-to basics > What is your evaluation of the potential usefulness of the library? Very useful > Did you try to use the library? With what compiler? Did you have any > problems? I have implemented it in a real project where RapidJSON was used before. No real problems found aside from getting exceptions because .as_double() is not able to get a technically integer from the json (which happens in user configs, example: { "test": 20 }) but that quickly resolved itself with number_cast, Ideally any number should be able to become a double in my opinion. Clang 10.0.1 was used. > How much effort did you put into your evaluation? A glance? A quick > reading? In-depth study? As stated above implemented in one project which took about an hour. > Are you knowledgeable about the problem domain? No, this review is coming from a pure casual user kindly, Ruthger Dijt

2 1

[json][review] JSON review
by Harshada Wighe 23 Sep '20

23 Sep '20

We vote to reject the JSON library from Boost and our review of the library and reason for rejection are below: DOCUMENTATION The documentation is very good. The usage and examples and reference are easy to understand and contain all the information we are looking for. IMPLEMENTATION The organization of source code is very messy: 1. Some files do not include all headers at the very start of the file but have some include statements at the end. For example the json/object.hpp file has 10 include statements at the start of the file before the opening namespace and then 1 include statement at the end of the file after the closing namespace. This is very confusing and not a good practice. 2. Splitting the files into HPP and IPP is also not good because it spreads the class into more files than necessary and causes compilation to be slower. Using the library in a project has a complication: 3. The library is complicated to use in a project because you cannot simply include headers in any file you want. You have to include a src.hpp file but this file must be included only once. Normal header libraries in Boost just let you include the headers without this strange requirement. The library adds a lot of size to your executable: 4. Unlike other JSON libraries which are smaller, this adds several KB to the exe size. This is possibly because of use of templates or because it contains very big lookup tables. The performance is not as fast for all use cases: 5. Compared to our in house JSON library, this library was slightly slower. We have very simple but very large JSON data. We use objects and arrays, but only integer numbers with no fractional part and very large ASCII strings, and no booleans and no nulls. In one of our standalone conversion tools we tried using this library and found it had a 4% longer runtime on a small 170 MB file. The parser allocates too much memory: 6. I think this is because the design is not optimized for the case when all the JSON content is already stored in memory. In our parser we point to the strings in the original buffer. The library duplicate code that is in the C++ compiler libraries: 7. In json/config.hpp there are implementations of move and remove_cv from the C++ standard. This introduces more duplicate of this functionality into programs. Besides these points the implementation is well written. DESIGN The allocator design is not standard: 1. The library has a different allocation system that is not the C++ standard system for allocators. We did not understand why storage_ptr was needed at first. It would be better for the library to just template on allocator just like the vector class in the C++ standard. The allocator system also uses virtual calls. The C++ standard vector class also does not have this problem when it uses the standard allocator system. Other than these points the design is ok. OVERALL We would not use this library because it does not offer anything special for JSON that we need. It also uses some designs that are not standard practice. We use JSON heavily in our historical market quote service and most of our tools deal with JSON files and we use many Boost libraries so we are hoping for a JSON library in Boost that we could use instead of having our own.

3 2

[json][review] Scott Determan review
by Scott Determan 23 Sep '20

23 Sep '20

I recommend ACCEPTING the proposed json library. - What is your evaluation of the design? Design is very good. The parser interfaces and object building interfaces meet my needs. I especially want to call out the pmr support and it's support for either sharing or transferring ownership of resources. It's unfortunate that this required a non-standard string class, but I think supporting shared ownership is worth it. I also like the use of tag_invoke for customization points. Apart from libunifex, this is the first library I've seen that uses this mechanism. I think this is the right way to do customization points. - What is your evaluation of the implementation? I looked at the value, storage_ptr, and object classes. It appears to be a very high quality implementation. However, I did this only as a sanity check, I did not spend much time looking at the implementation. The benchmarking numbers also point to a high quality implementation. - What is your evaluation of the documentation? Documentation is very good. Message for Vinnie: One minor nit caught my eye: My understanding is when implementing a tag_invoke customization point it should be a hidden friend because it improves compile times by keeping overload sets smaller. If that is best practice (and it may not be; I'm not a tag_invoke expert), then the examples in the documentation should follow this. - What is your evaluation of the potential usefulness of the library? It covers a widely useful range of json functionality and I expect this library to be widely used. If accepted I intend on using it immediately. - Did you try to use the library? With which compiler(s)? Did you have any problems? Yes: gcc 10.2 and clang 10.0.1; Linux; No problems. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? Read the documentation, browsed the code, wrote some toy programs. Spent about half a day on it. - Are you knowledgeable about the problem domain? As a library consumer, I am familiar with nlohmann_json as well as the custom json library used in my current work project. I am not familiar with the problem domain from a library writer point of view.

2 1

[review][json] Boost.JSON review
by Ruben Perez 23 Sep '20

23 Sep '20

Hi all, My recommendation is to ACCEPT Boost.JSON into Boost. Parsing and serializing JSON using a DOM representation is a common enough task to have a specialized library for it in Boost. Most other languages provide built-in support for this task. I see Boost as the repository where I look at when the standard doesn't provide me something it should, so having a DOM JSON library in Boost is a must for me. - What is your evaluation of the design? Generally sound. I especially value its simplicity. Concretely: 1. The memory management model is extremely simple. I wonder if storage_ptr could be moved somewhere else long term so other libraries can use it. 2. Using concrete types for numbers (uint64_t/int64_t/double) may not fit everyone's needs but will definitely fit most and makes things easy. I miss some way of accessing json::value from generic code, but I understand visitation is currently in progress. As a user I would also be satisfied with generic versions of get_xxx, is_xxx and so on: ```cpp json::value val = /* ... */ val.get<json::object>() // or json::get<json::object>(val) ``` I also miss some way of determining the location (line and column) of parse errors. I don't know how this feature would interact with performance though. Nice to have but not required for me. I would have also liked `json::kind` be streamable, as it makes logging/debugging easier in some contexts. Nothing I can't live without, though. Finally, I would also appreciate `json::string` have a straightforward way to be converted to `std::string` (e.g. by having a `to_std_string()` member function). - What is your evaluation of the implementation? I have just looked at the interface. - What is your evaluation of the documentation? The documentation is very good and detailed, nice work. I would say the main point of improvement would be `basic_parser`, as I had a hard time understanding when some of the callbacks get called, especially the `on_xxx_part` ones. It would be also good to know if accessing the `basic_parser` state (e.g. `depth()`) is allowed from within the handler. And if it is, maybe passing the parser instance at the beginning of the parse could be useful (e.g. in the `on_document_begin`, or in a separate hook). Ideally I would like to see an example of `basic_parser` used to parse a custom struct, if it's not excessively complex. Also note that `basic_parser` docs say that it's defined in `<boost/json/detail/basic_parser.hpp>` and that the convenience header is `<boost/json.hpp>`. Instead, it would be good to note that the user should include `<boost/json/basic_parser.hpp>` and that `<boost/json.hpp>` is not enough (I understand for build speed reasons). I've noted that streaming a `json::string` outputs the string within double quotes. It surprised me at first, as I was working with the 'json::string equals std::string' mentality. I guess this behavior makes sense, but I would make a note in the 'Using strings' section. 'See also' section for https://master.json.cpp.al/json/ref/boost__json__value_from_tag.html renders functions without separation, looks strange. I would make a note in the Value conversion mechanism referencing P1895 on `tag_invoke` so users who don't know this paper don't stare at the name wondering 'why?' As a final note, googling 'Boost json' renders this link to an old version of the docs: http://vinniefalco.github.io/doc/json/index.html Consider removing it so users don't get confused (I did at the very beginning). - What is your evaluation of the potential usefulness of the library? I think it will be incredibly useful. Parsing to DOM and serializing from DOM are very widespread tasks, and having a simple, well designed `json::value` container makes things much better. I would have used it for my use cases if it had been available. - Did you try to use the library? With which compiler(s)? Did you have any problems? I used the library in the Boost version and separate compilation model. I built and installed the library using CMake 3.16.3 and g++ 9.3.0 on Ubuntu 20.04, with no problems. I then built several toy examples with the library's CMake `find_package` interface. No problems. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I spent around 10h reading the documentation and building examples. - Are you knowledgeable about the problem domain? I have used JSON as a data exchange format in the embedded, financial and web worlds, in C++ and other languages (JavaScript, Python and PHP). In C++ I have mainly used rapidJSON, and I would replace it with Boost.JSON for future developments. I have no experience writing JSON libraries myself. Thanks, Ruben.

3 3

Re: [boost] [review][json] My review
by Gavin Lambert 23 Sep '20

23 Sep '20

At 00:22 24/09/2020, Vinnie Falco wrote: >On Wed, Sep 23, 2020 at 1:40 AM Gavin Lambert wrote: >> I've walked back on that one a little bit; I now think that it's >ok for >> basic_parser to do that (to retain a zero-allocation guarantee) >but in >> that case it should treat numbers the same way (which it >currently >> doesn't), and ideally there should be a standard >slightly-higher-level >> parser that has a similar interface but does preassemble the >values, >> eliminating the _part callbacks (for people who prefer ease of >use over >> raw performance). > >I had that originally, it was called number_parser, it was a >public >interface, and it was one of the first things to go during our >3-month >journey of optimization. You can fish it out of the commit log if >you >care to see it. I didn't fish out the code, but given the class name, that sounds like the complete opposite of what I was suggesting. To restate again, my suggestion is: 1. Remove on_int64/uint64/double from basic_parser's handler type. 2. Add on_number(string_view, size_t) to basic_parser's handler type -- having exactly the same semantics as on_key/string and their _part companions. 3. Add custom_parser, which wraps a basic_parser but has internal (dynamically allocated if needed) storage for partial keys/strings/numbers so it never calls _part methods on its own handler type. It still uses on_number(string_view) but note that this (and on_key/string) have no reason to pass a size_t any more, because it can never be any different from the size() of the string_view. 4. The int64/uint64/double parsing code that used to be in basic_parser should be moved to a dedicated helper class, in turn used by parser and *optionally* used by the user's handler implementation for basic_parser/custom_parser. This design has the proper separation of concerns between the parser layers (not dictating a numeric storage format, and keeping number parsing separate from JSON structural parsing). It should not negatively impact performance of parser, and should positively affect performance of basic/custom_parser consumers who didn't want the default conversion anyway.

2 1

[review][JSON] RJanson's review
by Janson R. 22 Sep '20

22 Sep '20

This is a review from a library user standpoint. Thanks to Vinnie, Krystian, and the other contributors for their work. > Do you think the library should be accepted as a Boost library? I recommend ACCEPTING this library into Boost. > - What is your evaluation of the design? The design takes some time to get used to, but it fits well to the problem being tackled (a JSON-DOM library). Switching my codebases using RapidJSON to this library made the code more readable and it will also allow me to get rid of a significant amount of boilerplate. > - What is your evaluation of the implementation? Looks solid. Boost-quality I find. > - What is your evaluation of the documentation? Solid. A month ago I still had to read the source code to find some details, but now all I need for my use-cases is in the docs, well explained. > - What is your evaluation of the potential usefulness of the library? Very useful; to me it takes the same place as Nlohmann.JSON and RapidJSON, both very popular and widely used libraries, but brings better performance and a better API. Regardless of the outcome of this review, I will use this library for my future JSON-DOM needs in C++. > - Did you try to use the library? With which compiler(s)? All in standalone mode on Debian: GCC 9, 10 Clang 9, 10, 11 [trunk of 03 Jul 2020] > - How much effort did you put into your evaluation? Converted two of my codebases (which have different use-cases for a JSON-DOM), read through the implementation, read the docs at multiple stages of its maturation over the last couple months. Amounts to about 40h. > - Are you knowledgeable about the problem domain? I do not have much experience designing nor writing libraries, however I do use a significant number of them, both Boost and non-Boost. JSON is my primary format for configuration files and web communication; In the past I have used Nlohmann.JSON and then primarily RapidJSON for its speed, despite its (in my eyes) terribly user-unfriendly interface. I have also used JSON a lot in other languages (TypeScript, PureScript, Java). -- RJanson

2 1

[JSON[review] Andrzej's review
by Andrzej Krzemienski 22 Sep '20

22 Sep '20

Hi Everyone, I recommend to *ACCEPT* the library into Boost. It meets the standard for BI am definitely going to use it. I tried to use the library in two configurations in header-only mode: g++ 9.2.0 with -std=c++2a on MinGW on Windows, g++ 10.2.1 on Fedora with -DBOOST_JSON_STANDALONE and -std=c++20 All worked fine. I studied library docs and played with examples for about 30 hours. I am not particularly knowledgeable about the problem domain. But my programs have to consume and produce JSON files. DOCUMENTATION It is very good, and meets the highest Boost standards. I mean both the tutorial and reference sections. While doing this review I had to look at the documentation of the competing libraries, and they are really bad. This is why I can appreciate the big effort you put in making these docs. There is a section "Comparison" that compares Boost.JSON to other JSON libraries. I personally think it should also mention the documentation. Some observations: 1. Add reference to RFC 7159 (https://www.rfc-editor.org/info/rfc7159) It allows implementations to set limits on the range and precision of numbers accepted. 2. The documentation for array object and index mention "Satisfies the requirements of ContiguousContainer, ReversibleContainer, and SequenceContainer" These requirements are never defined or referenced. The user will not even know if you defined them or if they are "commonly known". Maybe add link to cppreference.com: https://en.cppreference.com/w/cpp/named_req/ReversibleContainer https://en.cppreference.com/w/cpp/named_req/ContiguousContainer https://en.cppreference.com/w/cpp/named_req/SequenceContainer 3. String has two sections "Member Functions": https://master.json.cpp.al/json/ref/boost__json__string.html 4. Documentation for value::operator= seems corrupt: https://master.json.cpp.al/json/ref/boost__json__value/operator_eq_/overloa… * it has sections with no content * It is an unconstrained template, it look like it can accept any type 5. Examples of input JSON-s contain a lot of escape characters in string literals, like in https://master.json.cpp.al/json/usage/conversion.html This makes it difficult to read. Consider using raw string literals. Instead of assert( serialize( jv ) == "{\"x\":4,\"y\":1,\"z\":4}" ); We would have assert( serialize( jv ) == R"({"x":4,"y":1,"z":4})" ); 6. Some functions (like operator== or swap(x, y)) are naturally part of class interface, even though they are not member functions. It is annoying and confusing that rather than being listed along with the class they belong to they are instead listed separately as "free functions" The docs are not uniform about this issue. For `json::value` operator== is listed as friends along with other member functions, but not 2-argument swap(). 7. In https://master.json.cpp.al/json/usage/conversion.html we read: " Given the following definition of customer::customer( const value& ) [...] Objects of type customer can be converted *to* and from value". This "to" is surprising here. How can a constructor converting from `value` to `customer` could be used to convert `customer` to `value`? 8. As discussed in another thread, documentation would benefit from the info on implementation-limits-related design choices. a. Any json::value that you can build can be serialized and then deserialized, and you are guaranteed that the resulting json::value will be equal to the original. b. JSON inputs where number values cannot be represented losslessly in uint64_t, int64_t and double, may render different values when parsed and then serialized back, and for extremely big number values can even fail to parse. c. Whatever JSON output you can produce with this library, we guarantee it can be passed by any common JSON implementation (probably also based on uint64_t+int64_t+double implementation. DESIGN Clear, sound and practical. The design decisions correspond with my expectations of a library: fast, easy to use and learn, with value semantics. The interface is very intuitive, I practically do not have to consult the documentation: every functionality (serialization, value alteration) has the interface that I would expect. I appreciate that it targets embedded environments. Some observations: 1. The handling of infinities and NaN's needs to be addressed. Currently, I can do: `json::serialize(std::numeric_limits<double>::infinity())` and t produces an invalid JSON. You cannot easily make a precondition on json::value assignment/construction because you expose the double as a mutable member: void fun(double x) // precondition x is not infinity or nan { json::value v = x; // ok v.as_double() *= 2.0; // this may produce infinity } So maybe you need a precondition in the serializer. 2. Boost.JSON provides its own string type. This begs a lot of design questions for `json::string`. Is it a better replacement for std::string? std::string provides the following capability: ``` void set(std::string& s, char v, char x, char z, char k) { if (cond) s = {v, k}; else s = {v, x, z, k}; } ``` json::string does not. How do I do a similar thing with json::string? 3. string::swap() has a strange precondition: you cannot swap an object with itself https://master.json.cpp.al/json/ref/boost__json__string/swap.html This is a really unexpected and potentially dangerous, and at minimum requires a very strong rationale. This would be the first swap function in std or Boost that has a precondition. While discussing a somewhat related issue (on the guarantees of the moved-from state) in the LEWG, someone remarked that there was at least one STL implementation that did a self-swap for convenience of implementation. Is it really necessary? IMPLEMENTATION I didn't look at it much. Just some quick points I observed. 1. Values of type `double` representing Infinity and NaN can be stored in json::value and when later serialized, they render a malformed JSON output. json::value should either validate special values of `double` or indicate as precondition that they should not be assigned. Alternatively, the serializer should either validate this or indicate as precondition. 2. tag_invoke() takes tag arguments by reference, does it have to? Passing by reference requires the constexpr variables to be instantiated and their address assigned. 3. Customization through ADL-discovered overloads can cause ODR violations. There is no superior alternative though. But maybe put a warning in the docs that the user must make sure that all translation units must see the same set of overloads. 4. I do not like the 2-state-switch mechanics of BOOST_JSON_STANDALONE. It assumes that a user either wants a no-Boost version or that she wants a package: all Boost components. But what if I am using C++14 and I want to use std::error_code and Boost-emulated memoty_resource and string_view? What If I use Boost version and C++17 and want to use std::string_view? Maybe you can additionally provide macros for turning Boost-emulation libraries one-by-one? I mean, one for string_view, one for error_code, and one for memory_resource. 5. The header only mode -- I really appreciate it. It is good for initial experiments. Finally, I would like to thank Vinnie and Krystian for writing and sharing this library. Regards, &rzej;

3 4

[FPR] FPR review starts Sept 28
by Benedek Thaler 22 Sep '20

22 Sep '20

The Boost formal review of the FPR (Flat Precise Reflection, ex Magic Get, ex PODs Flat Reflection) will take place from September 28, 2020 to October 7, 2020. The library is meant for accessing structure elements by index and providing other std::tuple like methods for user defined types without any macro or boilerplate code. The library is authored by Antony Polukhin, author of Boost.DLL, Stacktrace, Type Index libraries. Documentation: http://apolukhin.github.io/magic_get/ Source: https://github.com/apolukhin/magic_get If the description isn't immediately obvious for you, here's a motivating example: // requires: C++14 #include <iostream> #include <string> #include "boost/pfr/precise.hpp" struct some_person { std::string name; unsigned birth_year; }; int main() { some_person val{"Edgar Allan Poe", 1809}; std::cout << boost::pfr::get<0>(val) // No macro! << " was born in " << boost::pfr::get<1>(val); // Works with any aggregate initializables! } See more at: https://github.com/apolukhin/magic_get/blob/develop/README.md Benedek

1 0

[JSON][Review] Another Boost.JSON review
by Hadriel Kaplan 22 Sep '20

22 Sep '20

Howdy, I vote to ACCEPT Boost.JSON into boost. I think it’s a fine implementation for what it’s intended for. I don’t understand some of the other reviews that reject it for not being something else - they got an apple and are complaining it’s the wrong color for an orange. It’s fine to prefer oranges over apples, but it’s also ok to offer apples for sale in your store even if you wouldn’t buy it yourself. A lot of people like apples - they’re simple and convenient, without having to peel away the layers of complexity involved in something more pulpy. (Ok, I’ve squeezed as much juice out of that metaphor as possible...) - What is your evaluation of the design? Good. It took me a bit to get used to it, because I was used to something else (Facebook’s folly::dynamic mostly); but that would be true of any new library. - What is your evaluation of the implementation? I didn’t study the internal details. I used it as a consumer of the library. I thought the user-facing API was reasonable. The one change I made to the code was to force it to use std::string_view instead of boost::string_view, even though I used it in non-standalone mode. It would be nice if there were a separate compile flag for that, but it’s not a big deal. Having yet another custom/purpose-built string implementation was surprising. It’s not an issue, but just surprising that yet another string type is being created. It was also a bit surprising there wasn’t a function to pretty-print the serialized output (e..g, via an option arg for `serialize()`). It’s fairly common to need such a thing, for example to write to user-viewable files (e.g., configuration or property files), and potentially to debug logs, etc. Obviously the user can write a pretty-printer of their own, but this is one of those things that I think should come out-of-the-box eventually. - What is your evaluation of the documentation? Very good. As a consumer, it had most everything I needed. - What is your evaluation of the potential usefulness of the library? I think it’s very useful. Whether it will become _popular_ I don’t know. There’s no debate there is high demand for a JSON-based variant-structure library similar to Boost.JSON, but there are several popular choices already. Boost.JSON has better performance and memory usage than some of the ones I know of (nlohman and folly::dynamic), so that should give it a leg up. But those libraries also have a lot of bells and whistles, and it’s not clear how much that will matter to people. While my company also has a custom implementation for some specific uses, we do still use folly::dynamic fairly heavily, and I would seriously consider migrating to Boost.JSON in our code-base instead. - Did you try to use the library? Yes, I used it in non-standalone mode, and ran it through some tests we have in my employer’s code base. - With which compiler(s)? Gcc 7.3, C++17 mode. - Did you have any problems? Just a couple bugs that were quickly fixed by the authors. One test failure I got was with the usage of scientific notation in serialized output, as has been noted by other reviewers. It’s not _wrong_, of course, but it’s unusual - and unusual is frequently bad for interoperability. Also, I first tried it in standalone and was surprised it didn’t work even though we use C++17 - sadly, gcc 7.3 doesn’t have the pmr::memory_resource implementation. Gcc doesn’t have pmr until version 9+, which is a shame because incorporating it as standalone would be ideal; we do use boost heavily, but we almost always have issues when upgrading boost versions, so we don’t upgrade often. I think being usable as standalone would increase the uptake for Boost.JSON… but requiring gcc 9+ negates that somewhat, perhaps. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I spent about a day testing it. I’m sorry I didn’t have more time this past week. Life happens. :( - Are you knowledgeable about the problem domain? I’ve been using `folly::dynamic` from Facebook’s folly library since 2014, and have made changes to it for my company’s code base. It provides a similar variant-style structure based on JSON-ish types, with a parser and serializer for JSON encoding. I’ve also written a custom JSON-based variant-structure for some of my company’s particular needs, with built-in schema support, JsonPointer and XPath-style query support, etc. -hadriel

2 1

Re: [boost] [review] [json] My JSON review
by Vinnie Falco 21 Sep '20

21 Sep '20

On Mon, Sep 21, 2020 at 10:29 AM Mathias Gaunard <mathias.gaunard(a)ens-lyon.org> wrote: > > > In both cases, I'd like to read/write my data from/to JSON with the > > > same framework. > > > > Why? What specifically, is the requirement here? > > What I'd like is a way to describe how my C++ types map to a key-value > structure with normalized types so that I can easily convert my > objects back and forth through a structured self-describing and > human-readable interchange format. Right, but what I'm asking you is: *specifically* in what way would a framework that offers both JSON DOM and JSON Serialization be "consistent?" Can you show in terms of declarations, what that would look like? In other words I'm asking you to show using example code how putting these two separate concerns in a single library would offer benefits over having them in separate libraries. Thanks

3 3

[review] [json] My JSON review
by Mathias Gaunard 21 Sep '20

21 Sep '20

Hi, This is my review of Boost.JSON by Vinnie Falco and Krystian Stasiowski. I believe that the library should be REJECTED, though it would me a much better candidate with just a few minor tweaks here and there. The library provides the following components: - a parser framework - a DOM to represent/construct/edit trees of JSON-like values (json::value) - a serializer for that type Unfortunately, all three of these components are not quite satisfactory when it comes to addressing the problem domain of JSON as an interchange format. JSON is ubiquitous technology, and I feel like this library is not deserving of a name such as Boost.JSON, which would suggest "the right way to do JSON as seen by the C++ community". Parser: -------- Parsing is arguably the most important feature of any such library. However with Boost.JSON it really feels like it is an afterthought, and only its coupling with json::value was considered. These are the main issues: - The lack of proper number support from the DOM apparently propagates to the parser as well, even though there is no cost for the parser to properly support them. - The interface is closely tied to a text format and doesn't support JSON-inspired binary formats. This kind of forward thinking is what would really elevate those parsers. - The fact that only a push parser is provided. Parsing into own types usually requires a pull parser in order for nested member's parsers to combine into the outer parser. The way the parser interfaces handles numbers is not only inconsistent with the rest of its interface but also limiting. bool on_number_part( string_view, error_code& ) { return true; } bool on_int64( std::int64_t, string_view, error_code& ) { return true; } bool on_uint64( std::uint64_t, string_view, error_code& ) { return true; } bool on_double( double, string_view, error_code& ) { return true; } I would expect an interface that looks like bool on_number_part( string_view, std::size_t n, error_code& ) { return true; } bool on_number( string_view, std::size_t n, error_code& ) { return true; } bool on_int64( std::int64_t, error_code& ) { return true; } bool on_uint64( std::uint64_t, error_code& ) { return true; } bool on_double( double, error_code& ) { return true; } I see the push parser nicely integrates with beast/asio segmented buffering and incremental parsing, which is nice. DOM: ------ The DOM provides a canonical or vocabulary type to represent any JSON document, which is useful in itself. The DOM does not support numbers that are neither 64-bit integral numbers nor double-precision floating-point. This is a common limitation, but still quite disappointing, since the proposal of it being a canonical type is quite tainted by the fact it can't represent JSON data losslessly. I think it would be better if another type was provided for numbers that don't fit in the previous two categories. It uses a region allocator model based on polymorphic value, which is what the new polymorphic allocator model from Lakos et al. was specifically intended for. It puts some kind of reference-counting for life-extension semantics on top, which I'm not sure I'm a big fan of, but that seems optional. Querying capabilities for that type are pretty limited and quite un-C++ like. Examples use a C-style switch/extractor paradigm instead of a much safer variant-like visitor approach. There is no direct nested xpath-style querying like with boost property tree or other similar libraries. It feels like the interface is only a marginal improvement on that of rapidjson. Serializer: ----------- There is no serializer framework to speak of: just a serializer for the DOM type json::value, i.e. the only way to generate a JSON document is to generate a json::value and have it be serialized. A serializer interface symmetric to that of the parser could be provided instead, allowing arbitrary types to be serialized to JSON (or any other similar format) without conversion to json::value. When it comes to serializing floating-point numbers, it seems it always uses scientific notation. Most libraries have an option so that you get to choose a digit threshold at which to switch. Integers always use a fixed representation, which makes this all quite inconsistent. Review questions: ---------------------- > - What is your evaluation of the design? I can see the design is mostly concerned with receiving arbitrary JSON documents over the network, incrementally building a DOM as the data is received, and then doing the opposite on the sending path. It is good at doing that I suppose, but however is otherwise quite limited as it is unsuitable for any use of JSON as a serialization format for otherwise known messages. > - What is your evaluation of the implementation? I only took a quick glance, but it seemed ok. Can't say I'm a fan of this header-only idea, but some people like it. > - What is your evaluation of the documentation? It is quite lacking when it comes to usage of the parser beyond the DOM. It also doesn't really say how numbers are formatted in the serializer. > - What is your evaluation of the potential usefulness of the library? Limited, given the scope of the library and the quality of its interface, I don't see any reason to use it instead of rapidjson, which is an established library for which I already have converters and Asio stream adapters. > - Did you try to use the library? With which compiler(s)? Did you have any problems? I didn't. > - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? Maybe half a day reading through the documentation and the mailing list. > - Are you knowledgeable about the problem domain? I have used a lot of different JSON libraries and I have written several serialization/deserialization frameworks using JSON, some of which are deeply tied into everything I do everyday.

5 14

[JSON][Review] Boost Json Review
by Maximilian Riemensberger 21 Sep '20

21 Sep '20

My review is primarily from the perspective of a library consumer. I use boost libraries and JSON on a daily basis. But neither did I ever write a boost library nor a JSON library. # Boost JSON Review > Please provide in your review information you think is valuable to > understand your choice to ACCEPT or REJECT including JSON as a > Boost library. Please be explicit about your decision (ACCEPT or REJECT). I recommend ACCEPTing the library into boost. It's well designed, professionally implemented and fills an important use case. The library is well maintained and I trust the authors to keep doing so. The documentation is good, but I the reference documentation could use some improvements in a few spots. Individual reference sections appear copy-pasted a bit too aggressively, which is clearly good for the doc writer but not so much for the doc reader. I consider none of the issues below show-stoppers that would block acceptance into boost. This is also based in the experience that the authors are responsive in maintaining the library (and other libraries) and I trust them that they will use their best judgement and whether and how to address any issues. Brief side note: I'm a user of boost libraries and I consider boost a very important part of the C++ library ecosystem, in particular, for building reliable and long lasting software components and systems. It sets a high bar for acceptance. When using individual boost libraries, an important consideration is whether the library has aged well with the evolution of the C++ standard, whether it is currently well maintained and whether it will remain well maintained in the future. Here boost sometimes appears to be a bit of a mixed bag. So I consider the maintenance promise to be quite important and in particular more important than any of the issues raise below. I want to thank all the people involved in boost for keeping it alive, useful and high quality. > Some other questions you might want to consider answering: > > - What is your evaluation of the design? I evaluated primarily the DOM interface, that is the value container, but didn't look into the parser and serializer in detail. The design is solid and clear. Smaller issues raised on slack during the review have been addressed. I particularly like that it provides an easy to use, automatic ownership allocator interface. It relieves me from the pain of manually maintaining the life-time of non-global memory pools etc. Minor comments: - The `value::is_xxx` functions inconsistently return bools or pointers. This can have unexpected consequences if they are use from a context that does not implicitly convert to bool. Also I really can't remember which one returns what. For example, `is_null`, `is_bool`, `is_number`, `is_string`, `is_structured` returns a really nice mix. (Fixed on develop) - The library provides `initializer_list` construction for `value`. I personally would prefer it wouldn't do that. Too many pitfalls with a missing pair of braces causing the meaning to change quite dramatically. Still I think many expect them in modern JSON libs. So it's up to the authors. - The library is inconsistent about `shrink_to_fit`. `object does not have it, `array` has it. Also the documentation does not state what `shrink_to_fit` does, basically repeating the errors of `std::vector`. I firmly believe that the function should either be removed or clear and precise semantics should be documented. In it's current state the following question comes to mind: Why would I ever want to call it? > - What is your evaluation of the implementation? I looked at the implementation of the `value` container. I did not look into the parser and serializer. The implementation appears clean and professional. The authors responded quickly to issues raised while studying the source code. > - What is your evaluation of the documentation? The documentation is generally good and well structured. However, the reference is occasionally pretty terse. Some parts of it are written with the assumption that any user going to the reference section has recently read the entire documentation including examples upfront. Some comments (mostly against master and develop from early Sep 20): * `value::storage()` should say: Return a reference to the `storage_ptr` to the `memory_resource` associated with the value. The same thing applies to a few other classes. * The reference for the `initializer_list` constructor could use some improvements: e.g. "Construct an object or array from an `initializer_list`". I find this hugely important since just the mere presence of an `initializer_list` constructor changes the meaning of brace initialization. * Additionally, I doesn't mention how nested `initializer_list` how are treated. Most importantly it should show or link to some examples on how the `initializer_list` are converted to objects, arrays, and non structured values because some developers (if not most) might not immediately realize from just looking at the reference documentation how nested `initializer_list` are parsed. Maybe also reference to the `initializer_list` constructors of `array` and `object`. * `object::insert` is not well-documented. I did not understand what it does when I insert an already existing key, in particular, whether assignment takes place or not. The same is true for `object::emplace`. * The `object` initializer constructor is listed under "Copy constructor". * The `object::erase` return value documentation is wrong. * `object::reserve` says "This inserts an element into the container." * `object::swap(object)` and the friend version `swap(object,object)` have rather inconsistent briefs. * The initializer_list constructor for array is listed under move constructors. * The pilfer constructors are not quite explained. There's a reference to the paper, but I think the library should have a page that concisely explains what it does and what a user of the library can expect from pilfered objects. In particular, how the thing differs from move and why a user would ever want to use it instead of move. Also the reference pages for pilfer and so on don't explain what it does. * `array::reserve` shows `BOOST_FORCEINLINE` in the docs. * `value_to` is unclear about the precedence of the three listed options (constructor, library provided conversion or `tag_invoke`) if multiple are present provided this is actually possible. The same is true for `value_from`. * `storage_ptr` thread safety is not documented. That's a real bummer for a `shared_ptr`-like construct. * I found it rather difficult to figure out the difference in the behavior of `parser::write` and `parser::write_some` just from reading the documentation. Either reference documentation should clearly state what the difference is. I had to open both pages side by side to actually spot the wording difference. For example, I think that the documentation of `write_some` should clearly state that it's intended use case is for parsing multiple JSON values from a byte stream presented through one or more buffers whereas `write` is intended for parsing one JSON value from one or more buffers. At least that's what I concluded from the documentation after some time. * The `parser` documentation should either link to or inline provide one or more examples how the parser is supposed to be used. In particular `write_some` would benefit hugely from an example that shows how to properly extract a stream of JSON values from a sequence of buffers. * If I read the documentation correctly, `basic_parser::write` behave more like `parser::write_some` than `parser::write`. I would prefer the call it `basic_parser::write_some`. * The `basic_parser` reference documentation would benefit from either a link to an example or an inline example. > - What is your evaluation of the potential usefulness of the library? I think the library supports a very narrow yet widespread use-case. It offers a modern interface and high performance. So I will definitely use and I believe it will also be useful to a wide audience. > - Did you try to use the library? With which compiler(s)? Did you have > any problems? I played with it in its STANDALONE mode with gcc 9.3. > - How much effort did you put into your evaluation? A glance? A quick > reading? In-depth study? Roughly 5 hours playing with small examples, but mostly looking into the documentation and into the code of `json::value` and related parts of the DOM implementation. > - Are you knowledgeable about the problem domain? I use JSON on a regular basis across different languages including c++ (mostly nlohmann's up to date). I've not implemented a JSON library myself. Kind regards, Max

3 4

[review][JSON] Boost.JSON Review starts Sept 14
by Pranam Lashkari 20 Sep '20

20 Sep '20

The boost formal review for JSON will take place from Sept 14, 2020 to Sept 23, 2020. JSON is authored by Vinnie Falco (author: boost beast and Static String), Krystian Stasiowski(author: Static String) and submitted with of Peter Dimov(author: Too many libraries to mention 😛) The use of JSON format to share the data is getting bigger and wider because it is lightweight and it's supported cross-platform and many more. JSON has numbers of uses and to make our life easy to deal with it in C++ this library is introduced. Documentation can be found here: <http://vinniefalco.github.io/doc/json/> It includes examples, usage and comparison. Documentation is still under progress but still provides enough information. Official repository: <https://github.com/CPPAlliance/json> Glad to address any questions or issues, here or in the repo: <https://github.com/CPPAlliance/json/issues/> The purpose of this announcement is to give people some time to start looking at the documentation and source code for the library. Feel free to begin asking questions. Most of all, get involved! Nowadays most of us(assuming) would have dealt with the JSON in one way or another. Tell the Boost community what you think of JSON. -- Thank you, Pranam Lashkari, https://lpranam.github.io/

5 8

Reviewing Boost.JSON
by Vinícius dos Santos Oliveira 19 Sep '20

19 Sep '20

I'm gonna use a perhaps unusual text structure for this review. Consuming text is a tiring activity that demands energy. Having that in mind, I ignored the order in which the library is presented through the docs and moved all trivial concepts discussion to the end. That's my attempt to make sure the message gets through while the reader still has it -- the energy. ## A short introduction to pull parsers vs push parsers The act of parsing an incoming stream will generate events from a predefined set. Users match the generated events against rules to trigger actions. For the following JSON sample: ```json { "foo" : 42 , "bar" : 33 } ``` One of such event streams could be: ``` begin_object -> string_key -> number -> string_key -> number -> end_object ``` The essential difference between pull parsers and push parsers is how this event stream is delivered. A push parser (a.k.a. SAX parsers) uses the callback handler model. A pull parser (a.k.a. StAX parsers) stores the current parsing state and allows the user to query this state. Essentially every push parser has an implicit pull parser that doesn't get exposed to the user. This is to say that a push parser would be implemented in the lines of: ```cpp // parsing loop while (it != end) { match_token(); dispatch_token(user_handlers); } ``` Whereas a pull parser doesn't have the loop construct above, just its body. The loop construct would be under user-control. That's why pull parsers are the more fundamental model. A push parser can be built on top of a pull parser, but not the inverse. Concrete examples will be shown later in this text. ## Performance: a common misconception Some push parsing proponents believe pull parsers are slower. This belief is unsubstantiated. As we have seen in the previous section, a push parser is always built on top of an implicit pull parser. The only thing that differs them both is that a small part of the parser logic under the pulling model moves out of the interface to become user code. That's the only difference. One of the arguments brought by push parser proponents is the ability to merge matching and decoding steps to perform less work. We can use the previous sample to illustrate this idea: ``` { "foo" : 42 , "bar" : 33 } ^^^^^ ``` If we are to decode the `"foo"` token above, the parser first needs to discover where the token begins and where the token ends. This is the required matching step that every parser needs to perform. JSON strings can be escaped and one such example would be: ```json "na\u00EFve" ``` The decoding steps take care of converting escaped string slices to unescaped ones. So user code would receive 3 string runs while this token is being decoded: - "na" - "ï" - "ve" The parser does not allocate any memory to decode any string. The slices are feeded to the user who in turns is responsible to put them together (possibly allocating heap memory). The idea here is... having separate match and decode steps forces the parser to walk over the tokens twice. For Boost.JSON, the merged match and decode steps materialize in the `on_string_part()` (and `on_string()`) callback: ```cpp bool on_string_part(string_view v, std::size_t, error_code& ec) { current_str.append(v.data(), v.size()); return true; } ``` For a pull parser with _separate_ match and decode steps the interface would be in the lines of: ```cpp reader.next(); if (reader.kind() == token::string) reader.string(current_str); ``` But as we've seen before, the only difference between pull and push parsers is something else. Any features lateral to this classification originate from different taxonomies. Just as a push parser can merge the matching and decoding steps so can a pull parser: ```cpp // A new overload. // // `my_match_options` here would provide // the string collector upfront. reader.next(my_match_options); ``` Another small note here is that just as merging matching and decoding steps can be beneficial performance-wise so can skipping the whole decoding step. I'll be touching on this very idea again in the section below. Appendix A will present how another feature from Boost.JSON can be implemented in pull parsers: strings split among different buffers. ## The case for pull parsers So far we've only been exposed to the difference between push parsers and pull parsers, but there wasn't a case for which choice is ideal (aside from the naturally understood concept that pull parsers are more fundamental and push parsers can be built on top when desired). The defining trait present in pull parsers that is impossible to replicate in push parsers is their ability to compose algorithms. Composition is enabled by temporarily handing over the token stream to a different consumer. Not every consumer needs to know how to process every part of the stream. That's a trait impossible to find in push parsers. I have created a [gawk](https://www.gnu.org/software/gawk/) plug-in to illustrate this property. The same plug-in was implemented several times using different libraries/approaches. I incrementally added more features under different branches (there are 8 branches in total). You can find a rationale for the AWK choice in Appendix B by the end of this review. The plugin's code can be found at <https://gitlab.com/vinipsmaker/gawk-jsonstream-archive>. And you may forgive my beginner code, but I had no experience to write gawk plug-ins beforehand. That's my first. AWK is a C-like language (in syntax) where you define pattern-action pairs. You can find a quick'n'dirty introduction to the language from Wikipedia. This plug-in adds JSON capabilities to the tool. Given the following data-set: ``` {"name": "Beth", "pay_rate": 4.00, "hours_worked": 0} {"name": "Dan", "pay_rate": 3.75, "hours_worked": 0} {"name": "Kathy", "pay_rate": 4.00, "hours_worked": 10} {"name": "Mark", "pay_rate": 5.00, "hours_worked": 20} {"name": "Mary", "pay_rate": 5.50, "hours_worked": 22} {"name": "Susie", "pay_rate": 4.25, "hours_worked": 18} ``` And the following AWK program (with this plug-in loaded): ```awk BEGIN { JPAT[1] = "/name" JPAT[2] = "/pay_rate" JPAT[3] = "/hours_worked" } J[3] > 0 { print J[1], J[2] * J[3] } ``` The output will be: ``` Kathy 40 Mark 100 Mary 121 Susie 76.5 ``` The paths for the JSON tree are given in [standard JSON Pointer](https://tools.ietf.org/html/rfc6901) syntax. That's the plugin's core and it's the code that doesn't change among branches, so it's the appropriate place to begin the explanation. In the file `jpat.ypp` you'll find a `read_jpat()` function that gets called for each record. There's some boilerplate in the beginning to bail out earlier if there were no changes to `JPAT`. Then there is this snippet that gets processed by [re2c](https://re2c.org/) to generate valid C++ code: ```cpp * { if (*begin != '\0') return bad_path(); jpat[i].components.emplace_back(std::move(current_component)); goto end; } {prefix} { jpat[i].components.emplace_back(std::move(current_component)); current_component = std::string{}; goto loop; } {unescaped} { current_component.append(begin, YYCURSOR - begin); goto loop; } {escaped_tilde} { current_component.push_back('~'); goto loop; } {escaped_slash} { current_component.push_back('/'); goto loop; } ``` The definitions for the data structures follow: ```cpp struct Path { std::vector<std::string> components; awk_value_cookie_t cookie = nullptr; awk_string_t orig; }; struct Tree: std::map<std::string, std::variant<Tree, size_t>> {}; static std::vector<Path> jpat; Tree tree; ``` By the end of the re2c-generated loop, the variable `jpat` will be filled. Then there is some validation/sanitization steps and we start to fill the `tree` variable out of the extracted paths. `tree` is a decision tree auxiliary to the process of parsing the JSON documents. Leaf nodes represent which index in the `J` AWK variable will receive that value. Given the following `JPAT`: ```awk JPAT[1] = "/foo/bar" JPAT[2] = "/foobar" ``` `tree` will have the following properties (pseudocode): ```cpp // AWK arrays are 1-indexed, so the code // translation has a 1-element offset assert(tree["foo"]["bar"] == 0); assert(tree["foobar"] == 1); ``` The user may find other uses for a JSON decision tree in Appendix C. And here is where the code branches diverge. So let's present an overview and then walk them one-by-one (or two-by-two): ``` +------------+---------------+--------------+ | | Boost.JSON | pull parser | +------------+---------------+--------------+ | basic | boostjson | master | | dom | boostjson-dom | | | transform | boostjson2-b | master2 | | transform2 | boostjson3 | master3 | +------------+---------------+--------------+ ``` The `master` branch is just 506 lines of C++ code total and `boostjson` is 613 lines. The description for the other branches follow: - `dom`: built after `basic`. This branch is useful to highlight performance differences between the DOM parser and the rest. - `transform`: `boostjson2-b` is built on top of `boostjson-dom` branch. `master2` is built on top of `master` branch. It adds the power to update the JSON document. Here we start to notice how the ability to mark certain points in the token stream are important for a JSON library. - `transform2`: built on top of the previous branch, it adds the `JUNWIND` operation in the likes of MongoDB's `$unwind` operator. The first thing I'll do is to compare the DOM parsing layer cost. [There's this 161MiB JSON file that'll be used](https://drive.google.com/file/d/1-lVxGYWvvmBsjrDVgju_lOTBaK03MdQc/vie…. Just as `jq` can be used to extract a field from this file: ```bash jq -cM '.features[10000].properties.LOT_NUM' citylots2.json ``` So can AWK: ```bash gawk -l jsonstream -e ' BEGIN { JPAT[1] = "/features/10000/properties/LOT_NUM" } { print J[1] }' citylots2.json ``` The execution time for the boostjson branch will be: ``` real 0m2,112s user 0m1,476s sys 0m0,618s ``` And the execution time for the boostjson-dom branch will be: ``` real 0m5,639s user 0m4,170s sys 0m1,418s ``` We should always strive to do more than just throwing random numbers. Anyone can do that. The timings show that the DOM layer is obviously much slower, but there is no surprise here. It's kind of ridiculous trying to compare both branches. They do different things (although the plug-in feature-set is the same). The DOM approach will always decode and save every matched token. And that's one of the principles to deliver faster code: do less. The reason why I've chosen a 161MiB JSON file was to make sure most of the program's time would be spent not in the gawk VM but in the parsing plug-in. If we wanted to expose every value from the DOM tree, `json::value` would also be useless because the consumer here expects `awk_value_t` objects (and the DOM interface would guarantee the costs for unnecessarily saving decoded results twice would be paid). We don't always control the interface/protocol -- that's part of the C++ programmer's job. Would it be reasonable to deliver code twice as slow to my client because "hey I have a nice DOM container... gotta use it"? As Marcel Weiher has noted in [his call for "less lethargic JSON support"](https://blog.metaobject.com/2020/04/somewhat-less-lethargic-json-support.html) [sic]: > Wait a second! 312MB/s is almost 10x slower than Daniel Lemire's parser, > the one that actually parses JSON, and we are only creating the objects > that would result if we were parsing, without doing any actual parsing. > > This is one of the many unintuitive aspects of parsing performance: the > actual low-level, character-level parsing is generally the least > important part for overall performance. Unless you do something crazy > like use `NSScanner`. Don't use `NSScanner`. Please. Let's dive in the code to see how they differ. ### `basic` Every branch reads a record one document at a line. That's the `get_record()` function. For the `boostjson-dom` branch, the code goes in the lines of: ```cpp try { parsed_value.emplace_null(); json_parser.reset(parsed_value.storage()); std::error_code ec; json_parser.write(line.data(), line.size(), ec); if (ec) throw std::system_error{ec}; json_parser.finish(ec); if (ec) throw std::system_error{ec}; parsed_value = json_parser.release(ec); assert(!ec); switch (parsed_value.kind()) { case json::kind::array: fill_j(parsed_value.get_array(), tree); break; case json::kind::object: fill_j(parsed_value.get_object(), tree); break; // ... } } catch (const std::exception& e) { warning(ext_id, "Error while processing record: %s", e.what()); } ``` The code parses the whole JSON at once and then follows to consume the document by calling the `fill_j()` function: ```cpp // Several overloads: static void fill_j(const json::array& arr, Tree& node); static void fill_j(const json::object& obj, Tree& node); static void fill_j(const json::value& v, Tree& node); // Here is the object overload. You can find the rest in the repository. static void fill_j(const json::object& obj, Tree& node) { for (auto& [key, value]: node) { auto it = obj.find(key); if (it == obj.end()) continue; std::visit( hana::overload( [&](Tree& tree) { fill_j(it->value(), tree); }, [&](size_t idx) { consume_value(it->value(), idx); }), value); } } ``` The code is pretty straight-forward. We check if the fields the user asked for are present in the document and recurse on them. When we reach a leaf node, the `J` AWK variable is set (that'd be the `consume_value()` function). Do notice that there is no uncontrolled recursion here. Recursion is always JPAT-delimited. And JPAT is part of the source code, not user-input. There is no risk of stack overflow here. The pull parser approach (branch master) somewhat follows the same trend, but the code is bigger. Here's a selected highlight: ```cpp void fill_j(json::reader& reader, Tree& node) { // ... switch (reader.symbol()) { case json::token::symbol::error: throw json::error{reader.error()}; case json::token::symbol::begin_object: { if (!reader.next()) throw json::error{reader.error()}; std::string current_key; for (;;) { if (reader.symbol() == json::token::symbol::end_object) { reader.next(); break; } // Key assert(reader.symbol() == json::token::symbol::string); current_key.clear(); auto ec = reader.string(current_key); assert(!ec); (void)ec; if (!reader.next()) throw json::error{reader.error()}; // Value auto it = node.find(current_key); if (it == node.end()) { json::partial::skip(reader); continue; } std::visit( hana::overload( [&](Tree& tree) { switch (reader.symbol()) { case json::token::symbol::begin_array: case json::token::symbol::begin_object: fill_j(reader, tree); break; default: json::partial::skip(reader); } }, consume_value), it->second); } if (reader.symbol() == json::token::symbol::error) throw json::error{reader.error()}; } // ... } } ``` And here is the first sample on what I meant by composability. Our "callback" only gets called to handle trees that are part of the decision tree one way or another (and the root node always belongs here). When it reaches a subtree that doesn't belong to the JPAT-defined decision tree, `json::partial::skip()` is called. `json::partial::skip()` is defined elsewhere and has no knowledge on the internals of our algorithm. This is a different "callback" being called to handle a different part of the tree. The two of them know how to handle different groups of tokens. A nice side effect is that these "phantom" trees will be fully skipped over. We don't allocate strings to collect values over them. This is an optimization we get for free. [Designing JSON parsers to exploit this idea is not unknown](https://github.com/zserge/jsmn). The last one will be the code for the `boostjson` branch, but I feel like it'll be easier to understand if I first introduce the tasks it has to perform in the token stream. ``` { "foo" : 42 , "bar" : { "foo" : 33 } } ^^ ``` When the field's value is parsed, we must match the key against our rule-set to possibly trigger an action. But it is not enough to store the current key. The whole path must be stored to support the JSON nest-able structure. We start with the structure to save current context information: ```cpp struct State { bool is_object; //< false = ARRAY std::string current_key; size_t current_idx; Tree* node; }; std::vector<State> state; ``` So the beginning of the code for `on_object_begin()` must be: ```cpp Tree* node = nullptr; if (state.size() == 0) { node = &tree; } else if (state.back().node) { auto it = state.back().node->find(current_key()); if (it != state.back().node->end()) node = std::get_if<Tree>(&it->second); } state.emplace_back(); state.back().is_object = true; state.back().node = node; ``` And a matching action in the code for `on_object_end()`: ```cpp state.pop_back(); ``` But it's not really that simple. Paths such as `/features/10000/properties/LOT_NUM` won't work because `current_key` won't be provided by any `on_key()`-like callback when current node is `10000`. `current_key` must be computed for arrays in the value handler itself. That's why the following snippet must be present in the beginning of every value-callback (and `on_{object,array}_begin()` too): ```cpp if (state.size() > 0 && /*is_array=*/!state.back().is_object) update_current_array_key(); ``` And the matching helper functions: ```cpp std::string_view current_key() { assert(state.size() > 0); return state.back().current_key; } void update_current_array_key() { assert(state.size() > 0); assert(!state.back().is_object); state.back().current_key.resize(max_digits); auto s_size = std::to_chars( state.back().current_key.data(), state.back().current_key.data() + state.back().current_key.size(), state.back().current_idx++ ).ptr - state.back().current_key.data(); state.back().current_key.resize(s_size); } ``` The final touch is to realize that the event stream will be something like: ``` string_key -> value -> string_key -> value -> string_key -> value -> ... ``` And we need to clear the `string_key` by the end of every value consumed: ```cpp if (state.size() > 0) state.back().current_key.clear(); ``` So the callback implementation for a simple atom value such as a string is: ```cpp bool on_string_part(std::string_view v, std::error_code&) { if (state.size() > 0) { if (state.back().node == nullptr) return true; auto it = state.back().node->find(current_key()); if (it == state.back().node->end()) return true; if (!std::holds_alternative<size_t>(it->second)) return true; } current_str.append(v.data(), v.size()); return true; } bool on_string(std::string_view v, std::error_code&) { awk_value_t idx_as_val; if (state.size() == 0) { current_str.append(v.data(), v.size()); make_number(0, &idx_as_val); } else { if (/*is_array=*/!state.back().is_object) update_current_array_key(); if (state.back().node == nullptr) goto end; auto it = state.back().node->find(current_key()); if (it == state.back().node->end()) goto end; auto idx = std::get_if<size_t>(&it->second); if (!idx) goto end; current_str.append(v.data(), v.size()); make_number(*idx + 1, &idx_as_val); } { awk_value_t value_as_val; char* mem = (char*)gawk_malloc(current_str.size() + 1); memcpy(mem, current_str.c_str(), current_str.size() + 1); set_array_element( var_j, &idx_as_val, make_malloced_string(mem, current_str.size(), &value_as_val)); } end: if (state.size() > 0) state.back().current_key.clear(); current_str.clear(); return true; } ``` But the code still performs differently. There's that optimization we got for free in the pull parser choice. We must implement a mechanism to skip over unwanted trees to make the comparison fair. And here's where a `skipped_levels` variable is introduced. It'll change every callback again but will finally let us have the following code for the `string_key` callback: ```cpp bool on_key_part(std::string_view v, std::error_code&) { if (skipped_levels) return true; // not called on phantom trees state.back().current_key.append(v.data(), v.size()); return true; } bool on_key(std::string_view v, std::error_code&) { if (skipped_levels) return true; // not called on phantom trees state.back().current_key.append(v.data(), v.size()); return true; } ``` The full code can be found on the `include/jsonstream/parser.hpp` file. This explanation should suffice to understand (to quote myself): > The defining trait present in pull parsers that is impossible to > replicate in push parsers is their ability to compose > algorithms. Composition is enabled by temporarily handing over the token > stream to a different consumer. Not every consumer needs to know how to > process every part of the stream. That's a trait impossible to find in > push parsers. Every callback depends on implementation knowledge from every other callback. Spaghetti code as they say. I also implemented another change to both branches -- `master` and `boostjson` -- to further demonstrate this property. What if we want to store string representation for array indexes in static arrays instead `std::string` objects? There's this changeset for `master` branch: ``` --- a/src/main.cpp +++ b/src/main.cpp @@ -184,20 +184,19 @@ void fill_j(json::reader& reader, Tree& node) case json::token::symbol::begin_array: { if (!reader.next()) throw json::error{reader.error()}; - std::string current_key; + std::array<char, max_digits> current_key; for (size_t i = 0 ;; ++i) { if (reader.symbol() == json::token::symbol::end_array) { reader.next(); break; } - current_key.resize(max_digits); auto s_size = std::to_chars(current_key.data(), current_key.data() + current_key.size(), i).ptr - current_key.data(); - current_key.resize(s_size); - if (node.count(current_key) == 0) { + auto it = node.find(std::string_view(current_key.data(), s_size)); + if (it == node.end()) { json::partial::skip(reader); continue; } @@ -214,7 +213,7 @@ void fill_j(json::reader& reader, Tree& node) } }, consume_value), - node[current_key]); + it->second); } if (reader.symbol() == json::token::symbol::error) ``` The changes are local to the code that parses this structure (only the `case json::token::symbol::begin_array` "callback" changed). It doesn't affect the rest. As for the `boostjson` branch, the code for every callback must now execute new code (if you're curious why we'd need a `current_key()` helper function): <https://gitlab.com/vinipsmaker/gawk-jsonstream-archive/-/commit/f4c81abf7e7…>. As an exercise for the reader, try to imagine what changes would be required in the push parser approach if we wanted to expose a DOM handle in the `J` AWK variable. As for the pull parser approach we have a call to `partial::skip()` within the `consume_value()` function. This call would be replaced with `partial::parse()` and that's the change. ### `transform` By now I already made my main point. Push parsers don't compose. I'll keep the explanation for remaining branches brief and only give an outline over them. We've made a plug-in to read JSON values. Naturally the next step is to have the ability to update/write them. It's not possible to do this with Boost.JSON parser, so I had to give up on branch boostjson2 and start a new one -- boostjson2-b. `boostjson2-b` is based on `boostjson-dom` so you can expect similar prohibitive costs from the DOM layer. The strategy here is the same as before. We recursively traverse the stored DOM using the decision tree to guide the whole process. The previous step had a `consume_value()` function and the new step will have a `produce_value()` one: ```cpp static void produce_value(json::value& dom, size_t idx) { awk_value_t idx_as_val; make_number(idx + 1, &idx_as_val); awk_value_t value; if (!get_array_element(var_j, &idx_as_val, AWK_UNDEFINED, &value)) { dom.emplace_null(); return; } switch (value.val_type) { case AWK_UNDEFINED: case AWK_ARRAY: dom.emplace_null(); break; case AWK_NUMBER: if (value.num_value == 1.0 && dom.is_bool()) { dom.emplace_bool() = true; } else if (value.num_value == 0.0 && dom.is_bool()) { dom.emplace_bool() = false; } else { // TODO: respect AWK's OFMT dom.emplace_double() = value.num_value; } break; case AWK_STRING: case AWK_REGEX: case AWK_STRNUM: { dom.emplace_string() = std::string_view{ value.str_value.str, value.str_value.len}; break; } case AWK_SCALAR: case AWK_VALUE_COOKIE: assert(false); } } ``` Then we just serialize the whole DOM tree when the AWK function `json::output()` is called: ```cpp static awk_value_t* do_output(int /*num_actual_args*/, awk_value_t* result, awk_ext_func_t* /*finfo*/) { query_j(parsed_value, tree); auto serialized = json::to_string(parsed_value); char* mem = (char*)gawk_malloc(serialized.size() + 1); memcpy(mem, serialized.c_str(), serialized.size() + 1); return make_malloced_string(mem, serialized.size(), result); } ``` For the pull parser we modify the read loop to store the document offsets (this small diff is the part impossible to achieve with Boost.JSON parser): ``` --- a/src/main.cpp +++ b/src/main.cpp @@ -96,6 +96,11 @@ void fill_j(json::reader& reader, Tree& node) // atom values at root level were moved out of this recursive function. auto consume_value = [&](size_t idx) { + Slice slice; + slice.rel_off = reader.literal().data() - cursor; + slice.jidx = idx; + slice.bool_src = false; + switch (reader.symbol()) { case json::token::symbol::end: case json::token::symbol::error: @@ -109,6 +114,8 @@ void fill_j(json::reader& reader, Tree& node) var_j, make_number(idx + 1, &idx_as_val), make_number(reader.value<bool>() ? 1 : 0, &value)); + + slice.bool_src = true; } if (false) case json::token::symbol::integer: @@ -133,12 +140,15 @@ void fill_j(json::reader& reader, Tree& node) make_malloced_string(mem, value.size(), &value_as_val)); } case json::token::symbol::null: + slice.size = reader.literal().size(); if (!reader.next()) throw json::error{reader.error()}; break; case json::token::symbol::begin_array: case json::token::symbol::begin_object: - json::partial::skip(reader); + slice.size = json::partial::skip(reader).size(); } + cursor += slice.rel_off + slice.size; + slices.push_back(slice); }; switch (reader.symbol()) { ``` And implement AWK `json::output()` to perform simple string concatenation: ```cpp static awk_value_t* do_output(int /*num_actual_args*/, awk_value_t* result, awk_ext_func_t* /*finfo*/) { size_t output_size = 0; for (auto& slice: slices) { output_size += slice.rel_off; slice.buffer.clear(); awk_value_t idx_as_val; make_number(slice.jidx + 1, &idx_as_val); awk_value_t value; if (!get_array_element(var_j, &idx_as_val, AWK_UNDEFINED, &value)) { slice.buffer = "null"; output_size += slice.buffer.size(); continue; } switch (value.val_type) { case AWK_UNDEFINED: case AWK_ARRAY: slice.buffer = "null"; break; case AWK_NUMBER: if (value.num_value == 1.0 && slice.bool_src) { slice.buffer = "true"; } else if (value.num_value == 0.0 && slice.bool_src) { slice.buffer = "false"; } else { // TODO: respect AWK's OFMT json::writer writer{slice.buffer}; writer.value(value.num_value); } break; case AWK_STRING: case AWK_REGEX: case AWK_STRNUM: { json::writer writer{slice.buffer}; writer.value(json::writer::view_type{ value.str_value.str, value.str_value.len}); break; } case AWK_SCALAR: case AWK_VALUE_COOKIE: assert(false); } output_size += slice.buffer.size(); } auto buffer_end = buffer.data() + record_size - record_rt_size; output_size += buffer_end - cursor; char*const mem = (char*)gawk_malloc(output_size + 1); auto out_it = mem; auto in_it = buffer.data(); for (auto& slice: slices) { memcpy(out_it, in_it, slice.rel_off); out_it += slice.rel_off; in_it += slice.rel_off + slice.size; memcpy(out_it, slice.buffer.data(), slice.buffer.size()); out_it += slice.buffer.size(); } memcpy(out_it, cursor, buffer_end - cursor); out_it += buffer_end - cursor; *out_it = '\0'; return make_malloced_string(mem, output_size, result); } ``` It's clear which solution will perform better here (string concatenation of a few string slices vs serializing the whole DOM tree again), but there is another interesting difference to highlight. The difference having nothing to do with pull parsers vs push parsers and everything to do with the serialization framework. As we can see from the alternative library here, we're able to serialize the document piece by piece. Boost.JSON has one serialization facility -- `json::serializer` and its by-products such as `json::to_string()` -- and that's it. If you have a domain object in the likes of: ``` struct Reply { int req_id; std::vector<std::string> result; }; ``` The only answer for Boost.JSON would be to first convert the whole object into `json::value` and serialize the resulting DOM tree. That's an unacceptable answer. ### transform2 For the last developed branches, I demonstrate how partial DOM trees fit in the design of pull parsers. MongoDB has this `$unwind` operator that will output a new document for each element in a certain array element of the input document. So, for the following document: ``` { "foo" 33 , "bar" : [ 3, 4, "abc" ] } ``` MongoDB's `$unwind` operator can be used to output: ``` { "foo" 33 , "bar" : 3 } { "foo" 33 , "bar" : 4 } { "foo" 33 , "bar" : "abc" } ``` For our plug-in, you just fill the index you'd like unwind: ```awk BEGIN { JPAT[1] = "/bar" JUNWIND = 1 #< index in JPAT array } ``` For `boostjson3` branch, the idea is to swap the desired array to a different place so we can later call `json::output()` cleanly: ``` --- a/src/main.cpp +++ b/src/main.cpp @@ -74,13 +74,32 @@ static const awk_fieldwidth_info_t zero_fields = { static json::parser json_parser; static json::value parsed_value; -static void consume_value(const json::value& v, size_t idx) +int unwind_idx; //< 1-indexed ; 0 is root ; -1 is unwind unset +json::array unwinded_values; +static bool unwind_in_progress; +static size_t nunwinded; + +static void consume_value(json::value& v, size_t idx) { switch (v.kind()) { case json::kind::array: + if ((int)idx + 1 == unwind_idx) { + v.get_array().swap(unwinded_values); + { + awk_value_t val; + auto ok = sym_update_scalar( + var_julength, + make_number(unwinded_values.size(), &val)); + assert(ok); (void)ok; + } + unwind_in_progress = true; + nunwinded = 0; + break; + } case json::kind::object: break; case json::kind::string: { ``` For `master3` branch, only one change happened to the read loop (again proving the composability of pull parsers). The idea is to store the unwind target directly into a domain object (so almost a partial DOM, but even cheaper). Do notice that only `json::token::symbol::begin_array` "callback" changed. Every other "callback" stays the same: ``` --- a/src/main.cpp +++ b/src/main.cpp @@ -146,6 +151,63 @@ void fill_j(json::reader& reader, Tree& node) if (!reader.next()) throw json::error{reader.error()}; break; case json::token::symbol::begin_array: + if ((int)idx + 1 == unwind_idx) { + const char* head = reader.literal().data(); + if (!reader.next()) throw json::error{reader.error()}; + for (;;) { + if (reader.symbol() == json::token::symbol::end_array) + break; + + awk_value_t value; + + switch (reader.symbol()) { + case json::token::symbol::end: + case json::token::symbol::error: + case json::token::symbol::end_object: + case json::token::symbol::end_array: + assert(false); + case json::token::symbol::boolean: + make_number(reader.value<bool>() ? 1 : 0, &value); + break; + case json::token::symbol::integer: + case json::token::symbol::real: + make_number(reader.value<double>(), &value); + break; + case json::token::symbol::string: { + auto data = reader.value<std::string>(); + char* mem = (char*)gawk_malloc(data.size() + 1); + memcpy(mem, data.c_str(), data.size() + 1); + make_malloced_string(mem, data.size(), &value); + } + break; + case json::token::symbol::null: + value.val_type = AWK_UNDEFINED; + break; + case json::token::symbol::begin_array: + case json::token::symbol::begin_object: + value.val_type = AWK_UNDEFINED; + unwinded_values.push_back(value); + json::partial::skip(reader); + continue; + } + unwinded_values.push_back(value); + if (!reader.next()) throw json::error{reader.error()}; + } + assert(reader.symbol() == json::token::symbol::end_array); + const char* tail = reader.literal().end(); + if (!reader.next()) throw json::error{reader.error()}; + slice.size = std::distance(head, tail); + { + awk_value_t val; + auto ok = sym_update_scalar( + var_julength, + make_number(unwinded_values.size(), &val)); + assert(ok); (void)ok; + } + unwind_in_progress = true; + nunwinded = 0; + break; + } ``` I have explored a few concepts, but there are many more that were left off (e.g. backtracking, wrapping parsers, chunking). I'm also familiar with these topics and my assessment here would be the same: either there is no loss in the pull parser approach and they both perform well, or the push parser will perform significantly worse. ## A faster DOM tree There is another class of parsers that I haven't talked about. If parsing-to-DOM speed is the only relevant pursuit for this library, then it's failing too. It should be using destructive parsing techniques. Decoding strings can be performed in-place to put less pressure on allocators (and to skip several memory copies). I can expand on this topic if there is interest. ## Misc Some assorted issues and observations: - Boost.JSON serialization framework is non-existing. There's only one feature. The functions don't help to generate any serialized JSON value. Any value to be serialized must first be converted to the allocating variant-like class `json::value`. - `json::value::get_*()` functions break current convention for boundary checking versions of accessor functions in a variant class. - There is no `std::monostate` overload to initialize `json::value`. On a personal note, I'd prefer to see an explicit `json::null_t`. I don't like to conflate meaning of terms created for different purposes. - Documentation for the parser interface is pretty much lacking. Parameters are poorly explained and there are plenty of other questions that go unanswered. Can the handler pause the parsing loop by returning `false` w/o setting any error code (and how'd the stream be resumed)? Can the handler callbacks access parser methods such as `depth()`? Can the handler callbacks copy the parser object (this would be useful for backtracking)? - The code compiles with warnings. - The library offers far too little. The soundness of its design can't be proven. All it does is converting to/from a variant-like class and every other JSON-related use-case is left off. In the future, another library in the likes of QtXmlPatterns (but for JSON) will be built but this core -- Boost.JSON -- will prove unfit to build anything higher-level and a new core will have to be developed from the ground-up because current bad design choices. ## Appendix A: Strings through multiple buffers in pull parsers There are always compromises when it comes to designing programming abstractions. As for parsers, one of such compromises is a parser that never "fails" to consume input. Every input given is fully consumed. The idea here is to avoid "reparsing" old elements from the stream. For instance, one document may be received as the following two separate buffers: ``` "na\u00 EFve" ``` The parser may fail to consume the escape sequence thanks to missing characters and return `nread=3` for the first buffer. The performance argument in favour of a parser that doesn't fail is irrelevant. There is no performance gain by not reparsing 4 bytes. If your buffer is that small that you find yourself constantly reparsing elements then you have a problem way before the parsing level. The read syscall overhead would dominate. That's a pathological case. You must never design abstractions for a pathological case. I'll refrain from initiating any debate here on what'd constitute a pathological case for JSON parsers. I just want to make sure the reader has this in mind and he should be careful to not ask for every feature under the sun. He must carry a grain of doubt over whether a feature is really desired. In fact, the parser may not buffer, but [now the consumer does so](https://github.com/CPPAlliance/json/blob/fc7b1c6fd229e884df09fd45c64c61… (so much for a chunking parser). As for the topic on this section: can a pull parser support this case? The answer will be the same as before (to quote myself): > But as we've seen before, the only difference between pull and push > parsers is something else. Any features lateral to this classification > originate from different taxonomies. The enabling feature here is not the push interface. Rather it is the chosen set of generated events for the JSON token stream. If you change the notified events for the stream, you enable the same feature. For a pull parser, these are the events it'd have to provide (among other valid choices): - `string` (if non-split) - `string_run_begin` - `string_run` - `string_run_end` ## Appendix B: Why gawk? There are a few reasons why I've chosen a (GNU) AWK plug-in to illustrate the concepts presented in this review. First, I don't want to invent a new tool. Artificial examples could always be created to fabricate an argument. AWK is one of the classical UNIX tools. It's established and old. From [gawk manual](https://www.gnu.org/software/gawk/manual/html_node/History.html): > The original version of `awk` was written in 1977 at AT&T Bell > Laboratories. In 1985, a new version made the programming language more > powerful [...] This new version became widely available with Unix System > V Release 3.1 (1987). We're talking 40 years of history here. If the tool is established, that's one reason. Another reason why I've chosen AWK is that AWK has no JSON processing facilities, so it fills a gap in an existing tool. The third reason why I've chosen GNU AWK to create this plug-in is that gawk already has an extension mechanism. Therefore I don't have the freedom to modify the AWK processor to accommodate the JSON library idiosyncrasies. On the contrary, it's the JSON library the one to prove its flexibility. Now there's another question to answer here: why take MongoDB as inspiration to design the gap that would be filled in AWK (such as the `$unwind` operator)? And the answer will be the same. MongoDB is one of the JSON-oriented DBMS that got popular back in the NoSQL boom days. The end result tries to fit in the AWK's mindset and not the other way around. AWK is a tool designed for tabular data processing. JSON is in no way tabular, but its structure allows us to build a tabular view for the underlying data. There are two main variables in gawk to control how the fields are extracted: - `FS`: defines how to find the field _separators_. - `FPAT`: defines how to find the field _contents_. The plug-in just follows this trend and introduces a `JPAT` variable that also instructs how to find the field _contents_. `JPAT` is an array and each element has a [standard JSON Pointer](https://tools.ietf.org/html/rfc6901) expression that was designed exactly for this purpose. There's one very interesting fact about this outcome. The usage pattern here is removing from the library the step to process the stream (as the step that'd happen with the DOM approach) and moving much of the processing logic to be embedded in the parsing phase itself. That's a continuation from the trend we saw earlier about merging matching and decoding steps. It's also the same approach we'd use to build serious JSON processing tools such as [JMESPath](https://jmespath.org/) (some JMESPath expressions require a partial DOM, but it's doable to use a hybrid approach anyway as it has already been demonstrated in the `JUNWIND` code). ## Appendix C: Static decision trees with Boost.Hana The reader may be interested to know that the decision tree auxiliar to the parsing process developed for our (GNU) AWK plug-in is not only useful for AWK but to C++ programs as well. The decision tree strategy may even use Boost.Hana CT structures to fully unroll the code. It can be used to implement facilities such as: ```cpp using hana::literals::operator""_s; int foo; std::string bar; json::scanf(input, "{ foo: { bar: ? }, bar: ? }"_s, foo, bar); ``` That's also possible with Boost.JSON's current parser. But the following is not: ```cpp using hana::literals::operator""_s; Foo foo; std::string bar; json::scanf( input, "{ foo: { bar: ? }, bar: ? }"_s, [&foo](json::reader& reader) { // ... }, bar); ``` As it has been explained before, push parsers don't compose. And you aren't limited to root-level scanning. You should have `json::partial::scanf()` to act on subtrees too. A prototype for this idea can be found at <https://github.com/breese/trial.protocol/pull/43>. ## Review questions > Please be explicit about your decision (ACCEPT or REJECT). REJECT. > How much effort did you put into your evaluation? A glance? A quick > reading? In-depth study? I've stalled a project of mine for a few weeks just to participate in this review process. I think I've spent close to 50 hours. I wrote a JSON tool in Boost.JSON and the same tool under a different parser model to highlight important traits that were being dismissed in previous debates. I've gone through the documentation and source code to better understand some Boost.JSON details. I've also been involved in previous Boost.JSON debates in this very mailing list. > Are you knowledgeable about the problem domain? I've tested many JSON parser libraries in C++ to this point. In my last job I have leveraged a JSON (pull) parser to merge multiple validation layers into an one-pass operation and later leveraged Boost.Hana to automate the parser generation. Our API is JSON-based and we apply plenty of transformations (including DOM-based when unavoidable), some similar to the GNU AWK plug-in demonstrated here (e.g. synthesizing a new field on routed messages while DOM tree related costs are avoided). I've contributed a few algorithms and subtree parsing ideas back to the leveraged JSON (pull) parser. I offered the same composability ideas to Boost.JSON a few months back when the topic appeared here in this very mailing list. In the past, I've designed parsers in C++ -- pull and push parsers. I've played with some tools for parser generation such as re2c, gperf (gperf might not be really parsing, but it is related), and Boost.Xpressive. I've also experienced the pain of parsers that try to be too lightweight but ultimately backfire by moving not only performance costs back to the consumer but also greatly increasing complexity. -- Vinícius dos Santos Oliveira https://vinipsmaker.github.io/

4 6

[review][json] Boost.JSON review
by Peter Dimov 19 Sep '20

19 Sep '20

This is my review of Boost.JSON by Vinnie Falco and Krystian Stasiowski. First, a disclaimer: I have contributed to the library, mostly with design advice, occasionally to the implementation. Second, let's start with the end. I think that Boost.JSON should be ACCEPTED. All of its aspects are of Boost quality, its maintainers are qualified and responsive, the lack of a JSON library in Boost is frankly embarrassing, and the json::value type serves as a standard variant for scripting and transferring language-independent scalar and structured values, which opens many possibilities for further integration with future parts of the Boost ecosystem (I'll show an example of that later on.) To evaluate the library (I was already broadly, but not intimately, familiar with the structure and the interface), I started with the documentation, specifically with the Quick Look and Usage sections. It's very well written, and meets and exceeds the usual Boost level. I only have a few remarks to make. In Using Numbers, the documentation should explain more clearly and prominently that v.as_double() does not succeed when v contains an integer, but throws an exception. This is a common user expectation. Instead of focusing on number_cast, It should advertise value_to<double>(v) as the way to obtain a double from v, converting if necessary; and similarly, value_to<int>(v) as the way to obtain an int, range-checking if necessary. number_cast should be considered legacy at this point, and not featured. I understand that it offers an error_code& overload for those who don't want exceptions, but the throwing version is entirely superseded by value_to. In Parsing, the examples using parse_options look like this: parse_options opt; opt.allow_comments = true; opt.allow_trailing_commas = true; opt.allow_invalid_utf8 = true; value jv = parse( "[1,2,3,] // comment, extra comma ", storage_ptr(), opt ); That's fine and is indeed the C++11 standard-compatible way to do things. However, when designated initializers are available, such as on g++ or clang++, or on VS2019 with /std:c++latest, the above can be written as value jv = parse( "[1,2,3,] // comment, extra comma ", {}, { .allow_comments = true, .allow_invalid_commas = true, .allow_invalid_utf8 = true } ); and it might be useful to show this style too. In Value Conversion, Converting to Foreign Types, it's shown that defining a constructor customer::customer( value const& jv ); enables value_to<customer>(jv) to work. This is, in my opinion, a legacy mechanism made obsolete by tag_invoke and should not be supported. This example should be removed from the documentation, and only the tag_invoke way should be featured. And since we're on this page, it's odd that the StringLike requirement wants construction from (char*, std::size_t) - pointer to non-const characters. I don't think that any string type has such a constructor, they all take char const*. With the documentation carefully read (as in, quickly scanned), I proceeded to try to use the library. A previous reviewer (Reiner Deyke) had brought up the subject of CBOR parsing and serialization, and I had looked into CBOR as a format. It seemed regular and easy to parse and serialize, so I decided to write a CBOR serializer and parser for boost::json::value. How hard can that be, really? Not that hard. The serializer was quickly up and running, and it seemed quite performant: Serializing canada.json to CBOR: 1056200 bytes, 4840 us Serializing canada.json to JSON: 2306988 bytes, 13036 us (You can find the code at https://github.com/pdimov/boost_json_review/blob/master/boost_json_cbor.cpp, the serializer is the first 140 or so lines.) As far as using the library went, it was completely painless and straightforward. Next up, the parser. I decided to write it using the json::value interface, like a reference implementation. This, too, went well, and the parser was soon parsing, and even producing the same value as the one serialized. The performance, though... Parsing canada.json from JSON: 10548 us Parsing canada.json from CBOR: 33941 us, successful roundtrip This wasn't good. Switching to a binary format such as CBOR is supposed to be faster, not 3.4 times slower! Before you say "well your parser is written like you were a college student who was taught Java", be aware that I actually refactored the parser a number of times; sometimes it was written in one style, sometimes in another, functions went from being hand-inlined to extracted and back a few times, but none of that made a dent on those 34ms. To find out where the time was going, I looked into way more detail I needed to. For instance, I saw that assigning f.ex. 1.0 to a default-constructed json::value executes much more code than it needs to, and makes non-inlined calls into libboost_json. One would expect this to take one check and branch (do we need to deallocate anything? no), and two assignments (kind = kind_double; dbl_ = 1.0). Turns out that the efficient way to assign 1.0 is not `v = 1.0;`, but `v.emplace_double() = 1.0;`. I of course replaced the assignments with this supposedly efficient form, and it of course made no difference at all. I then in desperation actually removed all the assignments. And it barely made any difference. At this point I decided that the json::value interface is not the right fit for a performant parser, and switched to using boost::json::value_stack. This is what Boost.JSON's parser itself uses, so if that's not fast, well, I don't know what will be. Updating the parser to use value_stack was basically trivial: https://github.com/pdimov/boost_json_review/commit/7aa8d2dc8c706c3f4204d6fe… and it worked. The performance was what I expected: Parsing canada.json from JSON: 10044 us Parsing canada.json from CBOR: 6332 us, successful roundtrip (You can find the value_stack parser at https://github.com/pdimov/boost_json_review/blob/master/boost_json_cbor_st.… and the results at https://github.com/pdimov/boost_json_review/blob/master/boost_json_cbor_st.….) (Since now changes to the parser code actually affected performance, I added two optimizations to the array parsing - a fast path for arrays containing only doubles, and a fast path for arrays containing only integers.) Is value hopeless for parsing, then? I went back to the other parser. This is how the part that parses objects looks like, with the irrelevant parts omitted: boost::json::object & o = v.emplace_object(); o.reserve( n ); for( std::size_t i = 0; i < n; ++i ) { // key string boost::json::string_view sv( ... ); // value boost::json::value w( v.storage() ); first = parse_cbor_value( first, last, w ); o.insert( boost::json::key_value_pair( sv, std::move( w ) ) ); } The reason I use o.insert here is that I wanted the CBOR parser to match the behavior of the JSON parser with respect to key order and duplicate keys. The JSON parser preserves key order, which is both user-friendly and better from a security perspective (observing key order may leak the seed of the hash function.) It also handles duplicates in an unspecified manner. Well, whatever that unspecified manner was, I wanted to do the same, and using `insert` looked like the way to do it. There however was no `insert` taking a string_view and a value, so I had to create a key_value_pair. There was `insert_or_assign` that did take a string_view and a value, but it overwrites duplicates, and I didn't want that. I consulted Vinnie Falco about the lack of `insert(string_view, value)`, and he pointed me at `emplace`. This sounded exactly like what my hypothetical `insert` would do. However, hearing `emplace` gave me an idea. In the same way `v.emplace_object()` gives me `object&`, `o.emplace(sv)` could have given me `value&`, which I could then pass directly to `parse_cbor_value`: first = parse_cbor_value( first, last, o.emplace(sv) ); I shared this brilliant observation with Vinnie, and he replied "obj[sv]". obj[sv] indeed. Sometimes we forget that C++98 did have a useful feature or two. This doesn't exactly work as I wanted it to, because it neither guarantees insertion order, nor has the "right" duplicate handling. Nevertheless, I did make the change: https://github.com/pdimov/boost_json_review/commit/e6baf8f1ce81966534ac93c5… and guess what? Parsing canada.json from JSON: 10261 us Parsing canada.json from CBOR: 6761 us, successful roundtrip So the problem the entire time was that one single line: o.insert( boost::json::key_value_pair( sv, std::move( w ) ) ); I haven't looked into it, but I suspect that it for some reason makes a copy of `w`, even though it's passed using `std::move` and uses the right `storage_ptr`. Oh well. At least it works now. So, there is nothing that much wrong with the `value` interface, and it's possible to build values the straightforward way, instead of using a value_stack, without significant loss in performance. These CBOR parsers are entirely functional, and only have the following limitations: * Binary strings are not supported (as they're not representable in json::value); * Infinite sequences are not supported (because I was too lazy); * 32 bit platforms need to check the string/array/object sizes for possible size_t overflow (because I was way too lazy.) I had also looked at, and tried, the value conversion interface, in the course of developing the examples of an unreleased library of mine, Describe. The value_from example is https://pdimov.github.io/describe/doc/html/describe.html#example_to_json value_to is at https://pdimov.github.io/describe/doc/html/describe.html#example_from_json and a simple JSON-RPC example is at https://pdimov.github.io/describe/doc/html/describe.html#example_json_rpc These all worked without any trouble, except I had to link with Boost.Container for one of them, which didn't seem necessary. They are also a good illustration of the possibilities that having a standard json::value type presents, in terms of further library development. (One could f.ex. imagine a Boost.Script interpreter that uses json::value as its data type, and can be used to script C++ code.) In conclusion, the library works, is well documented, and should be ACCEPTED. I have no acceptance conditions. Obviously, the performance problem concerning object::insert needs to be addressed, and more generally, the value interface needs a bit more performance scrutiny because it's not currently exercised by the benchmarks, but I trust that this will be done. Solid work, thanks to the authors for their submission, and thank you for reading this far.

2 1

Github CI testing with appveyor and travis
by Edward Diener 19 Sep '20

19 Sep '20

I can not seem to get appveyor or travis CI to test the repositories of which I am the maintainer, PP, VMD, and TTI, any more. Normally I would not care much about the CI testing, but the fact that they seem to have stopped wortking completely, or maybe just stopped sending me back results, bothers me. All three have appropriate .yml files. Have these CI services stopped working ? How do I integrate them again with these Boost libraries ?

3 8

[JSON][review] a small review of boost.json
by Julien Blanc 19 Sep '20

19 Sep '20

Hi, Here's my small review of boost json. ## Are you knowledgeable about the problem domain? I've written a json library (for reference: https://github.com/Julien-Blanc-tgcm/jbc-json ) which i used as playground / experiments. I guess this makes me knowledgeable enough of the domain. ## What is your evaluation of the design? I think the library is well designed, although it has several shortcomings. These shortcomings may not alter general usage of the library (with main target seeming to be REST api over http), but render it useless in specific cases. For me, a json library consists of three independant parts and some glue: * a parser * a type for representing values (usually a variant) * a printer (serializer) ### The parser In boost.json, the separation between the parser and seems neat enough (ie, you can use the parser without the json::value type). However, it resides in a <detail> header, which would indicate that it is not an expected use case. I believe it should be. The callbacks of the parser (push parser) are not documented. I'm a bit puzzled by what the on_number_part does, and how it fits with the on_int64/on_double callbacks, for example. The parser is an incremental push parser. However, it seems to expect to be feeded with partial json documents, but not consecutive json documents. I could not find a way to make : auto v = p.write(R"json({"foo":1}{"bar":2})json"); return 9. I may have missed something, but this use case is IMHO important to address. Since the only way to know that a json document has ended is to parse it, the parser should provide an option to parse a stream containing multiple consecutive json documents (ie, stopping without error at the end of each document). A common use case for this is JSON notifications received via raw sockets, so no envelope may be used to isolate JSON documents. ### The value type The value type is a variant, the kind that you would expect from a json library. Most of the interface uses string_view, which is a very good thing (the fact that boost.json uses another string type internally has not been an issue in my tests, due to the usage of string_view). It lacks visitation, though (IIRC there’s an open issue for this). The fact that this value type has been highly optimized is the great strength of this library. #### objects The object interface relies on the [] operator, which incurs the same shortcomings as with std::map : auto v = foo["bar"]; // will silently create bar if it does not exists Typos in keys had always been a source of hard to find bugs, i wished there was an option to disable operator[] and force the use of find. The library has no support for object with duplicate keys, which also are valid json. This, however, can be worked around by using the parser callbacks and not the variant (but that makes it difficult to produce such documents, cf later). Whether the order of the keys is kept by the object structure or not does not appear in the documentation. From my small testing, it seems that it is the case, but the api does not offer any interface to add elements at specific position, so i would assume it is not guaranted. The json specification, AFAIR, does not forbid to rely on a specific order of the keys. Some serialization libraries do expect the keys to be in a specific order. #### numbers Having two number types can lead to unexpected results : auto v = boost::json::parse("{\"foo\":1,\"bar\":2.0}"); std::cout << v.as_object()["bar"].as_double() << std::endl; // ok std::cout << " " << v.as_object()["foo"].as_double(); // asserts It is unexpected, because several json libraries will never output “2E0” or “2.0” for the value “2”, but a plain “2”. It means that whenever you expect a double value, you also have to check for the possibility that it has been parsed as an integer. The to_double function should silently convert integers to their double equivalent whenever it is possible. * as has already been noted by others, the number precision is hardcoded to IEE754 or 64bits integer. This is seldom an issue, since it is the case with most implementations. #### The printer The printer is not usable without the value item. I would expect it to be (producing json without resorting to an intermediate representation). Apart from this, it supports incremental printing, which is what is expected. It produces a strange representation for numbers, but it is conforming to the JSON specification. Because it is tied to the json::value, it cannot produce anything that cannot be represented by a json::value. This is an issue that should be addressed. ## What is your evaluation of the implementation? I did not have a thorough look into the implementation details. From what i have seen, the implementation looks fine. A parser is usually a landmine for security exposures, and would need a very careful review which i cannot do. IIRC the authors plan on having an independant audit on the library, which is a good thing. There is also fuzzing and tests, so we can expect that this point is correctly being taken care of. The rest of the code looks fine, easy to navigate into. ## What is your evaluation of the documentation? Very good. Easy to use, complete. There is some missing information, but they all seems related to design decision that have been taken implicitly (already discussed in design section). An example on using the parser on standalone mode would be a very welcome addition. Another thing that the doc should state clearly : boost.json is **not** a JSON (ECMA-404) library, but rather an I-JSON (RFC 7493) library with some extensions. ## What is your evaluation of the potential usefulness of the library? Very useful. While it does not address all use cases (and does not try to), it addresses what is probable the *most common* case. As JSON is very widespread, i expect this library to find a large audience, because it combines ease of use and performance. ## Did you try to use the library? With what compiler? Did you have any problems? I tried to use it on a small toy application, on a debian stable system. The main target seems to be to use the library with boost. I ran into several issues when trying to use the standalone version. The first one was easy to address, and due to gcc 8.3 lacking c++17 support. A patch was submitted, and merged quickly. The second one was incompatibilites between standalone asio bringing boost headers (because boost is installed on my machine) and json redefining them (because of standalone mode), resulting in multiple defenitions of throw_exception. I'm not sure yet if i should file this to boost.json, asio library, or asio debian package, or if it is just a missing define on my part. At that point, i switched to boost json, which fixed the issues i had. ## How much effort did you put into your evaluation? I'd say around 10 hours reading, testing and playing, including writing of the review. ## Do you think the library should be accepted as a Boost library? I think the library should be ACCEPTED. There are shortcomings in the library, it does not address every use case in the world, but it does a very good job at what it is designed to do. It is easy to use, performant, and it addresses a very widespread use case, so the pros clearly outweighs the cons. Regards, Julien

3 3

[preprocessor] Dropping support for compilers that do not implement variadic macros
by Edward Diener 18 Sep '20

18 Sep '20

In lieu with Boost's decision regarding dropping support for C++03 compilation, I would like to drop support in the preprocessor library for preprocessors which do not implement variadic macros. Variadic macros were added in the C++11 standard, although a number of compilers supported them even before the official C++ standard. I will of course keep the same macro APIs, although deprecating non-variadic macro forms when alternate variadic macro forms exist. This will simplify Boost PP. Ideally, once Boost PP supports only preprocessors which implement variadic macros, Boost VMD can be added to Boost PP, for anyone who wants to use its functionality, and Boost VMD can then be retired.

6 25

[JSON][review] Design: what is expected of a "JSON library"?
by Andrzej Krzemienski 18 Sep '20

18 Sep '20

Hi Everyone, This is not a complete Boost Review yet. In this email I wanted to discuss the design goals of the library. The point that I would like to raise is the tension between having relatively small types (int64_t, uint64_t, double) represent numbers in json::value on the one hand, and ECMA specification allowing arbitrarily big/precise numbers in JSON format on the other. Can I expect of a JSON library that when it converts a JSON contents into internal representation and then back to JSON contents, I should get the same contents (moduo white space)? This would be possible if the internal representation of JSON numbers was an arbitrary-precision decimal type or a string. But when we need to squeeze any number into 24 bits, we will soon get to the point when integer number `100000000000000000000001` after the experiment gets changed to `1E23`. Is this acceptable for a JSON library? But maybe it is not a valid goal? Maybe the goal of the JSON format is to have objects already created in internal representation converted to text and then back to objects? (assuming the recipient program is run on the same environment as the sender (no differences in word size or maximum int representation.) That is, as long as you agree to the constraints of json::value, whatever you manage to put inside, we guarantee that you get the same value when you serialize it and then parse it back. Boost.JSON does this nicely. In that other view, if I have objects of types `boost::multiprecision::cpp_int` my only option is to pass them as strings in JSON protocol. But I can pass any number as string anyway, so what is the use of numbers in JSON format? Uness it is just practical: you can choose to use numbers and then internal representations of JSON may be smaller. Do you get the concern that I am seeing? I mean I have used JSON libraries before, and this has never been a practical problem. But Boost Review puts the bar high for the libraries, so I guess this question should be answered: what guarantees can a JSON library give us with respect to accuracy of numbers? Regards, &rzej;

6 11

Re: [boost] [JSON[review] Andrzej's review
by Peter Dimov 17 Sep '20

17 Sep '20

> Vinnie Falco wrote: > > > Just open an issue and explain how you think self-swap, self-move, and > > self-assignment should work and I will make the changes. > > They should be no-ops. More specifically, self-swap should be a no-op, and assignments should be equivalent to: T& operator=( T const& rhs ) { T(rhs).swap(*this); return *this; } T& operator=( T&& rhs ) { T(std::move(rhs)).swap(*this); return *this; } This is important for classes such as json::value, object, array, where *this may own rhs, so it's important not to destroy rhs too early. E.g. v = v.at(0); v = std::move(v.at(0)); a = a.at(0).as_array(); a = std::move(a.at(0)).as_array(); o = o.at("key").as_object(); o = std::move(o.at("key").as_object()); This is not self-assignment, but it has the same lifetime pitfalls - if you destroy the old contents of *this, rhs is invalidated, so you can no longer copy (or move) it.

2 1

Interactive mail
by SIDDHARTH PANDEY 17 Sep '20

17 Sep '20

Hello everyone, myself Siddharth Pandey, a sophomore at Indian Institute of Information Technology Sri City, India. I love to do Competitive Programming and have a keen interest in Open Source Contribution. I have a strong foundation on C,C++ and some about Python & Java. Also I am trying to explore the field of Data Science & Machine learning. For more information do visit my Portfolio website https://siddharth25pandey.github.io/ I want to contribute to the Boost Organisation but I am quite confused how to start with. It will be really helpful if any mentor can guide me how to start Contributing to get a great learning from such a highly reputed organization. Thanks & Regards Siddharth Pandey

4 3

Boost Log 1.65.1 Linking Issue
by Karmethia Thompson 16 Sep '20

16 Sep '20

Good morning, I am trying to create a logger that uses Boost Log. I was successful getting my code to compile with 1.63.0, so I know that I am not too far off of the mark. cmake/make throws a linking issues whenever I attempt to create a sink: typedef sinks::synchronous_sink< sinks::text_ostream_backend > text_sink; boost::shared_ptr< text_sink > sink = boost::make_shared< text_sink >(); or create file sink logging::add_file_log ( keywords::file_name = log_specs.log_file, keywords::rotation_size = 10 * 1024 * 1024, keywords::time_based_rotation = sinks::file::rotation_at_time_point(0, 0, 0), keywords::filter = min_severity || severity >= file_severity, keywords::format = ( expr::stream << expr::format_date_time<boost::posix_time::ptime>("TimeStamp", "%Y-%m-%d_%H:%M:%S.%f") << ": [" << severity << "]" << expr::smessage ), keywords::auto_flush = true //flush to file.. ); My code will compile 95% and then produce the following: CMakeFiles/main.dir/src/Logger.cpp.o: In function `boost::enable_if_c<(boost::log::v2_mt_posix::aux::is_character_type<wchar_t>::value&&boost::log::v2_mt_posix::aux::is_character_type<char>::value)&&((sizeof (wchar_t))!=(sizeof (char))), bool>::type boost::log::v2_mt_posix::aux::code_convert<wchar_t, char, std::char_traits<char>, std::allocator<char> >(wchar_t const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, unsigned long, std::locale const&)': Logger.cpp:(.text._ZN5boost3log11v2_mt_posix3aux12code_convertIwcSt11char_traitsIcESaIcEEENS_11enable_if_cIXaaaasrNS2_17is_character_typeIT_EE5valuesrNS8_IT0_EE5valuenestS9_stSB_EbE4typeEPKS9_mRNSt7__cxx1112basic_stringISB_T1_T2_EEmRKSt6locale[_ZN5boost3log11v2_mt_posix3aux12code_convertIwcSt11char_traitsIcESaIcEEENS_11enable_if_cIXaaaasrNS2_17is_character_typeIT_EE5valuesrNS8_IT0_EE5valuenestS9_stSB_EbE4typeEPKS9_mRNSt7__cxx1112basic_stringISB_T1_T2_EEmRKSt6locale]+0x37): undefined reference to `boost::log::v2_mt_posix::aux::code_convert_impl(wchar_t const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, unsigned long, std::locale const&)' CMakeFiles/main.dir/src/Logger.cpp.o: In function `void boost::log::v2_mt_posix::sinks::basic_formatting_sink_frontend<char>::feed_record<boost::recursive_mutex, boost::log::v2_mt_posix::sinks::basic_text_ostream_backend<char> >(boost::log::v2_mt_posix::record_view const&, boost::recursive_mutex&, boost::log::v2_mt_posix::sinks::basic_text_ostream_backend<char>&)': Logger.cpp:(.text._ZN5boost3log11v2_mt_posix5sinks30basic_formatting_sink_frontendIcE11feed_recordINS_15recursive_mutexENS2_26basic_text_ostream_backendIcEEEEvRKNS1_11record_viewERT_RT0_[_ZN5boost3log11v2_mt_posix5sinks30basic_formatting_sink_frontendIcE11feed_recordINS_15recursive_mutexENS2_26basic_text_ostream_backendIcEEEEvRKNS1_11record_viewERT_RT0_]+0x141): undefined reference to `boost::log::v2_mt_posix::sinks::basic_text_ostream_backend<char>::consume(boost::log::v2_mt_posix::record_view const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' CMakeFiles/main.dir/src/Logger.cpp.o: In function `void boost::log::v2_mt_posix::sinks::basic_formatting_sink_frontend<char>::feed_record<boost::recursive_mutex, boost::log::v2_mt_posix::sinks::text_file_backend>(boost::log::v2_mt_posix::record_view const&, boost::recursive_mutex&, boost::log::v2_mt_posix::sinks::text_file_backend&)': Logger.cpp:(.text._ZN5boost3log11v2_mt_posix5sinks30basic_formatting_sink_frontendIcE11feed_recordINS_15recursive_mutexENS2_17text_file_backendEEEvRKNS1_11record_viewERT_RT0_[_ZN5boost3log11v2_mt_posix5sinks30basic_formatting_sink_frontendIcE11feed_recordINS_15recursive_mutexENS2_17text_file_backendEEEvRKNS1_11record_viewERT_RT0_]+0x141): undefined reference to `boost::log::v2_mt_posix::sinks::text_file_backend::consume(boost::log::v2_mt_posix::record_view const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' CMakeFiles/main.dir/src/Logger.cpp.o: In function `void boost::log::v2_mt_posix::sinks::basic_formatting_sink_frontend<char>::feed_record<boost::log::v2_mt_posix::aux::fake_mutex, boost::log::v2_mt_posix::sinks::basic_text_ostream_backend<char> >(boost::log::v2_mt_posix::record_view const&, boost::log::v2_mt_posix::aux::fake_mutex&, boost::log::v2_mt_posix::sinks::basic_text_ostream_backend<char>&)': Logger.cpp:(.text._ZN5boost3log11v2_mt_posix5sinks30basic_formatting_sink_frontendIcE11feed_recordINS1_3aux10fake_mutexENS2_26basic_text_ostream_backendIcEEEEvRKNS1_11record_viewERT_RT0_[_ZN5boost3log11v2_mt_posix5sinks30basic_formatting_sink_frontendIcE11feed_recordINS1_3aux10fake_mutexENS2_26basic_text_ostream_backendIcEEEEvRKNS1_11record_viewERT_RT0_]+0x141): undefined reference to `boost::log::v2_mt_posix::sinks::basic_text_ostream_backend<char>::consume(boost::log::v2_mt_posix::record_view const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' CMakeFiles/main.dir/src/Logger.cpp.o: In function `void boost::log::v2_mt_posix::sinks::basic_formatting_sink_frontend<char>::feed_record<boost::log::v2_mt_posix::aux::fake_mutex, boost::log::v2_mt_posix::sinks::text_file_backend>(boost::log::v2_mt_posix::record_view const&, boost::log::v2_mt_posix::aux::fake_mutex&, boost::log::v2_mt_posix::sinks::text_file_backend&)': Logger.cpp:(.text._ZN5boost3log11v2_mt_posix5sinks30basic_formatting_sink_frontendIcE11feed_recordINS1_3aux10fake_mutexENS2_17text_file_backendEEEvRKNS1_11record_viewERT_RT0_[_ZN5boost3log11v2_mt_posix5sinks30basic_formatting_sink_frontendIcE11feed_recordINS1_3aux10fake_mutexENS2_17text_file_backendEEEvRKNS1_11record_viewERT_RT0_]+0x141): undefined reference to `boost::log::v2_mt_posix::sinks::text_file_backend::consume(boost::log::v2_mt_posix::record_view const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' collect2: error: ld returned 1 exit status make[2]: *** [main] Error 1 make[1]: *** [CMakeFiles/main.dir/all] Error 2 Can someone please help? I have tried everything from trying to dynamically and statically link. I've added both "log" and "log_setup" to the find_packages in cmake and have also explicitly specified the locations of those libraries as well. Please let me know if there is anything else to try. Thanks -- Karmethia Thompson Applied Mathematics, PhD

4 6

Re: [boost] Boost Log 1.65.1 Linking Issue
by Peter Dimov 15 Sep '20

15 Sep '20

Although, now that I think of it, the shown CMakeLists.txt file didn't use imported targets, so perhaps the order of libraries is literal and putting log_setup before log may indeed solve things. -----Original Message----- From: Peter Dimov Sent: Tuesday, September 15, 2020 20:28 To: boost(a)lists.boost.org Subject: Re: [boost] Boost Log 1.65.1 Linking Issue Andrey Semashev wrote: > I'm not sure, but I think the above would generate incorrect order of > libraries in the linker command line. You need to ensure that boost_log > comes after boost_log_setup, and their dependencies come after still. CMake automatically orders libraries so that the dependencies come after. Unfortunately, FindBoost has a bug and declares log as depending on log_setup, rather than vice versa. https://github.com/Kitware/CMake/blob/bdcde7762c71cd27fcba6dc1a3efe12fa7a09… No matter how you order the libraries in target_link_libraries, the order between these two will never be correct. Not sure if this is the problem here though.

1 0

[Math] Adding prime sieves - design discussion
by John Maddock 15 Sep '20

15 Sep '20

Just a heads up, that Boost.Math has an open PR for adding prime sieves, and there is an ongoing design discussion on interface etc. If anyone has any strong feelings on the matter, usage requirements etc, then comments would be most welcome. See https://github.com/boostorg/math/pull/400. Many thanks, John. -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

1 0

[JSON][review] Some early feedback
by Andrzej Krzemienski 15 Sep '20

15 Sep '20

Hi Everyone, The formal review of Boost.JSON has not started yet, but I thought I could offer some feedback from my playing with the library earlier in the process. Documentation --------------- Very well written. It makes it easy to grasp quickly the spirit and the potential of the library, and the reference section makes it easy to navigate through the features and their interfaces. Some minor issues: 1. Section "Using Numbers" is missing content. 2. Description of storage_ptr's destructor and move constructor is missing noexcept. 3. Docs in a couple of places call `storage_ptr` a "smart pointer container". Is it intentional? "Container" has a very specific meaning in C++. Design. ----------- It is generally sound, and well explained in the rationale. Here are a couple of minor remarks. 1. The usage of `nullptr` for null values. `nullptr` has been introduced to mean "a pointer with null value" and is only used in the standard library for smart pointers, with an ignoble exception of `std::function<>`. Using it for `json::value` is not an obvious choice for me. On the other hand, we do not have a universal type representing a missing value. A library could provide its own `json::null`, although I am not sure if this is an ideal solution either. 2. Why is json::value Semiregular rather than Regular? Equality comparison has a natural interpretation and application for checking if the produced JSON looks exactly as expected, or for checking if some two subtrees are the same. Is it because of efficiency issues with json::object comparison? 3. The library is using a custom string type. We get a rationale for this, and I accept it. But there are still some concerns about it that I would like to mention. 3.1. It is not the first library reviewed his year to have a custom type with std::string-compatible interface. It looks like the idea of having a "vocabulary" string type is unrealistic. If this should be the case, maybe we need a concept specifying what a string-compatible type should provide. It is obvious that json::string does not literally provide the same interface as std::string. Nor does anyone expect 100% compatibility. 3.2. Docs say that json::string is like std::string *except for* a number of things. For instance it does not provide the controversial assignment from `char`. Given that you are "fixing" the shortcomings of std::string, maybe it is an opportunity to fix other things in it also. For instance, std::string has been criticised for having too big an interface. That it mixes an iterator-based interface with an indexed-based one. Myb functions like `find_first_not_of()` are not necessary, given that users can use all sorts of iterator-based algorithms? 3.3. The rationale for json::string says that inisializer_list constructor is unnecessary, because string literal can always be used instead. This does not take into account that initializer_list does not necessarily operate on constant values. There is no concise substitute for: ``` std::string arrange(char a, char b, char c) { return {c, a, b, c}; } ``` I would not miss this construction much, I guess, but I would expect this to be mentioned in the rationale. 4. One thing one often expects of a JSON library is to be able to pretty-print a JSON document. The docs give an example how to do it, but the library does not provide this feature out of the box. Is pretty-printing not compatible with the design goals of the library? Implementation -------------- I didn't look much at it yet, but here is something I found. 1. `json::string_view` is convertible from `std::stirng`, but it is not convertible from `std::string_view`. I think there is no harm in enabling this conversion, and it seems useful. 2. The following piece of code ``` json::value v = 50.0; std::cout << json::serialize(v); ``` oututs: 5E1 which is technically compliant with JSON specification, but quite surprising. Is there a reason for this? Can serialization be controlled to output 50.0 or 50? 3. It is confusing for me to see member function serializer::read. I would think that the action that a serializer does is *write* rather than read. Am I missing something? 4. I really appreciate the ability to use the library in header-only mode with <boost/json/src.hpp>. This makes library testing much simpler. Regards, &rzej;

7 10

[json][review] A review from an embedded perspective
by Brooks Prumo 15 Sep '20

15 Sep '20

Hi everyone, this is my first time mailing the Boost developer list, and also submitting a review. Thank you in advance for understanding any issues it may have! I'm writing to review Boost JSON. I've been using it very lightly for the last week or so. My usage has been exclusively on embedded MCUs (ARM Cortex M4, compiling with ARM GCC 9 with C++17), which I'm guessing hasn't been the common case so far. Here's a list of things that matter a lot to me: - Compile from source. Boost JSON passes. - No exceptions. C++ exceptions are turned off, so all libraries I use must support not throwing. Boost JSON passes. - No iostreams/locale. If a library uses iostreams, my binary jumps ~220 KB which is a deal breaker. Boost JSON passes. - Binary size matters a lot, since onboard Flash is very limited. I am using a "big" MCU, and that means my whole binary must be under ~512 KB. Right now Boost JSON is not great. In doing a simple serialization and parse, the library itself adds ~58 KB to my .text section. This is OK for now, but surprisingly big. I know the team is aware of this and has some ideas to reduce the binary size. I will likely need those fixes to actually port my app to Boost JSON. - Control over the sizes of any "large" stack buffers. Boost JSON passes. - No external dependencies that do not meet the above requirements. Boost JSON passes. With this, Boost JSON is an ACCEPT from me. THe biggest reason for me is that the API is nice to work with. I started off with Nlohmann, but the binary size was too big due to iostreams. I'm currently using RapidJSON, and it's definitely not easy to use. In my app, JSON is not in the hot path. I want a library that is easy to use and meets my requirements above. I greatly appreciate being able to use a modern C++ library. And answers to the other questions: - What is your evaluation of the design? So far so good. I do not have complicated JSON uses though. - What is your evaluation of the implementation? I've only looked at the implementation when it comes to binary size. - What is your evaluation of the documentation? Documentation is nice. I personally would enjoy more examples, as that is how I commonly learn new libraries. - What is your evaluation of the potential usefulness of the library? HUGE! I haven't come across another modern C++ JSON library that'll work in the embedded space I live in. It's really exciting! - Did you try to use the library? With which compiler(s)? Did you have any problems? Yep! I used ARM GCC 9.2.1 with C++17. No issues with compilation. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I'd say my evaluation is "initial". I've used it in a way that mimics my real usage, but haven't used many aspects of the whole library. - Are you knowledgeable about the problem domain? Minimally. Just basic JSON knowledge. Thank you! Brooks

3 2

[preprocessor] Appveyor weirdness
by Edward Diener 14 Sep '20

14 Sep '20

In my appveyor.yml file for preprocessor one of the b2 command lines for a test run uses an explicit alias for the target. When run in appveyor.yml I get the messages: notice: could not find main target preprocessor_limits notice: assuming it is a name of file to create . and don't know how to make <e>preprocessor_limits Yet when I run the same test locally using the preprocessor_limits target, everything works fine. Is Appveyor somehow different when running Boost Build tests than when I run it locally ?

1 0

No doc for qvm in 1.74 distro
by Edward Diener 12 Sep '20

12 Sep '20

There is no doc for the qvm library in the 1.74 distro.

2 1

[JSON][review] Should there be an equality operator/function?
by Hadriel Kaplan 12 Sep '20

12 Sep '20

Howdy, I’ve been testing Boost.JSON in preparation for writing a review, and one thing that I found surprising might be better discussed in a separate thread, so here goes… There’s currently no comparison function or operator to compare one json::value with another. While I appreciate that a deep comparison isn’t cheap - it’s O(N) for the average case but O(N^2) in the worst case I believe - I still think it’ something a JSON-based container value type should have out-of-the-box. It’s something other popular JSON libraries offer; usually with equality operators, although a named function would be ok too if that was required instead. I hit the lack of it almost immediately, since test code/assertions frequently check equality. I’m not sure how frequently it’s used for run-time code in general, however. -hadriel

1 1

Looking for a review manager for Real
by Damian Vicino 10 Sep '20

10 Sep '20

Hi, GSOC finished and we are almost-ready for proposing to review a new library, Real. We are looking for a volunteer to be a review manager for it. Best regards, Damian

9 15

Do I need to use boost::mutex in deadline_timer?
by Jupiter 10 Sep '20

10 Sep '20

Hi, I used boost::asio::io_service for a single thread implementation but with multiple boost::asio::deadline_timer objects, because it is a single thread, I don't think I need to use boost::mutex to lock and unlock tasks running by deadline_timer, am I wrong about it? Thank you. Kind regards, jupiter

3 4

Re: [boost] boost.real
by Kostas Savvidis 09 Sep '20

09 Sep '20

Hi Damian, Laouen, I have been able to compile the test programs with your library after making some simple changes. The patch is below the message. However the example program from your documentation still does not compile, unless I comment out some lines - you can see the listing below. I ran a tricky calculation which is from the Aberth book, your program is able to give the answer and moreover the true value is indeed between the upper and lower bounds that your program gives. This is good! The conclusion is that your library can do useful work, but it is very limited in capability. No special functions, trig is there but not working well, and no simple way to interface with linear algebra. I may want to use it at some point, but I would need at least the ability to cast int and double to your boost::real::real type, because as it is now, the only way to import numbers into your software seems to be through std::string. Finally what seem to be some bugs: 1) unsigned int real::maximum_precision = 10; // error: specializing member 'boost::real::real<int>::maximum_precision' requires 'template<>' syntax 2) auto it = e.cbegin(); // error: 'using real = class boost::real::real<int>' {aka 'class boost::real::real<int>'} has no member named 'cbegin' 3) real::sin(f) // enters infinite loop 4) real c = (std::string)"0.999999"; prints: c: [0.99999899999999999999999999999999999999999999999999999999999999999999999999999999999999999933600778729, 0.99999899999999999999999999999999999999999999999999999999999999999999999999999999999999999982691714296] as far as i can see the interval is incorrect ! Best Regards, Kostas ============================================================================================= diff -r -u real.orig/integer_number.hpp real/integer_number.hpp --- real.orig/integer_number.hpp 2020-09-07 16:22:04.000000000 +0300 +++ real/integer_number.hpp 2020-09-08 14:11:22.000000000 +0300 @@ -103,7 +103,8 @@ } // this function always expects (*this) to bigger than other, so if borrow is not zero at the end // there should be error - if(borrow != 0); + if(borrow != 0) + ; //throw subtraction_number_required_lower_exception(); diff -r -u real.orig/real_rational.hpp real/real_rational.hpp --- real.orig/real_rational.hpp 2020-09-07 16:22:04.000000000 +0300 +++ real/real_rational.hpp 2020-09-08 11:59:51.000000000 +0300 @@ -262,7 +262,7 @@ // constructor to generate get number from string constexpr explicit real_rational(std::string_view num){ // searching for "/" in string - size_t pos; + size_t pos=0; bool found = false; std::string_view num1, num2; pos = num.size(); ============================================================================================= #include <iostream> #include <string> #include <real/real.hpp> using real = boost::real::real<int>; //unsigned int real::maximum_precision = 10; //unsigned int real_algorithm::maximum_precision = 10; int main() { real c = (std::string)"0.999999"; real d = (std::string)"0.999999"; real e = c + d; std::cout << "c: " << c << std::endl; std::cout << "d: " << d << std::endl; std::cout << "e=c+d: " << e << std::endl; // ANSWER IS WRONG ??!!! // for(auto it = e.cbegin(); it != e.cend(); ++it) { // std::cout << "e iteration " << it.approximation_interval << std::endl; // } real g = (std::string)"0.999998"; std::cout << "g: " << g << std::endl; if (g < d) { std::cout << "g < d --> true" << std::endl; } else { std::cout << "g < d --> false" << std::endl; } real h = d - g; std::cout << "h=d-g: " << h << std::endl; std::cout << "h++ is: " << h+real("1.0") << std::endl; { // Example from Aberth 1992, Precise Computation Using Range Arithmetic real a("77617.0"); real b("33096.0"); std::cout << "\n\n" << "a: " << a << std::endl; std::cout << "b: " << b << std::endl; real f = real("333.75")*b*b*b*b*b*b + a*a*(real("11.0")*a*a*b*b-b*b*b*b*b*b - real("121.0")*b*b*b*b-real("2.0")) + real("5.5")*b*b*b*b*b*b*b*b + a/(real("2.0")*b); // should be -0.827396 std::cout << "f: " << f << std::endl; //std::cout << "sin(f): " << real::sin(f) << std::endl; // doesn't come out of the loop } return 0; }

2 1

[LEAF] Review Results
by Michael Caisse 09 Sep '20

09 Sep '20

I want to thank Emil for his patience for the publishing of the review results as life got in my way. Thank you Emil for submitting LEAF to be reviewed for inclusion into Boost. As an already-Boost-author, you knew what you were getting into and continued into the process. I also want to thank you for once again pushing our thoughts on error handling. -------- Result -------- This is the part everybody scrolls to the bottom to read, so let’s just put it at the top. Boost.LEAF is ACCEPTED into Boost. After the following modifications are made to the library, the repository can be transitioned to Boost for inclusion into the next release (with the release cycle timing constraints): Specify the TLS requirement sooner in the documentation. Optional usage of Boost.Config to perform all of the Boost Configy things. (further discussion on this below) Macro naming and general Boostification clean-up. -------- Reviews -------- There was a good amount of discussion about the library on both the dev and user mail lists. I want to personally thank those who got involved in the review. You make the process work! Result summary: Accept: 7 Reject: 1 No-decision: 3 The no-decision reviews provided valuable input and discussion threads. While it would be my preference to have the review authors provide a final verdict, I was grateful for the time and energy they spent with the library and formulating thoughts into emails for discussion. Some comments from reviews: “I find function preload() really great and novel. … This really makes the error reporting code brief, clean and bug-free.” “I like LEAF very much and there's a lot offered here…” “I like it.” “...it allows bridging multiple error handling strategies and usage patterns in an efficient way.” About the docs: “They look good and manage to explain hard concepts in a very approachable fashion.” “The documentation is great -- in terms of styles it's probably the best I've seen for a C++ library.” “Clear, friendly and complete.” -------------- Some Thoughts -------------- While there was overwhelming consensus that LEAF should be included into Boost, the process is not democratic. I have spent considerable time understanding the concerns raised in each of the reviews and in particular the singular rejection. Emil took time responding to each of the reviews. The exchanges were fruitful and both “conditional accept” and “reject” reviewers changed their verdicts to accept along the way. Most of the concerns were amicably resolved. I am comfortable with the design choices and implementation made by the author for the remaining items. One reviewer brought up the re-implementation of Boost.Config type facilities. This of course is a complex issue for a library with zero dependencies. The ability to simply drop the Boost.LEAF directory into a project and run is a definite plus for a specific user segment. Being able to modify a single Boost.Config repository to support WackyCompiler14 targeting LEG Core and get a bunch of libraries for the ride is also a plus. Emil and I have discussed these concerns and the best choice seems to be defaulting to zero dependencies and the option to pull in Boost.Config to provide configuration type facilities if needed. Accepting a library into Boost is as much about the library as it is bringing the author into a group. While Emil is already part of the author crowd, it was clear to me through his careful, informative, and patient responses that the addition of Boost.LEAF brings professional maturity that strengthens the community. A user with more than a year of experience with the library voted acceptance based on the “qualities of the library itself and the responsiveness and openness for feedback of its author.” Throughout the review period, Emil integrated feedback to both the documentation and code in another branch. It was wonderful to see the community provide input and the author respond so quickly to produce a better final result. Thank you Emil for your vision of better error handling and striving to bring us along! -- Michael Caisse Ciere Consulting ciere.com

3 2

[log] logfile not created where expected
by Barend Gehrels 08 Sep '20

08 Sep '20

hi, I encountered a bug, most probably, in Boost.Log It is reproduceable in the tutorial_logging.cpp : https://www.boost.org/doc/libs/1_74_0/libs/log/example/doc/tutorial_logging… The short code should create a file named "sample.log". But it does not, instead it creates a file called "00000.log", with the expected lines logged in that file. If I compile and run exactly the same tutorial in Boost 1.70, the file sample.log is indeed created with the same content. The problem is already there in Boost 1.73 What is the problem? Is there a workaround for it? Thanks, regards, Barend

3 2

Re: [boost] Review Manager needed for Boost.Lambda2
by Mateusz Loskot 08 Sep '20

08 Sep '20

Bjorn, thank you for reminding me about it. I've been offline for several weeks and I might have missed it happened already (lots of communication moved to Slack too). Mateusz Loskot, mateusz(a)loskot.net (Sent from mobile, may suffer from top-posting) On Tue, 8 Sep 2020, 20:39 Bjorn Reese, <breese(a)stofanet.dk> wrote: > On 2020-09-08 19:39, Mateusz Loskot via Boost wrote: > > On Tue, 8 Sep 2020 at 17:02, Peter Dimov via Boost > > <boost(a)lists.boost.org> wrote: > >> > >> I turned this into a mini-library, https://github.com/pdimov/lambda2, > and > >> I'd like to submit that to Boost. > >> > >> I'm looking for a review manager. > > > > FYI, added to the schedule. > > Strictly speaking, he needs an endorsement first. >

2 1

buffer_size behavior in boost::iostreams::filtering_istream
by Justin McManus 07 Sep '20

07 Sep '20

I have some code that works as intended, but it requires setting a buffer_size parameter to zero on a std::ifstream pushed onto a filtering chain, and I'd like to understand why, to ensure I'm not introducing a bug or a hack. I have essentially the following code: -------------------------------------------------------------------------------------------------------- std::ifstream m_jf("json_filename", std::ios_base::in | std::ios_base::binary); std::locale utf8_locale("en_US.UTF-8"); m_jf.imbue(utf8_locale); boost::iostreams::filtering_istream m_inbuf; m_inbuf.push(boost::iostreams::bzip2_decompressor()); m_inbuf.push(m_jf); std::string m_line; while (std::getline(m_inbuf, m_line)) { // Process the current line from the JSON file } -------------------------------------------------------------------------------------------------------- What I find is that the std::getline call will fail before the code has reached the EOF. It will always fail at the same line in a given JSON file, but it will fail on different lines in different JSON files. It's perfectly reproducible. However, if I change lines 4 and 5 to m_inbuf.push(boost::iostreams::bzip2_decompressor(), *0*); m_inbuf.push(m_jf, *0*); then the problem goes away. My question is, Why does setting the buffer_size parameter to zero solve the issue? What does this do, exactly? I saw the suggestion to set the buffer size this way from an old post in 2009, and it appears to work, but I'd like a deeper understanding of what's happening under the hood. If the buffer size is set to zero, what does the underlying implementation do, and how might this influence whether std::getline fails before the EOF? Thanks very much, Justin -- Justin McManus, Ph.D. Principal Scientist Lead Computational Biologist and Statistical Geneticist Kallyope, Inc. 430 East 29th Street, Suite 1050 New York, NY 10016

3 4

type detection macros + multiprecision + io
by Gero Peterhoff 07 Sep '20

07 Sep '20

Hello Developer, I noticed that some type detection macros and boost::float128_type or numeric_limits do not work reliably: - BOOST_HAS_FLOAT128 with clang and icc do not work, gcc is ok - BOOST_HAS_CHAR32/16_T, BOOST_NO_CHAR32/16_T are not defined, but either BOOST_HAS_TYPE or BOOST_NO_TYPE should be defined; see later - with clang there is no boost::float128_type and an error when including boost/multiprecision/float128.hpp (quadmath.h) - with clang and icc the numeric_limits<float128_t> are not available (gcc ok) see https://godbolt.org/z/95PfTG In general it would make sense: - convert the detection macros to a uniform naming scheme: always BOOST_HAS_TYPE_T (with _T at the end) or BOOST_HAS_TYPE (without _T), but not mixed/both - in multiprecision and math: 1) float: renaming typenames to eg floatBits_mbt (m: multiprecision, b: binary); floatBits_mdt (d: decimal) 2) integer: renaming typenames to e.g. u/intBits_mt; u/intBits_mct (c: checked) 3) based on this, renaming boost::float/int128_type to boost::float/int128_t so that it is always clear and there is no confusion. You can do that for all types, e.g. some DSP/microcontrollers do not support u/int8/64_t etc. I did that quickly for gcc (has_type.hpp) - of course that has to be in the corresponding boost/config/compiler/* and platform/* headers are implemented. Building on this, feature test macros can then be made available (features.hpp). Can you please implement the io-operators and string-conversions for u/int128 (in e.g. boost/math/cstdint/cstdint_iostream.hpp)? thx and regards Gero

5 4

[sync] Documentation ?
by Edward Diener 06 Sep '20

06 Sep '20

There is a 'sync' library in the Boost build tree as a submodule, but I see no documentation for it in Boost 1.74 itself. Is this currently a Boost library, or what is it ?

2 1

Cryptography for Boost. Boost.Crypto3.
by Mikhail Komarov 04 Sep '20

04 Sep '20

Hello! Some time ago I promised to show something about cryptography library architecture and implementation for Boost (https://lists.boost.org/Archives/boost//2020/02/248205.php <https://lists.boost.org/Archives/boost//2020/02/248205.php>), so here I am. I’m actually glad my two innocent letters triggered such a conversation. That is what I wanted to see! Please, read this letter very carefully, because some explaining needs to be done before you go to the code. By the way. The code: https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>. Warning. This letter contains mentioning of my non-commercial organizations I develop Boost.Crypto3 with. Don’t consider that as advertising. It is not supposed to be an advertisement. So, what cryptography for Boost (and probably STL some time after) should look like? 1. It should definitely be simple to use and implemented with some generic programming spirit. That is exactly what was done. Remember std::transform? Typical usage technique for all of the modules of a suite looks as follows. Classic iterator pair with output iterator: using namespace boost::crypto3; std::string input = “Weird German 2 byte thing: Ã.“, out; encode<codec::base64>(input.begin(), input.end(), std::inserter(out, out.end())); Classic iterator pair with output value return: using namespace boost::crypto3; std::string input = “00112233445566778899aabbccddeeff”; std::string key = “000102030405060708090a0b0c0d0e0f”; std::string out = encrypt<block::rijndael<128, 128>>(input.begin(), input.end(), key.begin(), key.end()); C++20 Ranges (or Boost.Range) usage: using namespace boost::crypto3; std::vector<std::uint8_t> input = {0x27, 0x0f, 0xb1, 0x89, 0x82, 0x80, 0x0d, 0xa6, 0x40}; std::string out = encode<codec::base32>(input); // or encode<codec::base32>(input, out); What about generic encoding/encryption/signing/hashing/%name_your_action% interface for classes? You can definitely do that in case you define implicit conversion of your class to integral type. I’m thinking of introducing Inspectable concept which will require for the developer to have a function in his class which would return every class member which developer thinks needs to be encrypted/hashes/encoded/etc, converted to the particular algorithm’s format. (e.g. satisfying Integral concept for ciphers/codes/hashes). 2. It should contain only time-proven schemes implemented with well-known and proven techniques. Yes. Boost.Crypto3 (https://github.com/NilFoundation/boost-crypto3 <https://github.com/NilFoundation/boost-crypto3>) is derived from =nil; Crypto3 C++ Cryptography Suite (https://github.com/NilFoundation/crypto3 <https://github.com/NilFoundation/crypto3>), which my non-commercial foundation designed and developed for it’s projects. A little bit of information on how exactly it was derived from the original suite you can get from the readme.md in both or repositories. As far as you can see in the original repository, the suite contains some very novel schemes (e.g. Verifiable Delay Functions: https://github.com/nilfoundaton/crypto3-vdf.git <https://github.com/nilfoundaton/crypto3-vdf.git>) and it is going to contain more (threshold cryptography, zero-knowlege cryptography, cryptographic accumulators etc.). But Boost-ified version has to be a solid one. No novel schemes. Only proven ones with proven implementation techniques. Which ones? This is what Boost community is up to decide. So, the question for the community. What schemes should be included? For now we only present small set of block ciphers, hashes and codecs listed in here https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__codec.html>, https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html <https://crypto3.nil.foundation/projects/crypto3/db/d97/group__hashes.html>, https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html <https://crypto3.nil.foundation/projects/crypto3/d2/dba/group__block.html>. (Or you can find the list in the include directories of particular repositories: https://github.com/nilfoundation/crypto3-block.git <https://github.com/nilfoundation/crypto3-block.git>, https://github.com/nilfoundation/crypto3-codec <https://github.com/nilfoundation/crypto3-codec.git>.git <https://github.com/nilfoundation/crypto3-codec.git>, https://github.com/nilfoundation/crypto3-hash <https://github.com/nilfoundation/crypto3-hash.git>.git <https://github.com/nilfoundation/crypto3-hash.git>). Peter (Dimov), I’m aware of your abandoned hash library implementation (https://github.com/pdimov/hash2 <https://github.com/pdimov/hash2>) based on Vinnie’s paper. I would really like to see it’s contents and concepts (probably) adapted to hash module of Boost.Crypto3. Would be a great synergy. 3. It should have a concept-based architecture. Concept-based architecture supposes every reused type trait to be formalized, and standardized. This is what exactly has been done. In particular, concepts for contents of Boost.Crypto3 are listed in sufficient documentation chapters: https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concep… <https://crypto3.nil.foundation/projects/crypto3/df/d5d/block_ciphers_concep…>, https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d7/da8/codec_concepts.html>, https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html <https://crypto3.nil.foundation/projects/crypto3/d1/d1f/hashes_concepts.html>. Long story short. We standardize stateless policies containing all the params as constexpr ones used for functions (interface-functions like encrypt/decrypt) and actual algorithms. We also figured out that most of cryptographic algorithms are about accumulation and only then processing. So we used Accumulator concept from Boost.Accumulator to accumulate (surprise surprise) data in a format, which is required by particular algorithm to be passed into. You can get the idea of how I’ve merged the Accumulator concept in from architecture and implementation notes (for Block Ciphers) in here: https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.h… <https://crypto3.nil.foundation/projects/crypto3/db/dee/block_ciphers_impl.h…>. Pretty drafty as well, but better than nothing. Yes, Hash concept should be made consent with currently existing Hash concept. This is in progress. Vinnie (Falco), I remember, you’ve mentioned some more generic concepts like OneWayFunction (in here: https://lists.boost.org/Archives/boost//2020/02/248228.php <https://lists.boost.org/Archives/boost//2020/02/248228.php>). I agree, we should do that, but we simply had no time to put this fine idea to our architecture. We would really appreciate any help with that. Cristopher (Kormanyos), I’m aware of your CRC catalog library (https://github.com/ckormanyos/crc-catalog <https://github.com/ckormanyos/crc-catalog>) I like it because of how in particular it is implemented. I would really like to merge it into our CRC implementation in Crypto3.Hash. By the way. I wonder if we could sometime remove separate Boost.CRC library and use CRC from more generic Boost.Crypto3? Hypothetically. Just asking. Yeah, we have checksums being implemented in Crypto3.Hash (and in Boost.Crypto3 as well), but they were simply not verified yet, so they are hanging in a PR (https://github.com/NilFoundation/crypto3-hash/pull/68 <https://github.com/NilFoundation/crypto3-hash/pull/68>). 4. Other modules? Roadmap? Yes, there will be other modules being proposed in some time. Public Key cryptography in Boost? Yes. Based on enhanced for cryptographic purposes Boost.Multiprecision (https://github.com/NilFoundation/crypto3-multiprecision <https://github.com/NilFoundation/crypto3-multiprecision>). SSL based on Boost.Crypto3? Yes. Why not. We are already in progress. Major updates are going to be performed with merging in the Boost-ified version of bunch of cross-dependent modules of the original suite. For example, public key cryptography would require enhancing Boost.Multiprecision (which is already in progress) and introducing finite fields library (https://github.com/NilFoundation/crypto3-algebra <https://github.com/NilFoundation/crypto3-algebra>). By the way. The original =nil; Crypto3 suite is highly modular. Every cryptography field is emplaced in separate submodule. There are more than 30 of them now in total (counting closed ones)(yeah, it is huge). Would you like to keep this modularity (like introducing Boost.BlockCiphers, Boost.Hashes and Boost.Codec instead of Boost.Crypto3)? Or do we need to keep everything in header-only Boost.Crypto3? I would vote for the second option. John (Maddock), do you understand now why I was attempting to get accepted those strange PRs in Boost.Multiprecision for the last year? Degski (Sorry, I’m not aware of your name), I remember you’ve mentioned some kind of compression library of yours. I would like to see the compression in the suite. Do you think I could fork it and fit into our architecture? 5. Why do we think we can implement properly design and cryptography? Ask me :). 6. Conclusion. In some time, this would definitely result in C++ Standardization Committee proposal. It is a long way, but I hope to bring it there. What we would like to hear from Boost community is some guidance about what in particular from this field would be useful for Boost, proposals and suggestions about architecture and concepts. Let us standardize some cryptography together (starting with this). Sincerely yours, Mikhail Komarov nemo(a)nil.foundation

6 8

Re: [boost] ASIO SSL Memory Leaks
by abrarov＠mail.ru 01 Sep '20

01 Sep '20

Hi John, Welcome to this mailing list! Doesn't Asio call SSL_shutdown here: https://github.com/chriskohlhoff/asio/blob/master/asio/include/asio/ssl/deta il/impl/engine.ipp#L325 ? Regarding memory leaks - cold you please provide a minimal example reproducing memory leak including the way you detect memory leak? Could chriskohlhoff/asio GitHub issue #368 (https://github.com/chriskohlhoff/asio/issues/368) be your case? Regards, Marat Abrarov.

1 0

ASIO SSL Memory Leaks
by John A 01 Sep '20

01 Sep '20

Hello y'all, This is my first post to this listserv though I've been on others for ~30 years. I have a Boost question, ob duh. Using the SSL_shutdown() outlined here: https://www.openssl.org/docs/man1.0.2/man3/SSL_shutdown.html from within Boost, there isn't an object to associate with this shutdown call. Secondly, we have memory leaks (~10k bytes) using the SSL library via Boost socket calls using the ASIO SSL streams. What am I missing? What advisement can be offered? Ideally, we want to call SSL_shutdown() from within Boost, though I am missing something. Best regards, John Aronetics.com We Speak ITR +1-216/307-5760

3 3