- Boost - lists.preview.boost.org

Boost.Parser Question about digit separator support
by Olaf Peter 04 Mar '24

04 Mar '24

Hi all, In the past I had some attempts with Spirit.x3. C++, VHDL and other languages support literal digit separators. On Stackoverflow.com there are some workarounds (in the context of VHDL's digit separator '_' ) for X3, which works but are unattractive and to unwieldy/unflexible. Is there a native/direct support for it at Boost.Parser? Thanks, Olaf

2 1

Reminder: Master branch closes for the 1.85.0 beta release on WEDNESDAY
by Marshall Clow 03 Mar '24

03 Mar '24

Calendar is here: https://www.boost.org/development/ — Marshall

1 0

Re: [boost] Reminder: Review for Zach's parsing library starts on Feb 19th
by Peter Dimov 02 Mar '24

02 Mar '24

Zach Laine wrote: > On Fri, Mar 1, 2024 at 2:15 AM Peter Dimov <pdimov(a)gmail.com> wrote: > > > > Zach Laine wrote: > > > > > I'm not sure what the tests reported earlier were doing, but I > > > > > definitely don't see orders of magnitude difference. > > > > > > > > I used this file: > > > > > > > > https://github.com/boostorg/json/blob/develop/bench/data/twitter.j > > > > son > > > > > > Using that file, I get similar results to what I already posted (the > > > numbers are different, but the ratios between them are the same). > > > > > > I managed to make a change that was much smaller -- no template > > > parameter required. I consistenty see 1.5x slowdown for Parser vs. > > > Boost.JSON for files around this size, and 2x or so for much larger > > > files (~25MB). > > > > That's probably because you are measuring I/O in addition to parsing. > > > > https://github.com/cmazakas/parser-review/blob/main/test/json.cpp > > > > only measures the parsing part. > > Quite right! After building it locally, I do see an advantage to the template > parameter approach -- that is, controlling the contents of scoped_trace_t with > an NTTP that is true/false when trace is enabled/disabled. scoped_trace_t is > specialized to be empty when the NTTP is false. I'll be committing the template > param-based approach soon. > > For those keeping track, with Christian's test framework, I get 35X worse > performance of the JSON parser example vs. Boost.JSON. This seems like a > reasonable factor to me. I wrote that example in a single, and made no > attempt to optimize it in any way; it is an example. Boost.JSON is a dedicated > JSON parser emphasizing efficiency. There's probably some room for Boost.Parser automatically applying certain optimizations to "naively" written parsers. For example, a lot of the time here is spent in skipping whitespace, using this rule: auto const ws_def = '\x09'_l | '\x0a' | '\x0d' | '\x20'; This generates an or_parser that contains four separate omit_parsers, each with a char_ inside. If I change it by hand to auto const ws_def = bp::omit[ bp::char_( "\x09\x0a\x0d\x20" ) ]; which I assume is equivalent, time improves from 365ms to 316ms (*). And that's still far from optimal, because (1) skipping whitespace can in principle be done with the equivalent of `find_first_not_of` and (2) the characters are a runtime value in the above, but something like `cxchar<'\0x09', '\0x0a', '\0x0d', '\x20'>` could be even faster. It should be possible for the library to automatically turn the "naive" ws_def into something more optimal, and then special-case `skip` for this common case. In the general case this is complicated by the existence of attributes and semantic actions; transforming parsers should take care to preserve those. But in this specific case none are present. (*) Under clang-cl. GCC -O3 has different views of the world and times there are 165ms and 181ms, respectively.

1 0

Late review of Boost.Parser
by David Sankel 02 Mar '24

02 Mar '24

# Are you knowledgeable about the problem domain? Yes. I've written several parser combinator libraries in C++ and used still more in C++ (Spirit v1, v2, and qi) and other programming langauges (Haskell's Parsec, attoparsec, and others). # How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I read through the tutorial documentation and some of the reference documentation. I also tried to create several examples with the library and made comparisons to Haskell's Parsec. # What is your evaluation of the potential usefulness of the library? Very # What is your evaluation of the documentation? The documentation isn't great. Consider char_ <https://tzlaine.github.io/parser/doc/html/boost/parser/char_.html>'s documentation. It includes no examples and fails to mention it can be used without arguments. It seems strange that the tutorial focuses so much on semantic actions. It seems like the more common case would be to parse into some kind of abstract syntax tree. # What is your evaluation of the design? Much of the wealth of research into parser combinators wasn't incorporated into this library. I didn't see in the documentation, for example, an easy way to "map" one parser's attribute into another. This kind of thing should be a basis operation provided by the library. # What is your evaluation of the implementation? The usage of has_include is a ODR nightmare for large codebases. It would be better to generate some kind of config.hpp that sets these definitions at compile time. For custom variant/tuple types, user specialization of variable templates is also an ODR nightmare. Again, a config.hpp would be better for this. For this, though, I don't think the configurability is worth the complexity. We have standard types for these so we should use them. # Did you try to use the library? With what compiler? Did you have any problems? Yes, I tried to use the library with a modern compiler on a MacOS machine. I didn't have any problems building. # Do you think the library should be accepted as a Boost library? No, I do not think it should be accepted. Overall, I think the syntax has too much focus on magically filling in user-provided structures instead of the basics of monadic parser combinators and basis operations. Consider the following Haskell parser which evaluates simple parenthesized sum expressions. I wasn't able to use Boost.Parser to accomplish this after reading the documentation and several attempts. I'm sure it's possible, but I don't see how it can be done using the combiners and primitives provided. import Text.Parsec.String (Parser)import Text.Parsec integer :: Parser Integerinteger = do n <- many1 digit return (read n) integerPlus :: Parser IntegerintegerPlus = do x <- integer y <- (try $ char '+' >> expression) <|> return 0 return $ x+y parentheses :: Parser Integerparentheses = do char '(' x <- expression char ')' return x expression :: Parser Integerexpression = integerPlus <|> parentheses

5 9

Boost Tools
by Robert Ramey 01 Mar '24

01 Mar '24

For some time I've felt that our Boost tooling could be improved. By boost tooling I mean things like b2, boost unit testing and integration testing, documentation generation, reports like dependencies, website, etc. Currently we rely on certain privileged authors to develop these things then then upload them to the boost directories. This has worked, but it could be better. We spend what seems to me an inordinate amount of time chasing down anomalies, researching errors, responding for requests for help, etc. I think design, robustness, documentation could be improved. I think that the the most natural way for us to do that would be to adopt the practices that we have used successfully for decades to develop superior quality libraries to the design and development of tooling. That is: a) Someone proposes a new tool or improvement in an existing one on this list. His proposal includes a working though likely incomplete prototype, documentation, tests, demo etc. b) This is subjected to the same formal review process that libraries are. c) Subject to this review, the proposal is either rejected, accepted or accepted with conditions. Of course this would only apply to new tools and updates of existing ones. Robert Ramey

1 0

The Review for Zach's parser library has concluded
by Marshall Clow 01 Mar '24

01 Mar '24

Thanks to everyone who participated, and especially to those to submitted reviews. (And Zach, of courser) I’ll have results soon. — Marshall

1 0

Boost.Parser parsing length-prefixed strings efficiently
by Mohammad Nejati [ashtum] 01 Mar '24

01 Mar '24

Hi everyone, I have two questions about the following code, which attempts to parse length-prefixed strings using the proposed Boost.Parser: ``` std::string input = "[5]HELLO[5]WORLD"; unsigned int size = 0; auto action = [&size](auto & ctx) { size = bp::_attr(ctx); }; auto lps_parser = '[' >> bp::uint_[action] >> ']' >> bp::repeat(std::ref(size))[bp::char_]; auto result = bp::parse(input, *lps_parser); ``` 1. Is it possible to eliminate the need for the action and size variable by using some kind of placeholder in bp::repeat that utilizes the attribute of the first parser? 2. How can we make this part of the parser more efficient: bp::repeat(std::ref(size))[bp::char_] The current implementation seems to loop and execute bp::char_ parser on each character, while all it needs to do is chunk a portion of the input string. Regards, Mohammad Nejati

2 1

New website is open-source!
by Vinnie Falco 29 Feb '24

29 Feb '24

The new website and documentation are provided under the BSL, and the source repositories have been transferred to the Boost GitHub organization: https://github.com/boostorg/website-v2 https://github.com/boostorg/website-v2-docs While we sort out the unresolved issues related to the licensing of artwork and logo, the new website has moved to this temporary location for now: https://boost.io Please continue to experience, enjoy, and most importantly test the new website. A gentle reminder: it is not “finished” and likely will not be “perfect” for a while. The launch of the new website is the beginning of the journey rather than the end. Issues opened here will be acted upon by our dedicated website development team: https://github.com/boostorg/website-v2/issues For documentation-related issues, please open and monitor issues here: https://github.com/boostorg/website-v2-docs/issues Pull requests are always welcomed! Thanks

6 6

Boost.Parser questions
by Ruben Perez 29 Feb '24

29 Feb '24

Hi all, Sorry for jumping into this so late. I'll send my review tomorrow (that's still on time, isn't it?). I've been playing with the library and some questions popped up (please note I've no prior experience with Spirit): * What's the rationale for using operator[] for semantic actions (instead of a regular method like .action())? It looks like the parsing problem with inline lambdas could be easily worked around with this. * What's the rationale for using operator[] for directives? repeat(24)[p] looks like an over-complication, where a regular function repeat(24, p) would have just worked fine. * For non-recursive rules, what's the rationale on preferring the macro over something like: constexpr bp::rule<struct replacement_field_tag, std::string> replacement_field { "replacement_field", +bp::char_ }; * My use case involves parsing identifiers that can only contain ASCII lowercase, uppercase, digits and the underscore. I'd usually do this by defining a `bool is_identifier_char(char c)` function. This is what I could do with Boost.Parser - is there a better way (one that doesn't force me to list all valid characters)? constexpr const char name_chars[] = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_0123456789"; constexpr auto name_continuation_rng = std::span<const char>( name_continuation, sizeof(name_continuation) - 1 ); * What's the rationale behind the special naming convention for functions like _attr, _value, _globals and the like? Is there a reason why these can't be member functions of the context? * I've found really frustrating the generic lambda approach for semantic actions, since this completely inhibits tooling like clangd. Is there a reason why we can't have a public definition of the parsing context I could use in my functions? * I started my evaluation using clang-18 (Linux) but builds fail (seems to happen under all clang versions I tried). gcc-12 seems to work fine. Note that I'm using the develop branch, since I understand it contains the bug fixes for the issues that previous reviewers found. Regards, Ruben.

6 9

Boost.Parser review
by Richard 29 Feb '24

29 Feb '24

- Do you think the library should be accepted as a Boost library? Yes, this would be a great addition to Boost as a successor to Spirit. ====================================================================== Details: Note: I've only had time to read through the documentation. I'm familar with Spirit Classic, Spirit V2 and X3 parsing libraries and have done presentations on them using my own written example code. - What is your evaluation of the design? It's very similar to Boost.Spirit in philosophy, but other than the (optional) connection to Boost.Hana feels very decoupled from all the other bits of boost at a user level. Spirit's direct connection to things like Phoenix and Fusion made Spirit more complicated to explain to other developers and more difficult to use IMO. I understand why such things were done at the time of Boost.Spirit's writing, but being a C++17 library frees Boost.Parser from all these digressions which is good. The only thing obviously lacking were the binary parsers from Spirit V2 (bug filed). - What is your evaluation of the implementation? I've only reviewed the documentation, so I haven't had any time to dig into the implementation before the review deadline. I will say that there are no comments in the header files and that could be improved. It means that without consulting the tutorial documentation, I have no information in the included headers to guide me. - What is your evaluation of the documentation? The documentation is well written and follows a logical progression. It could use a math expression example as this is one of the most common reasons for needing to write a parser (typical C style math operators and precedence rules). The JSON example is nice (shows recursive rule parsing), but doesn't deal with operator precedence at all, so another detailed example showing this would be helpful. There were some minor typos and mistakes; I filed github issues on those. The concepts and reference section are just raw code dumps. They could use some additional doxygen style comments to tie things together. - What is your evaluation of the potential usefulness of the library? I would find this library a great successor to Spirit. - Did you try to use the library? With what compiler? Did you have any problems? No, I only had time to read through documentation. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I carefully read through every page of the documentation in the Tutorial section and skimmed over the reference section. - Are you knowledgeable about the problem domain? Yes. -- "The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline> The Terminals Wiki <http://terminals-wiki.org> The Computer Graphics Museum <http://ComputerGraphicsMuseum.org> Legalize Adulthood! (my blog) <http://LegalizeAdulthood.wordpress.com>

3 2

Boost.Parser review
by Ruben Perez 29 Feb '24

29 Feb '24

Hi all, This is my review of the proposed Boost.Parser. As in other occasions, I've taken a practical approach and used the library to (re)implement an existing, real feature in Boost.MySQL: client-side SQL formatting. This requires parsing a format string with replacement fields (similar to std::format strings). - What is your evaluation of the potential usefulness of the library? I think it's useful. Parsing is a common task, and this library simplifies it. I tend to prefer hand-crafted parsers in general, but libraries like the proposed tend to save time (and bugs). Also, many other users have manifested interest in it. I think the library is too heavyweight regarding compile times to be used in header-only libraries like Boost.MySQL, but it should be fine in compiled libraries and applications. - What is your evaluation of the design? I think it's pretty good in general. It allows you to express complex parsers using little code, which is good. In my opinion, it may be too generic (allowing any forward character range as input, for instance), which tends to increase complexity. I tend to prefer simpler approaches like the one Boost.URL uses - but I understand not everyone thinks like me, and there's a use case for each approach. As we discussed in previous posts, if I was writing this, I would have made some decisions differently (see https://lists.boost.org/Archives/boost/2024/02/256010.php) but I don't think there is any problem with the current design. The author has justified his decisions, and everything makes sense. - What is your evaluation of the implementation? I can tell it's really complex. I've attempted to debug a problem I had for which parser::trace::on wasn't enough (see below), and navigating the code was extremely difficult (reminds me of debugging Asio). This is probably inherent to the level of complexity of the library. This is not a criticism, but a thought I wanted to share. - Did you try to use the library? With what compiler? Did you have any problems? Yes, I did. As mentioned above, I re-implemented Boost.MySQL SQL formatting (see https://www.boost.org/doc/libs/master/libs/mysql/doc/html/mysql/sql_formatt…) using the proposed library. I've uploaded the code to a branch - you can check it here: https://github.com/boostorg/mysql/blob/66c99e31c9e548b6cf5b98e3d08255f22390… For reference, I've also pasted the grammar and the code at the end of this email. I've found the following problems: * Rule parsers using sequence + optional parsers yield unexpected results when the optional rule doesn't match. Making it a regular parser, rather than a rule parser, fixes the problem. The impact is that automatic replacement fields (i.e. `{}`) were being parsed as `{0}`. https://github.com/tzlaine/parser/issues/125 tracks this. * My clang-11 build fails. Looks like a concept is being used without proper protection, and the compiler doesn't support it. https://github.com/tzlaine/parser/issues/129 * My clang-16 build (and my local build attempts with clang 17, 18 and 19) fail because of a non-matching template redeclaration in parser/detail/text/transcode_view.hpp. I don't know if it's clang being picky or a clang bug. https://github.com/tzlaine/parser/issues/130 * My clang-14 build fails. This looks like it's hitting a clang + libstdc++ bug. See https://github.com/tzlaine/parser/issues/131. * My Apple clang 14 build fails. It also looks like an issue in clang or libc++. See https://github.com/tzlaine/parser/issues/133. * My MSVC 14.2 and 14.3 builds under C++17 fail (C++20 works fine). I've completely failed to parse what's the problem here. Description here: https://github.com/tzlaine/parser/issues/132 Some other minor problems: * As reported by other reviewers, enabling warnings=all or warnings=extra issues a lot of warnings. * [[no_unique_address]] (C++20) is used unprotected, which causes warnings under clang-7 (see https://github.com/tzlaine/parser/issues/128) * The CMakeLists.txt is not Boost-compliant (yet). * There is a #include <boost/hana.hpp>, which is not very friendly for compile-times - better replace it by the individual header that is needed. I've submitted the build logs together with all details I've been able to gather. I haven't been able to provide a minimal reproducible example for some of these (they were a lot and I had limited time). Note that all the builds referenced in the issues (except the OSX one) are run in Docker containers. If you have access to Docker, reproducing these environments should be trivial. Since I've done this in the past, should the author need any help with this, please let me know, I'm happy to help. Reading the CI files for the proposed library, I think compiler coverage is insufficient - for instance, there are no Linux clang builds, and the only Apple clang build uses clang 13 and C++17. - What is your evaluation of the documentation? The presentation is really good. I struggled to understand the concepts at first because I had no previous experience with parsers. I ended up reading the Bison concepts section (https://www.gnu.org/software/bison/manual/bison.html#Concepts) after which everything made perfect sense. I'd advise to expand the introduction a bit, or just point to other material for people like me. I fail to understand what RESOLVE means in https://tzlaine.github.io/parser/doc/html/boost_parser__proposed_/tutorial/…. Docs state "RESOLVE() is a notional macro that expands to the resolution of parse argument or evaluation of a parse predicate", but I'd say this resolution hasn't been explained yet - a link here would help. I've found the reference to be more confusing that helpful - it looks like a straight header dump. This is made worse by the concepts not being linked. Although it has many terrible things, the quickbook toolchain is (partly) capable of linking concepts - have a look at https://www.boost.org/doc/libs/master/libs/mysql/doc/html/mysql/ref/boost__…. This still has limitations, but I'd say it's better than what's currently there. If this is something you'd be interested in, please let me know and I will help you with it. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I've spent around 2 full days of work reading about the problem domain, studying the docs, writing my use case, diagnosing the problems I reported and writing this review. - Are you knowledgeable about the problem domain? I've never used a parsing library like this before. I've read about the problem in the past, but wasn't very knowledgeable about it before the review. I've written small parsers like the one for format strings in the past, though. - My conclusion My recommendation is to REJECT the library in its current form. This is not a hard rejection - I completely think this library can be useful to people. But I found too many problems while using it - too many to get it into Boost in its current form. I'd be happy to see this library being proposed again in a couple of months, once all the issues found during the review get fixed. - Affiliation disclosure Please note that I'm affiliated with the C++ Alliance. I'd like to thank Zach for proposing the library, and Marshall for managing the review. Regards, Ruben. - Reference: parsing code in Boost.MySQL /** * format-str = *( text | literal-opening-brace | literal-closing-brace | replacement-field ) * text = *( any characters except {} ) * replacement-field = "{" [ replacement-id ] "}" * replacement-id = ushort | field-name * field-name = (character a-z, A-Z, _) *( any characters 0-9, a-z, A-Z, _ ) */ // Note: globals is a format_state structure, which has member functions // to append raw text and replacement fields to the output string // Actions constexpr auto action_text = [](auto& ctx) { _globals(ctx).append_text(_attr(ctx)); }; constexpr auto action_replacement_field = [](auto& ctx) { const std::optional<std::variant<unsigned short, std::string>>& val = _attr(ctx); format_state& st = _globals(ctx); if (val) { if (const auto* idx = std::get_if<unsigned short>(&*val)) { st.append_indexed_field(*idx); } else { st.append_named_field(std::get<std::string>(*val)); } } else { st.append_auto_field(); } }; constexpr auto action_opening_brace = [](auto& ctx) { _globals(ctx).append_text("{"); }; constexpr auto action_closing_brace = [](auto& ctx) { _globals(ctx).append_text("}"); }; // Character ranges constexpr std::array<const char, 63> name_chars{ 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '_', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9' }; constexpr boost::span<const char, 53> name_start_chars{name_chars.data(), 53}; // Parsers and rules constexpr auto field_name = parser::char_(name_start_chars) >> *(parser::char_(name_chars)); constexpr auto replacement_id = parser::ushort_ | field_name; constexpr auto replacement_field = ('{' >> -replacement_id) > '}'; constexpr auto literal_open_brace = parser::lit("{{"); constexpr auto literal_close_brace = parser::lit("}}"); constexpr auto text = +(parser::char_ - '{' - '}'); constexpr auto format_str_parser = *( text[action_text] | literal_open_brace[action_opening_brace] | literal_close_brace[action_closing_brace] | replacement_field[action_replacement_field] ); // Top-level parsing function inline void parse_format_str(string_view format_str, format_context_base& ctx, span<const format_arg> args) { // By default, parser prints to cerr, which we don't want parser::callback_error_handler handler([](const std::string&) {}); // Create a parsing state format_state st(ctx, args); // Invoke the actual parsing bool result = parser::parse( format_str, parser::with_globals(parser::with_error_handler(format_str_parser, handler), st) ); // If there was an error, report it if (!result) { ctx.add_error(client_errc::format_string_invalid_syntax); } }

2 1

[release] The 1.85.0 release will close for major changes TOMORROW
by Marshall Clow 28 Feb '24

28 Feb '24

Calendar here: https://www.boost.org/development/ Next deadline: The master branch will close for the beta next Wednesday. — The release managers

1 0

Boost.Parser review by Tobias
by tobi＠die-loews.de 27 Feb '24

27 Feb '24

Boost.Parser review Why do we need a DSEL-Parser library in Boost: I think nobody, who once has written a parser with a DSEL (like e.g. Spirit) in C++, ever want's to go back to out off source parser-generators like yacc/bison or ANTLR (or even hand written parsers): it is so convenient having the parser definition as a grammar within the code. Additionally, you can quite easily hook into the parser (e.g. Spirit's Phoenix functions). So, IMHO there is definitely need for such C++ libraries. On the other hand, the task of creating such a library is very challenging and requires high expertise and coding standards, as library-users have to rely on its correctness. So, Boost with its standards and peer-review process is definitely the place for such a library. (Personally, I wouldn't use a parser-library the hasn't gone through such a process.) Should accept Boost.Parser into Boost? There are already three parser libraries in Boost: Spirt and its successors Qi and X3 - Spirit: very outdated interface, IIRC C++98 compatible, superseded by Qi - Qi: Simply a revolution in parsing! It was blowing my mind, when I first used it. The way attributes were composed and semantic actions could applied with Phoenix was totally exciting. But of course, there was also the downside of (very) long compilation-times and frustrating long error-messages when something got wrong. And (again IIRC) Qi came to life before C++11, so it is also C++98. - X3: I only wrote one larger X3 parser, and to be frank, IMHO it never made it to production-level. There are several things that worked perfectly with Qi, which I never got running the same way in X3. Additionally, the Spirit libraries seem no longer to be in active development. Now, let's get to Boost.Parser: First, I don't want to go into the details of the library, as I think other reviewers did a very good job there. I've been following its evolving since I heard of it in 2022. Back then, I did some test with it and reported some Issues, which got resolved. During the last two years many additional features got added to it. So, Boost.Parser is definitely under active development. Furthermore, the author is an expert for Unicode, which is reflected in the built-in Unicode support of the library. It is written in modern C++ (requires C++17). It is compile-time friendly, supports std::ranges and concepts. It offers an option/abstraction to choose between std::tuples, if needed, or the more flexible hana-implementation. It's syntax is close to Qi, so transition for Qi-users should not be too hard. The documentation contains a lot of examples that help to understand how the library works and how it can be used. Finally, I really like the very easy to use trace-mode. Again, should accept Boost.Parser into Boost? Definitely Yes! Tobias

3 2

Boost.Parser Review
by Phil Endecott 26 Feb '24

26 Feb '24

Boost.Parser Review =================== I've been using Spirit for almost 20 years. It has always been an amazing example of pushing the limits of what is possible in C++, even by the standards of Boost. So it has always left me thinking that we just need to wait a few more years until the language has the few more features needed to smooth over its remaining awkwardness. Now a few years have passed and we have, for example, lambdas, auto, constexpr, concepts. So I was excited to see what improvements Zach's proposed library has over Spirit. For example, when Metaparse was reviewed in 2015, compile-time parsing of BNF was discussed. Perhaps we should be able to write actual BNF in compile-time strings now, rather than re-use C++ operators? And now that we have constexpr, should we be able to convert the production rules to tables at compile time and do proper linear-complexity LALR parsing, rather than top-down? We do now have compile-time regular expression libraries than can do something like this. Initial Reaction ---------------- My first reaction when I looked at the docs was: - I'd like to congratulate Zach on choosing a sensible descriptive name for the library. - The rule syntax looks very much like Spirit. The docs look like the Spirit docs. It's more of an evolution than a revolution. - There's an ugly MACRO to define rules! That doesn't look very 2024 to me. Test Case --------- I decided to try to translate an exiting parser from Spirit v2 (qi) to the new library. It's a parser for latitude-longitude coordinates entered by a human, i.e. it is supposed to try to recognise a range of possible formats. Examples: 12.34 N, 56.78 E 56.78,-12.34 12d 30' n 45d 15' 7" w 12 30 45 N, 45 15 7 W 12d 30m 15s S 50d 59m 59s W There are some subtle issues with this grammar. Note for example that 's' could mean seconds or South. One advantage of Bison is that it gives compile-time diagnostics reporting some of these. Neither Spirit nor the new library can do this. The core of the Spirit code is as follows: (I will link to a .zip with the complete code at the end; I bet the mailing list messes up the long lines.) ///////////////////////////////////// degrees_symbol = lit(UTF8_DEGREES) | no_case[ lit("degrees") | lit("deg") | lit('d') ]; minutes_symbol = lit(UTF8_PRIME) | no_case[ lit('\'') | lit("minutes") | lit("min") | lit('m') ]; seconds_symbol = lit(UTF8_DOUBLE_PRIME) | no_case[ lit('"') | lit("seconds") | lit("sec") | lit('s') ]; uint_0_60 %= uint_ [ _pass = _1 < 60U ]; double_0_60 %= basic_udouble [ _pass = _1 < 60 ]; decimal_degrees %= basic_udouble >> -degrees_symbol; degrees_decimal_minutes = (uint_ >> -degrees_symbol >> double_0_60 >> -minutes_symbol) [ _val = _1 + _2/60.0 ]; degrees_minutes_seconds = (uint_ >> -degrees_symbol >> uint_0_60 >> -minutes_symbol >> double_0_60 >> -seconds_symbol) [ _val = _1 + _2/60.0 + _3/3600.0 ]; degrees %= degrees_minutes_seconds // Longest first. | degrees_decimal_minutes | decimal_degrees; northsouth %= no_case[ char_("ns") ]; eastwest %= no_case[ char_("ew") ]; latitude = (degrees >> northsouth) [( _pass = _1 <= 90, _val = if_else(_2=='S' || _2=='s', -_1, _1) )]; longitude = (degrees >> eastwest) [( _pass = _1 <= 180, _val = if_else(_2=='W' || _2=='w', -_1, _1) )]; signed_degrees %= basic_double >> -degrees_symbol; signed_latitude %= signed_degrees [ _pass = -90 <= _1 && _1 <= 90 ]; signed_longitude %= signed_degrees [ _pass = -180 <= _1 && _1 <= 180 ]; latlon = ((latitude >> longitude) [ _val = construct<g2d::Vector>(_2,_1) ]) | ((longitude >> latitude) [ _val = construct<g2d::Vector>(_1,_2) ]) | ((signed_longitude >> signed_latitude) [ _val = construct<g2d::Vector>(_1,_2) ]); ///////////////////////////////////// The complete Boost.Parser code is as follows: ///////////////////////////////////// #include "boost/parser/parser.hpp" #include <cassert> #include <iostream> #include <optional> #include <string> namespace bp = boost::parser; using namespace boost::hana::literals; namespace g2d { struct Vector { double x, y; }; }; bp::rule<struct uint_0_60, unsigned int> uint_0_60 = "uint_0_60"; bp::rule<struct double_0_60, double> double_0_60 = "double_0_60"; bp::rule<struct degrees_decimal_minutes, double> degrees_decimal_minutes = "degrees_decimal_minutes"; bp::rule<struct degrees_minutes_seconds, double> degrees_minutes_seconds = "degrees_minutes_seconds"; bp::rule<struct degrees, double> degrees = "degrees"; bp::rule<struct latitude, double> latitude = "latitude"; bp::rule<struct longitude, double> longitude = "longitude"; bp::rule<struct signed_latitude, double> signed_latitude = "signed_latitude"; bp::rule<struct signed_longitude, double> signed_longitude = "signed_longitude"; bp::rule<struct latlon, g2d::Vector> latlon = "latlon"; //const auto degrees_symbol = bp::no_case[ bp::lit("degrees") | bp::lit("deg") | bp::lit('d') ]; //const auto minutes_symbol = bp::no_case[ bp::lit('\'') | bp::lit("minutes") | bp::lit("min") | bp::lit('m') ]; //const auto seconds_symbol = bp::no_case[ bp::lit('"') | bp::lit("seconds") | bp::lit("sec") | bp::lit('s') ]; // FIXME those rules don't work and I don't know why. They work if the symbol is // the first alternative; it it's any of the other alternatives the double returned // by e.g. decimal_degrees below is always zero. // Use these instead: const auto degrees_symbol = bp::lit('d'); const auto minutes_symbol = bp::lit('m'); const auto seconds_symbol = bp::lit('s'); // Beware, 's' can also mean south. const auto uint_0_60_def = bp::uint_ [( [](auto ctx) { _pass(ctx) = _attr(ctx) < 60U; _val(ctx) = _attr(ctx); } )]; const auto double_0_60_def = bp::double_ [( [](auto ctx) { _pass(ctx) = _attr(ctx) < 60; _val(ctx) = _attr(ctx); } )]; const auto decimal_degrees = bp::double_ >> -degrees_symbol; const auto degrees_decimal_minutes_def = (bp::uint_ >> -degrees_symbol >> double_0_60 >> -minutes_symbol) [( [](auto ctx) { auto d = _attr(ctx)[0_c]; auto m = _attr(ctx)[1_c]; _val(ctx) = d + m/60.0; } )]; // FIXME this rule matches e.g. 150.5 as 150 .5; we want that input to // be matched by the decimal degrees rule as 150.5. In the qi code, which // has the same fundamental problem, it is avoided because the double_ // parser is configured to reject input with a leading decimal point. // It would be more correct to require that either a degrees symbol or // whitepace or both between the degrees and minutes numbers. That's // a bit complicated because the whitespace is handled by the skipper. // In a bottom-up parser I think the ambiguity is resolved by putting // decimal_degrees first in degrees_def below. That doesn't work for a // top-down parser. // FIXME for similar reasons, "150d 0.5m n" fails to parse, because // the degrees_minutes_seconds rule matches 150,0,.5 and then the next // rule up fails to parse the 'm'. const auto degrees_minutes_seconds_def = (bp::uint_ >> -degrees_symbol >> uint_0_60 >> -minutes_symbol >> double_0_60 >> -seconds_symbol) [( [](auto ctx) { auto d = _attr(ctx)[0_c]; auto m = _attr(ctx)[1_c]; auto s = _attr(ctx)[2_c]; _val(ctx) = d + m/60.0 + s/3600.0; } )]; const auto degrees_def = degrees_minutes_seconds | degrees_decimal_minutes | decimal_degrees; const auto northsouth = bp::no_case[ bp::char_("ns") ]; const auto eastwest = bp::no_case[ bp::char_("ew") ]; const auto latitude_def = (degrees >> northsouth) [( [](auto ctx) { auto d = _attr(ctx)[0_c]; auto ns = _attr(ctx)[1_c]; _pass(ctx) = d <= 90; _val(ctx) = ns=='S' || ns=='s' ? -d : d; } )]; const auto longitude_def = (degrees >> eastwest) [( [](auto ctx) { auto d = _attr(ctx)[0_c]; auto ew = _attr(ctx)[1_c]; _pass(ctx) = d <= 180; _val(ctx) = ew=='W' || ew=='w' ? -d : d; } )]; const auto signed_degrees = bp::double_ >> -degrees_symbol; const auto signed_latitude_def = signed_degrees [( [](auto ctx) { auto d = _attr(ctx); _pass(ctx) = -90 <= d && d <= 90; } )]; const auto signed_longitude_def = signed_degrees [( [](auto ctx) { auto d = _attr(ctx); _pass(ctx) = -180 <= d && d <= 180; } )]; const auto latlon_def = ((latitude >> longitude) [( [](auto ctx) { _val(ctx) = g2d::Vector{_attr(ctx)[1_c], _attr(ctx)[0_c]}; } )] ) | ((longitude >> latitude) [( [](auto ctx) { _val(ctx) = g2d::Vector{_attr(ctx)[0_c], _attr(ctx)[1_c]}; } )] ) | ((signed_longitude >> signed_latitude) [( [](auto ctx) { _val(ctx) = g2d::Vector{_attr(ctx)[0_c], _attr(ctx)[1_c]}; } )] ); BOOST_PARSER_DEFINE_RULES(uint_0_60); BOOST_PARSER_DEFINE_RULES(double_0_60); BOOST_PARSER_DEFINE_RULES(degrees_decimal_minutes); BOOST_PARSER_DEFINE_RULES(degrees_minutes_seconds); BOOST_PARSER_DEFINE_RULES(degrees); BOOST_PARSER_DEFINE_RULES(latitude); BOOST_PARSER_DEFINE_RULES(longitude); BOOST_PARSER_DEFINE_RULES(signed_latitude); BOOST_PARSER_DEFINE_RULES(signed_longitude); BOOST_PARSER_DEFINE_RULES(latlon); static std::optional<g2d::Vector> try_parse_latlon(std::string_view s) { auto input = latlon >> bp::eoi; return parse(s,input, bp::ws|bp::lit(',')); } int main(int argc, const char* argv[]) { assert(argc==2); auto opt_coords = try_parse_latlon(argv[1]); if (!opt_coords) { std::cerr << "parse error\n"; return 1; } else { std::cout << opt_coords->x << " " << opt_coords->y << "\n"; return 0; } } ///////////////////////////////////// That Ugly Macro --------------- bp::rule<struct latitude, double> latitude = "latitude"; const auto latitude_def = (degrees >> northsouth) .... ; BOOST_PARSER_DEFINE_RULES(latitude); Is a macro really the best way to do this? Spirit seems to manage without. Despite having a macro this is still very verbose. I've had to write the name of the rule FIVE times in three lines. Once there is a macro you might as well put everything inside it: BOOST_PARSER_DEFINE_RULE(latitude, double, (degrees >> northsouth) ....); Of course if you need to declare the rules before defining them due to recursion or other forward references you will need separate macros for declaration and definition. These rule declarations seem to need to be at global scope (maybe namespace scope?). Is that fundamental? Can't I put them in a class or a function? Semantic Actions ---------------- In Bison I can write semantic actions like this: { $$ = $1 + $2/60.0; } After expanding the $ variables that gets copied verbatim into the bison output, so it can include any code. Spirit clearly wanted to replicate that sort of functionality, but it pre-dates lambdas, so it includes a lot of amazing functionality that allows me to write semantic actions like: [ _val = _1 + _2/60.0 ] This is nice and concise but its limitation is of course that I cannot write arbitrary code in there. (BTW, are we strictly allowed to use identifiers starting with _ like _val?) What should this look like now that we have lambdas in the language? Of course it's going to be a bit more verbose because of the syntax overhead, but I was hoping that it would be something like this: [ [](auto deg, auto min) { return deg + min/60.0; } ] (Aside: That doesn't actually work because [[ can only appear in attributes even if there is a space, so it needs to be written [( [] ... )]. I did wonder if changing from operator[] to operator() might make sense?) That's longer than before, but I have given meaningful names to the parameters (which I can also do in Bison BTW) and I can include any code. But that's not the syntax that the current proposal needs. It needs this: [( [](auto ctx) { _val(ctx) = _attr(ctx)[0_c] + _attr(ctx)[1_c]; } )] Frankly I think that combines the worst of all the other options and adds some extra new verbosity of its own. One other advantage of having the lambda return a result could be improving how the semantic type (aka attribute type) of a rule is set. For example, if I have the rule degrees >> minutes where degrees and minutes return doubles, the semantic type of the combined rule is something like tuple<double,double>. But I'm going to add a semantic action that adds the two values as shown above, returning one double. If the return value of the semantic action's lambda were used as the semantic type of the rule then we could write: auto rule = (degrees >> minutes) [( [](auto d, auto m) { return d + m/60.0; } )]; and rule would return a double to its parent rule. As it is, the semantic action doesn't change the type and we have to explicitly and verbosely declare the rule as returning a double. Of course the context includes more than just the child rule semantic values and a slot for the result; for example it includes the _pass boolean that I use to clamp the range of values in my example: [( [](auto ctx) { _pass(ctx) = _attr(ctx)[0_c] <= 90; } )]; I suggest that either a special lambda signature is used to indicate that it is a predicate, or an additional "directive" e.g. predicate[ ... ] is added. Compilation Times and Errors ---------------------------- Compilation times for my example were: Boost.Parser: 28.6 s Boost.Spirit: 30.1 s but the grammar is not quite identical, so the small difference is not significant. Code size is somewhat larger for Parser than for Spirit. (g++ 12.2.0, -O3, Linux, aarch64.) I did not particularly notice improved error messages compared to Spirit, though they weren't worse. Consider for example this code from my example: return parse(s, input, bp::ws|bp::lit(',')); Originally I had this: auto skipper = bp::ws | bp::lit(','); return parse(s, input, skipper); But that failed to compile, and I got 15 screenfuls of inpenetrable error messages. I fixed it more-or-less by randomly changing things until it worked. If I now change it back I get only two screenfuls that accurately say that I can't return a bool from a function that's supposed to return an optional, but I still don't know why the return type of parse() changes when I move the declaration of the skipper. So I fear that the idea that compile times and error verbosity are improved compared to Spirit are not borne out. Bugs ---- I found a couple of bugs while writing this review which Zach has resolved. I suspect that this code has not had enough varied use to flush out all the issues. Other Issues ------------ The double_ parser is not configurable in the way that the Spirit double_ parser is. This is problematic; in my example 3e6n means "3 east 6 north", not "3000000 north", so I need a double_ parser that doesn't recognise exponents. I'd also like to ignore values with leading .s, i.e. 10.1 should not match int_ >> double_ as "10 0.1", and I really don't want a hacker to sneak a nan or inf into my code if it recognises those. The docs are not bad and are reminiscent of the Spirit docs. Personally I would prefer to see an example early on that takes a chunk of real BNF and translates it into a parser. I had issues with Apple Clang (15.0.0); it would not compile "#include <boost/parser.hpp>" until I turned off concepts. Has anyone tried with other Clangs? If this is not fixable maybe the library should detect this and use non-concepts mode automatically. Compiling with -Wall resulted in lots of unused parameter warnings. These can be fixed by commenting-out the parameter name, leaving just the type. Boost libraries should compile without warnings, some people use -Werror. Conclusion ---------- I was hoping to find a library that was an improvement on Spirit, but I regret to say that that's not what I've found here. The only notable improvement compared to Spirit.Qi that I've found is that no_case[] is propogated to nested rules, which isn't the case with Qi. Rules themselves are essentially the same as Spirit. The method for defining rules is more verbose and now needs an ugly macro. Semantic actions are less readable. It is possible that there are significant advantages compared to Spirit that I have missed, as a result of having used working Spirit code as my starting point, but I have read the doc's comparison with Spirit page. My conclusion is that this library should not be accepted. If others believe it should be accepted then I feel that should be conditional on many of the issues I've mentioned above being resolved, e.g. unused parameter warnings, configuration of the double_ parser, etc. File containing example: https://chezphil.org/tmp/parser_review.zip Regards, Phil.

4 4

Doxygen+Quickbook and C++20 concepts
by Zach Laine 26 Feb '24

26 Feb '24

I've seen concepts just vanish from Doxygen+Quickbook docs in the past. I thought this was just a limitation. However, looking at the Quickbook docs, I see that there's now a conceptref reference link. So, I put this very simple concept in a header (parser.hpp) to test things: /** \concept non_void */ template<typename T> concept non_void = (!std::same_as<T, void>); Doxygen appears to process it ok, or at least it says in its output that it does. Then I see non_void show up in these files: bin/parser_reference-xml.doxygen bin/parser_reference-xml/index.xml bin/parser_reference-xml/parser_8hpp.xml (I put the concept in this header, parser.hpp) bin/parser_reference-xml/conceptboost_1_1parser_1_1non__void.xml However, when I put [conceptref non_void Non-void.] in a .qbk file, all that shows up in the rendered HTML is "Non-void." (it's not a clickable link). Also, the no_void concept is not in the Doxygen reference for that header. Most frustratingly, concept constraints disappear from constrained templates in the Doxygen reference docs. Is there some special sauce that makes Quickbook+Doxygen process concepts? Is it doing it, but I'm not looking in the right place? Zach

1 0

Review of Parser library ends Wednesday, February 28th.
by Marshall Clow 25 Feb '24

25 Feb '24

So far we’ve received five reviews, and one of them was “docs only”. Keep those reviews coming! — Marshall

1 0

Boost.Parser review by Andrzej
by Andrzej Krzemienski 24 Feb '24

24 Feb '24

Hi, This is another review of Boost.Parser. Usefulness If Boost.Parser is a maintained version of Boost.Spirit that is already a worth goal. If Boost.Parser is a better documented version of Boost.Spirit that is already a gain. Note that the docs for Boost.Spirit X3 are hidden, and I was not aware of its existence until the review of Boost.Parser. I hear that the selling point of this library is Unicode support "through and through". I cannot confirm that it works as advertised, nor benefit from this guarantee, as all my use cases for a parser so far only dealt with the ASCII set. Design Boost.Spirit has a great design, and because Boost.Parser adopts the same design, it must be also good. I would intuitively expect that C++17 offers sufficient number of new features that the design for a parser library would be improved over Boost.Spirits. For instance, I am surprised that we still need to use a macro. Not that I can offer a better alternative. I do not know much about the implementation difficulties. so maybe the macro is the best choice. Wouldn't template explicit specializations do the job? The fact that this library automatically adapts aggregate types as attributes in a Boost.PFR way is really cool. I am convinced that compiler error messages could be better. This can be done by using a combination of static_assert and implementation details SFINAEd away based on the same condition as in static_assert. See https://github.com/tzlaine/parser/issues/56 I am unhappy about the fact that as simple task as parsing a JSON-like array of ints `[1, 2, 3]` into a vector requires me to define a rule. Spirit introduced operator% that doesn't work for empty lists, but doe to its clever attribute conversions (at least in Spirit2), I could define: auto array = '[' > ((int_ % ',') | eps) > ']'; And it would deduce the return attribute alright. But because Boost.Parser wants to be more strict about clever attribute conversions this option is gone, and no alternative is provided other than to declare a rule, which I never needed with Boost.Spirit. See https://github.com/tzlaine/parser/issues/67 I am not comfortable that I have to use the macro BOOST_PARSER_DEFINE_RULES. I would expect that template specializations should give a similar effect. But I do not know how to make it better. But if I have to use a macro, maybe it could offer more, like checking if the rule is compatible with its definition (for earlier bug reporting). I am missing a tool that would give me a compile-time answer * can I pass this attribute to that parser when parsing this type of character? * will my rule with its parser compile for a given input character type? * will my rule with its parser compile for at least one input character type? Documentation To me, the tutorial and overview in the documentation were sufficient to understand the library. But this is because I had learned Boost.Spirit before. If this were my first exposure to "a parser implementation via expression templates", I would be overwhelmed. Even though a lot of effort went into preparing a soft introduction to the subject, the subject is so complicated that it requires a super-gentle introduction. Some things remain underdocumented, like the interface of context. See https://github.com/tzlaine/parser/issues/104 After having stared at the docs for more than a month, I still have trouble finding the table with all the parsers listed. The reference section of documentation is missing details, such as what functions throw, what they expect of their arguments, what they guarantee, what is the exception safety level. For instance, see https://github.com/tzlaine/parser/issues/103 Also, the layout of the reference section looks like it is driven by an auto-generation tool run on the header files. It is not very user friendly. for instance, trying to find the specification of function parse() in https://tzlaine.github.io/parser/doc/html/header/boost/parser/parser_hpp.ht… isn't easy: neither visually (the amount of content is overwhelming) nor with grep-like tool (a lot of things there have "parse" in the name). What is good for header organization is not necessarily good for documentation organization. I am uncomfortable with the responses from the author that the reference section is missing parts, such as concepts, because the documentation toolchain does not support them. This would look like a bad choice of the tool. On the other hand, I do not know of any satisfactory tool for good C++ documentation. My testing I spent like a week trying to compile my parser for a text-based game engine, which was previously compiled with Boost.Spirit v2. Here's a sample file to parse (in Polish): https://raw.githubusercontent.com/akrzemi1/wdungeon/master/wdungeon.plot Here's the original Spirit V2 parser: https://raw.githubusercontent.com/akrzemi1/wdungeon/master/parser.cpp The new parser in BoostParser is included below. I didn't have to use rules in the original version. I am forced to use them in Boost.Spirit. I was not interested in performance testing as my use case is to read the config file upon application startup, so I can afford to wait. I tried the library with the newest versions of MSVC, GCC and Clang. It works in all of them. I filed a number of bug reports, to which Zach has been responding extremely quickly. Thank you! I also appreciate the negative responses, as they indicate that the author has a clear and coherent vision of what he wants the library to be. I recommend accepting the library into Boost. I would also like to thank Zach for writing and sharing this library! Regards, &rzej; ``` namespace parser { using namespace plot2; const auto identifier = bp::lexeme[+(bp::digit | bp::lower | bp::upper | bp::char_('_'))]; const bp::rule<struct cond_tag, std::vector<Variable>> cond = "cond"; const bp::rule<struct set_vars_tag, std::vector<Variable>> set_vars = "set_vars"; const bp::rule<struct choice_tag, Choice> choice = "choice"; const bp::rule<struct continuation_tag, Alternative> continuation = "continuation"; const bp::rule<struct map_tag, std::vector<MapStep>> map = "map"; const bp::rule<struct label_tag, std::string> label = "label"; const bp::rule<struct text_tag, std::string> text = "text"; const bp::rule<struct transition_tag, Transition> transition = "transition"; const bp::rule<struct redirection_tag, Redirection> redirection = "redirection"; const bp::rule<struct comment_tag> comment = "comment"; const bp::rule<struct plot_tag, GamePlot> plot = "plot"; const auto cond_def = ("[" >> ((identifier >> "=" > bp::int_) % ",") > "]") | bp::attr(std::vector<Variable>{}); const auto set_vars_def = ("[" >> ((identifier >> ":=" > bp::int_) % ",") > "]") | bp::attr(std::vector<Variable>{}); auto end_tag = bp::lit("[") >> bp::char_("+!") >> bp::lit("]"); auto label_def = bp::lit("[") >> identifier >> bp::lit("]"); auto text_def = bp::lexeme[+(bp::char_ - ']' - '[')]; auto map_step = bp::lit("^") >> -bp::char_("NESW") >> -(bp::lit(".") >> (+bp::char_("STCLPRXHUVD#"))); auto map_def = bp::lit("[") >> +map_step >> bp::lit("]"); auto redirection_def = +(bp::lit("[->]") >> cond >> (-map) >> label >> set_vars); auto option = bp::lit("[#]") >> cond >> text >> (-map) >> label >> (-text) >> set_vars; auto choice_def = bp::lit("[?]") >> text >> (+option); auto continuation_def = redirection | end_tag | choice; auto transition_def = bp::lit("[##]") > label >> -map >> -text > continuation; auto comment_def = bp::lit("[//]") >> bp::omit[*(bp::char_ - ']' - '[')]; auto plot_def = +transition; BOOST_PARSER_DEFINE_RULES(cond, set_vars, choice, continuation, map, label, text, transition, redirection, comment, plot); } ```

2 3

Parsers vs Unicode
by Arno Schoedl 24 Feb '24

24 Feb '24

Since the reviews of Boost.Parser are currently on, I wanted to share an insight we had at think-cell when working with Boost.Spirit. We have standardized on it for years for all custom parsing needs. Most parsers are small, but some are larger, like Excel expressions. Of course, our input is mostly Unicode, either UTF-8 or UTF-16. Matching Unicode is complex. Comparison by code point is usually not the right thing. Instead, we must normalize, for which we even have various choices of what to accept as equal: https://en.wikipedia.org/wiki/Unicode_equivalence Case-insensitive matching is even more complex, slow, and even language-dependent. Input is often not guaranteed to be valid Unicode. For example, file names on Windows are sequences of 16-bit units, allowing unmatched surrogates, same with input from Win32 edit boxes and file content. But we realized that for almost all grammars we have, all this complexity does not matter. The reserved symbols of most grammars (JSON, XML, C++, URLs, etc.) are pure ASCII. Semantically relevant strings are ASCII as well („EXCEL.EXE“). ASCII can be correctly and quickly matched on a per-code-unit basis. Case-insensitive matching for ASCII is simple and fast. User-defined strings, such as JSON string values, may contain Unicode, but then they usually do not affect parsing decisions. The user may want Unicode validation for these strings, but this can be done by the leaf parser for these strings, rather than for the whole input. Since so much matching is against ASCII, we found it useful to have compile-time known ASCII literals in the parser library. With them, the same grammar can be used for all input encodings. When parsing user-defined strings, they will have the encoding of the input, but that’s fine. Any encoding conversion can be dealt with separately from the parser. Finally, we may want to parse more than just strings. Parsing binary files, or sequences of DNA, should be possible and efficient. Thus I recommend separating Unicode processing from the parser library. The parser library operates on an abstract stream of symbols. For Unicode text these would be code units. It provides the composite parsers such as sequences with and without backtracking, alternatives, Kleene star etc., and leaves the interpretation of the symbols entirely to the leaf parsers, which may or may not care about Unicode. We started modifying Boost.Spirit in this direction, and it serves us well. -- Dr. Arno Schödl CTO schoedl(a)think-cell.com<mailto:schoedl@think-cell.com> | +49 30 6664731-0 We are looking for C++ Developers: https://www.think-cell.com/developers think-cell Software GmbH (Web site<https://www.think-cell.com>) Leipziger Str. 51, 10117 Berlin, Germany Main phone +49 30 6664731-0 | US toll-free +1 800 891 8091 Amtsgericht Berlin-Charlottenburg HRB 180042 Directors: Christoph Hobo, Dr. Arno Schödl Please refer to our privacy policy<https://www.think-cell.com/privacy> on how we protect your personal data.

3 4

Odd build error in 1.85 pre-release - filesystem
by Tom Kent 23 Feb '24

23 Feb '24

When building for windows against the 1.85.0 nightly snapshot, I'm seeing an odd error with no info around it: - type_erasure : building - url : building - wave : building ...patience... ...patience... ...patience... ...patience... ...patience... ...patience... ...patience... [errno 22] failed to scan file '': Invalid argument...found 39744 targets... Upping the verbosity in B2, I was able to find that it appears to happen in filesystem: bind -- <object(c-scanner)@8602>boost\filesystem\detail\type_traits\disjunction.hpp: boost\filesystem\detail\type_traits\disjunction.hpp time -- <object(c-scanner)@8602>boost\filesystem\detail\type_traits\disjunction.hpp: 2024-02-22 07:18:09.000000000 +0000 make -- <object(c-scanner)@8602>boost\filesystem\detail\type_traits\disjunction.hpp make -- <object(c-scanner)@8602>boost\filesystem\detail\type_traits\disjunction.hpp time -- <object(c-scanner)@8602>boost\filesystem\detail\type_traits\disjunction.hpp: unbound made stable <object(c-scanner)@8602>boost\filesystem\detail\type_traits\disjunction.hpp make -- <object(c-scanner)@8602>boost\filesystem\detail\header.hpp make -- <object(c-scanner)@8602>boost\filesystem\detail\header.hpp bind -- [errno 22] failed to scan file '': Invalid argument <object(c-scanner)@8602>boost\filesystem\detail\header.hpp: boost\filesystem\detail\header.hpp time -- <object(c-scanner)@8602>boost\filesystem\detail\header.hpp: 2024-02-22 07:18:09.000000000 +0000 make -- <object(c-scanner)@8602>boost\filesystem\detail\header.hpp make -- <object(c-scanner)@8602>boost\filesystem\detail\header.hpp time -- <object(c-scanner)@8602>boost\filesystem\detail\header.hpp: unbound made stable <object(c-scanner)@8602>boost\filesystem\detail\header.hpp Anyone have ideas on what this can be? Why it doesn't show up more like a normal error? Thanks, Tom

3 4

2023 Transparency Report
by Louis Tatta 21 Feb '24

21 Feb '24

Hello, Louis Tatta here, CEO of The C++ Alliance, Inc, a registered 501(c)(3) non-profit organization dedicated to the support of C++. This is our 2023 transparency report regarding our charitable work for the Boost C++ Library Collection. First, some background on our operations. The C++ Alliance seeks out the highly talented and experienced C++ experts and hires them, usually full-time, so they can focus completely on their open-source projects. Our Staff Engineers maintain existing Boost libraries, write new Boost libraries, and contribute to other Boost projects. We employ IT experts full-time to maintain and improve the infrastructure which supports Boost, including the legacy web server, the Drone continuous integration servers, the scripts used to deploy GitHub Actions for Boost repositories, and the scripts which publish the Boost release. In addition, we provide on-demand technical support services for Boost authors and maintainers who have infrastructure related problems. The Boost Mailing List continues to be the premier place for high-quality discussion of the Boost libraries and tangential C++ Standards related topics. The C++ Alliance supports the use of the mailing lists going forward. We are upgrading it to the latest Mailman version, and building an optional web-based front-end allowing interaction using only a browser. We continue to operate the Official C++ Language Slack Workspace (https://cpp.al/slack) which hosts the most popular interactive discussions regarding the Boost libraries. The C++ Alliance sponsors a paid plan to provide full history, video and audio conferences, collaboration tools, and unlimited document storage for seventeen dedicated Boost library channels and one main Boost discussion channel. Our largest project for 2023 is the continuation of work on the new Boost website. The new site is a C++ social media portal with rich support for user-generated content, and a powerful back-end which has visibility into the contributions made on GitHub and the release process. This allows us to deliver an elevated user experience with the hopes of attracting fresh talent to Boost. Also,note that the graphics on the website are hand-drawn by our contract artist and reflect our level of commitment to ensure that Boost, to put it simply, looks good. In 2023 there were a total of four Boost Formal Reviews. One of the reviewed libraries is authored by a C++ Alliance Staff Engineer. Three of the reviews were managed by C++ Alliance team members. Out of a total of twenty-eight submitted reviews for the year, twelve were from C++ Alliance team members. Last year we increased our efforts to build Boost’s social media presence. There were a total of twelve messages from the @Boost_Libraries X (formerly Twitter) account, all submitted by C++ Alliance team members. Three of them contained artwork from Bob Ostrom commissioned by us. Messages containing images get three times more engagement than text-only messages. We believe that the X platform has unique strategic value for informing newer generations about the Boost value proposition. A total of $1,768,000 was spent on Boost-related contributions, broken down thusly (and rounded to the nearest thousand): Staff Engineers Compensation $1,240,000 Website Software Development $570,000 Server Hosting $62,000 C++ Slack Workspace $96,000 Conference Sponsorship $10,000 Conference Attendance $10,000 During 2023 we employed a total of 12 individuals to contribute to Boost. Here is a high-level summary of accomplishments: Sam Darwin (Chief Technical Officer) * Boost website maintenance and administration * Boost per-pull-request documentation previews * Boost Release Process development, maintenance, and administration * Boost Mailman services maintenance and upgrade * Boost GitHub Actions self-hosted runner project development * Boost Drone service administration and maintenance * New Boost website deployment, administration, and development Dmitry Arkhipov (Staff Engineer) * Boost.JSON maintenance Alan de Freitas (Staff Engineer) * Boost.URL maintenance * Mr. Docs project management (https://mrdocs.com) * GitHub actions support scripts for Boost CI (https://alandefreitas.github.io/cpp-actions/cpp-actions/index.html) Christian Mazakas (Staff Engineer) * Boost.Unordered evolution project * Boost.Buffers development (new library) * Boost.Http.Proto, Boost.Http.Io development (new libraries) * Boost.Websocket.Proto, Boost.Websocket.Io development (new libraries) Krystian Stasiowski (Staff Engineer) * Boost.StaticString maintenance * Mr. Docs Lead engineer (https://mrdocs.com/) * Clang/LLVM contributions Peter Turcan (Senior Technical Writer) * Boost User Guide development (new documentation) * Boost Contributor Guide (new documentation) * Boost Formal Reviews (new documentation) Matt Borland (Staff Engineer) * Boost.Charconv (new, accepted library) Ruben Perez * Boost.MySQL maintenance (new, accepted library) * User Guide, Contributor Guide contribution (new documentation) Mohammad Nejati (Staff Engineer) * Boost website full-text search feature * Boost.Beast maintenance Fernando Pelliccioni (Staff Engineer) * Mr. Docs development (https://mrdocs.com/) Spencer Strickland (Senior Web Developer) * New website front-end and scripting Klemens Morgenstern (Staff Engineer) * Boost.Beast maintenance * Boost.Cobalt (new, accepted library) Joaquin M Lopez Munoz (Staff Engineer) * Boost.Unordered evolution project lead * Boost Contributor Guide contributions (new documentation) Our sources of funding are secure and we look forward to continuing our charitable contributions throughout this decade and beyond. It is an honor and a pleasure to support the Boost C++ Library collection. Thank You Louis Tatta

5 4

Reminder: Boost 1.85.0 will be closing for new libraries and breaking changes THIS WEDNESDAY
by Marshall Clow 20 Feb '24

20 Feb '24

As always, look at the calendar at https://www.boost.org/development/ — Marshall

2 1

Review of Parser library begins today
by Marshall Clow 19 Feb '24

19 Feb '24

The review of Zach Laine’s proposed Boost.Parser library begins today and will end on February 28th. From the introduction page of the documentation: Boost.Parser is a parser combinator library. That is, it consists of a set of low-level primitive parsers, and operations that can be used to combine those parsers into more complicated parsers. There are primitive parsers that parse epsilon (the empty string), chars, ints, floats, etc. There are operations which combine parsers to create new parsers. For instance, the Kleene star operation takes an existing parser p and creates a new parser that matches zero or more occurrences of whatever p matches. Both callable objects and operator overloads are used for the combining operations. For instance, operator*() is used for Kleene star, and you can also write repeat(n)[p] to create a parser for exactly n repetitions of p. Boost.Parser also tries to accommodate the multiple ways that people often want to get a parse result out of their parsing code. Some parsing may best be done by returning an object that represents the result of the parse. Other parsing may best be done by filling in a preexisting data structure. Yet other parsing may best be done by parsing small sections of a large document, and reporting the results of subparsers as they are finished, via callbacks. Boost.Parser accommodates all these ways of working, and even makes it possible to do callback-based or non-callback-based parsing without rewriting any code (except by changing the top-level call from parse() to callback_parse()). All of Boost.Parser's public interfaces are sentinel- and range-friendly, just like the interfaces in std::ranges. Boost.Parser is Unicode-aware through and through. When you parse ranges of char, Boost.Parser does not assume any particular encoding — not Unicode or any other encoding. Parsing of inputs other than plain chars assumes that the input is Unicode. In the Unicode-aware code paths, all parsing is done by matching code points. This means that you can feed UTF-8 strings into Boost.Parser, both as input and within your parser, and the right sort of matching occurs. For instance, if your parser is trying to match repetitions of the char '\xcc' (which is a lead byte from a UTF-8 sequence, and so is malformed UTF-8 if not followed by an appropriate UTF-8 code unit), it will not match the start of "\xcc\x80" (UTF-8 for the code point U+0300). Boost.Parser knows that the matching must be whole-code-point, and so it interprets the char '\xcc' as the code point U+00CC. Error reporting is important to get right, and it is important to make errors easy to understand, especially for end-users. Boost.Parser produces runtime parse error messages that are very similar to the diagnostics that you get when compiling with GCC and Clang (it even supports warnings that don't fail the parse). The exact token associated with a diagnostic can be reported to the user, with the containing line quoted, and with a marker pointing right at the token. Boost.Parser takes care of this for you; your parser does not need to include any special code to make this happen. Of course, you can also replace the error handler entirely, if it doesn't fit your needs. Debugging complex parsers can be a real nightmare. Boost.Parser makes it trivial to get a trace of your entire parse, with easy-to-read (and very verbose) indications of where each part of the trace is within the parse, the state of values produced by the parse, etc. Again, you don't need to write any code to make this happen — you just pass a parameter to parse(). Github: https://github.com/tzlaine/parser Docs https://tzlaine.github.io/parser — Marshall

1 0

Re: [boost] Should boost::core::string_view be made public?
by g.peterhoff＠t-online.de 18 Feb '24

18 Feb '24

Have you come to a conclusion how this should be handled (in the future)? The problem is basically that there are different implementations for string_view. I don't know why it was done this way. Presumably it's because various boost libraries want to be independent of others, so that some things are implemented anew each time (within the library) instead of using existing libraries. This will also be due to the fact that boost is not atomistic enough, so that large libraries might have to be included/used, although only a fraction of the functionality is needed. My suggestion: - Throw out boost::utility::string_view - Make boost::core::detail::string_view public, just like all other backports in boost::core. - Check if there are other multiple implementations. If so, pull them out into separate small boost libraries. (For databases, this would be called normalization, but it must not be over-normalized). regards Gero

5 4

Potential Launch of Boost Website: Request for Final Feedback
by Louis Tatta 17 Feb '24

17 Feb '24

Louis here from the C++ Alliance. With the potential launch of the new Boost website on the horizon this week, we're making a final call for your feedback. Your insights a few months ago were incredibly helpful, and as we approach the launch, we'd value your final review. If you have a moment to look over the site again and note any issues or suggestions, it would be of great help. Feedback can be submitted on our GitHub issues page: https://github.com/cppalliance/temp-site/issues. Thank you for your ongoing support and contributions to the Boost community. Looking forward to your feedback. Best regards, Louis C++ Alliance

18 79

Re: [boost] New Website Repositories
by Glen Fernandes 16 Feb '24

16 Feb '24

On 2024-02-16, Vinnie Falco wrote: > On Fri, Feb 16, 2024 at 12:33, Glen Fernandes wrote: > > If you want to follow the tradition of source files, the easiest > > convention to follow is to state that your files, e.g. > > > >  > > We can do that, or alternatively is it possible to just plop a LICENSE > file at the root of the repository? I would prefer both (thanks). Glen

1 0