- Boost - lists.preview.boost.org

[review][beast] Late review
by Christopher Kohlhoff 16 Jul '17

16 Jul '17

I realise this is quite late, but Vinnie asked for my opinion, so I asked for special dispensation from the review manager. Hopefully that is ok! In a nutshell, I think Beast should be accepted. Background: Although I may have some use for websockets in the future, my use case right now requires only HTTP (sorry Vinnie!). The use case involves incorporating HTTP protocol support into network applications whose primary purpose is something otherwise unrelated to HTTP. HTTP is used to support monitoring, configuration, and debugging facilities. Unlike typical HTTP use, the protocol is implemented not just over TCP sockets but also over local (UNIX domain) sockets. It is essential that the HTTP implementation minimises runtime overhead to avoid negatively impacting the quality of service for the other protocols. - What is your evaluation of the design? Overall the design is good. The library seems to be structured into several layers: 1. low level stateful parsers and serializers 2. operations to send/receive partial messages (*_some operations) 3. operations to send/receive header 4. operations to send/receive complete messages, using parser/serializer supplied by me 5. operations to send/receive complete messages This is a good thing, because it lets me select the appropriate layer for a particular use case. Whichever of layers 2 - 4 I choose, the best thing is that they all follow the same model for asynchronous operations as in Asio. All four of those layers work with arbitrary Asio streams, which was a requirement of my use case. This made using the library seamless, easily replacing my otherwise quick and dirty HTTP encoding and decoding approaches. The benefit of course is that someone else is taking care of ensuring the messages are well-formed, and doing a much better job of it. Layer 5 is good for simplicity and clarity. Based on my benchmarking, layer 4 seems to be the sweet spot for performance sensitive use cases, as it gives sufficient control of the placement of the "transient" objects used by the asynchronous operations. I use HTTP/1.0 almost exclusively, and one reason for this is that HTTP/1.0 allows the response to be sent without a Content-Length, with the end of content delimited by stream closure. One use for this is an infinite stream of data (e.g. monitoring output). Another use case is the ability to use the file descriptor as stdout for a child process. In both these cases, layer 3 fits the bill where I can send just the response header, and then take over the socket for my own needs. As for layer 2, the documentation for these functions still refers in places to a return value (or additional size_t argument to the completion handler), which is no longer in the code: synchronous read_some has a void return and async_read_some has a void(error_code) handler. Although this is probably vestigial documentation, actually I would prefer it if these lower level operations *did* return something, like the number of bytes transferred in the operation. This would assist in detecting badly behaved or pathological clients (like ones doing everything one byte at a time), and this is probably the main reason I would consider using the layer 2 operations. I haven't yet needed layer 1, but I can imagine how it might be used in transport agnostic code. I guess my main point is that by supplying all these layers, Beast is both powerful and flexible, and meets pretty much all of my likely needs. The body/reader/writer design feels a little awkward, and it seems like there is a more elegant design struggling to get out. However, my primary reason for looking at implementing a custom body was to support serving static files. As Vinnie now plans to include file support in the library itself, I have little reason to work directly with body/reader/writer directly any longer. Consequently, I am less concerned about the design here. However, I'd encourage Vinnie to keep exploring this. On a related note, once file body was added I was pleased to see how straightforward it was to incorporate platform-specific file transmission (TransmitFile, sendfile, etc) in a way that was transparent to the library user. There are some minor inconsistencies in template parameters, e.g.: parser has Allocator where serializer has Fields. I would have expected some symmetry here. The library includes reimplementations of a number of facilities from the networking TS. However, now the TS is done I plan to update Boost.Asio to match. If Beast is accepted then hopefully by the time it is incorporated, it will no longer need to include these at all. I'm not a huge fan of the library including typedefs for names from other boost libraries as part of the documented public interface, e.g. beast::error_code, beast::string_view. However, if it is to retain these, I'd at least like to see them given their own section in the documentation quick reference to make it clear that they are just aliases for external functionality. (Note: I'm just fine with the aliases themselves as a convenience for the library implementer.) - What is your evaluation of the implementation? The implementation is high quality. I do wonder about the implementation approach taken in the basic_fields storage. It uses a set, which is great asymptotically, but does mean allocation of a lot of small objects. In my experience (and in a totally unscientific survey of commonly used websites) the number of headers is usually small -- a dozen or two. In addition, when constructing or decoding a message, one would normally construct the set of fields in one go. Deletion of a field seems like it would be rare. Thus I wonder whether basic_fields could use a storage strategy geared towards the common case, e.g. a single storage blob for the fields themselves (vector<char>) plus an index (vector<size_t> of offsets to each field's beginning) for quick iteration. I don't think this would impact the API, so it's not essential. - What is your evaluation of the documentation? The documentation appears to be comprehensive. As mentioned above I did spot a few stray things that seemed to be left over from previous design iterations. There's also the occasional incomplete specification, e.g. basic_fields::at doesn't way what it does with multiple fields. Probably best to just raise these as issues. - What is your evaluation of the potential usefulness of the library? Very useful. It fits very well with my use cases. - Did you try to use the library? With which compiler(s)? Did you have any problems? Using gcc 6.1 on linux, I spent some time writing and rewriting several small http servers to evaluate the library, and Vinnie has now incorporated some of these into the beast repository as examples. Along the way I did discover some minor opportunities for improvement in the library, and Vinnie was quick to respond. - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I spent many hours examining the library over a period of a few weeks, both before and during the review period. I gave the implementation and examples in-depth study, less so the documentation. - Are you knowledgeable about the problem domain? I hope Vinnie asked me to review Beast because I know a thing or two. However, I must qualify that: my experience in HTTP is almost entirely 1.0 and I only have a passing knowledge of websockets. Many thanks to Vinnie for creating Beast and proposing it for inclusion in boost. It is nice to see the addition of protocol implementations that build on top of Asio's asynchronous model so seamlessly. Cheers, Chris

1 0

Re: [boost] [Boost-users] Boost 1.65.0 Beta 1 [boost.python] issue
by Rene Rivera 15 Jul '17

15 Jul '17

On Fri, Jul 14, 2017 at 7:23 AM, Raffi Enficiaud via Boost-users < boost-users(a)lists.boost.org> wrote: > As I said in a previous post, I cannot get boost.python to build on > Windows: the target is just ignored (see stack trace below). > Missed that post :-( > I think there is a bug in .\tools\build\src\tools\python.jam A highly probable statement all on its own :-) > concerning the version computation: the variable "major-minor" is not set > properly if "version" is optional. > > In this block: > > ============= ("rule configure") > if $(version) > { > if ! $(version) in [ feature.values python ] > { > feature.extend python : $(version) ; > py$(major-minor[1])-version ?= $(version) ; > ============= > > The variable "major-minor" is empty. > > The "if" in the following block prevents from computing major-minor as > "version" is optional and is not set: > > ============= > local major-minor ; > if $(version) > { > major-minor = [ split-version $(version) ] ; > version = $(major-minor:J=.) ; > } > ============= > > I believe this can be fixed by recomputing "major-minor" in this block, > after "version" has been inferred. > > ============= > if $(cmd) > { > debug-message ...requested configuration matched! ; > > exec-prefix = $(sys.exec_prefix) ; > > compute-default-paths $(target-os) : $(sys.version) : > $(sys.prefix) : $(sys.exec_prefix) ; > > version = $(sys.version) ; > interpreter-cmd ?= $(cmd) ; > cmds-to-try = ; # All done. > } > ============= > Seems reasonable.. Can you put up a PR for that? -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

2 1

Re: [boost] [fiber] PR: Replace #include "Windows.h" by "windows.h"
by Frédéric Bron 14 Jul '17

14 Jul '17

>> I have just checked and both mingw and mingw-w64 have only fully lower >> case >> headers. I just checked all includes files from boost (1.65.0 beta2) that have upper case character in their names and are present in mingw or mingw-w64 in lower case. Here is what I found: 3 files, Activation.h, Memory.h, Windows.h. Those are in the following files: ./libs/thread/src/win32/thread.cpp:36:#include <Activation.h> ./libs/iostreams/test/windows_pipe_test.cpp:9:#include <Windows.h> ./libs/process/test/bind_stderr.cpp:32:# include <Windows.h> ./libs/process/test/bind_stdin.cpp:34:# include <Windows.h> ./libs/process/test/sparring_partner.cpp:31:# include <Windows.h> ./libs/process/test/bind_stdout_stderr.cpp:30:# include <Windows.h> ./libs/process/test/exit_code.cpp:21:# include <Windows.h> ./libs/process/test/bind_stdout.cpp:32:# include <Windows.h> ./libs/process/test/group.cpp:32:# include <Windows.h> ./libs/process/test/spawn.cpp:33:# include <Windows.h> ./libs/process/test/spawn_fail.cpp:29:# include <Windows.h> ./libs/process/test/system_test1.cpp:38:# include <Windows.h> ./tools/build/src/engine/boehm_gc/MacOS.c:19:#include <Memory.h> ./boost/fiber/detail/futex.hpp:21:#include <Windows.h> ./boost/fiber/detail/cpu_relax.hpp:19:# include <Windows.h> So that is not a lot. For Memory.h, it seems related to MacOS, not Windows. If people really want to keep the upper case letter, we could then have #ifdef __MINGW32__ #include "windows.h" #else #include "Windows.h" #endif Frédéric Frédéric

1 0

[beast] Version 80: code review opportunity
by Vinnie Falco 14 Jul '17

14 Jul '17

Beast version 80 is sitting in the pull request queue if anyone is interested in providing a code review. This contains changes motivated by Bjorne's desire for more control over chunk-encoded output. <https://github.com/vinniefalco/Beast/pull/641> List of changes since beginning of the formal review: -------------------------------------------------------------------------------- * Javadoc tidying * Add basic_dynamic_body.hpp * Shrink buffer_prefix_view * Remove unused file_path * Add basic_file_body.hpp * buffers_ref is Assignable * Shrink chunk header buffer sequence size * Refactor chunked-encoding serialization * Remove spurious fallthrough guidance * Add span * Documentation work * Use make_unique_noinit * Fix warning in zlib * Header file tidying * Tidy up FieldsReader doc * Add Boost.Locale utf8 benchmark comparison * Tidy up dstream for existing Boost versions * Tidy up file_posix unused variable * Fix warning in root ca declaration * Tidy up basic_string_body * Add vector_body * span, string, vector bodies are public * Fix spurious uninitialized warning * fields temp string uses allocator * Add message::keep_alive() * Add message::chunked() and message::content_length() * Remove string_view_body * file_posix works without large file support * Always go through write_some * Use Boost.Config * BodyReader may construct from a non-const message * Add serializer::get * Add serializer::chunked * Serializer members are not const * serializing file_body is not const * Add file_body_win32 * Fix parse illegal characters in obs-fold * Disable SSE4.2 optimizations * Rename to serializer::keep_alive * BodyReader, BodyWriter use two-phase init * Use file_body for valid requests, string_body otherwise. * Construct buffer_prefix_view in-place * Shrink serializer buffers using buffers_ref * Tidy up BEAST_NO_BIG_VARIANTS * Shrink serializer buffers using buffers_ref * Add serializer::limit * file_body tests * Using SSE4.2 intrinsics in basic_parser if available * Add file_stdio and File concept * Add file_win32 * Add file_body * Remove common/file_body.hpp * Add file_posix * Fix Beast include directories for cmake targets * remove redundant flush() from example * Jamroot tweak * Verify certificates in SSL clients * Adjust benchmarks * Initialize local variable in basic_parser * Fixes for gcc-4.8 * basic_parser optimizations * Add basic_parser tests * Refactor header and message constructors * serializer::next replaces serializer::get * Tidy up set payload in http-server-fast * Refine Body::size specification * Newly constructed responses have a 200 OK result * Refactor file_body for best practices * Add http-server-threaded example * Documentation tidying * Various improvements to http_server_fast.cpp * Add websocket-server-async example * Fix extra ; warning * Documentation revision * Fix spurious on_chunk invocation * Call prepare_payload in HTTP example * Check trailers in test * Fix buffer overflow handling for string_body and mutable_body * Concept check in basic_dynamic_body * Tidy up http_sync_port error check * Tidy up Jamroot /permissive- * Fine tune websocket op asserts * Refactor websocket composed ops * Allow close, ping, and write to happen concurrently * Fix race in websocket read op * Fix websocket write op * Add cmake options for examples and tests * Return `std::size_t` from `Body::writer::put` Thanks

3 4

Re: [boost] [fiber] PR: Replace #include "Windows.h" by "windows.h"
by Frédéric Bron 14 Jul '17

14 Jul '17

>> Maybe we could use lower case windows.h only if __MINGW32__ is defined? > > It's still a fudge that might break. Isn't it better to get to the bottom of > the (MinGW-)problem (other headers are affected as well, as Windows.h is not > the only header starting with a capital)? That's right, we could look at all #includes in boost. I have just checked and both mingw and mingw-w64 have only fully lower case headers. Frédéric

1 0

[beast] Formal review
by Bjorn Reese 14 Jul '17

14 Jul '17

I am going to focus exclusively on the HTTP API in the review below because that needs most attention. Furthermore, to keep this review within reasonable bounds, I am omitting issues already reported by others. I. DESIGN --------- Beast offers a well-designed WebSocket API along with a minimal and deficient HTTP API. Beast claims to offer low-level building-blocks that can be used to produce higher-level HTTP server and client libraries. The low-level approach makes Beast minimal. I do not have a problem with a minimal API, other than it diminishes the usefulness of the library. Although this can be solved by adding more functionality over time, if I was to evaluate Beast with its current feature-level, I would only choose it if I needed WebSocket. This, however, is not a precondition for library acceptance. But the low-level approach also means the abstractions must be right. The abstractions that Beast currently offers are deficient, which is best shown by the poor support for HTTP chunking. Chunking can and should be represented more naturally by a HTTP API. HTTP chunking splits a message into several parts, where each part can contain metadata (chunk extensions) and data (chunk.) The metadata is a set of key-value pairs, just like the HTTP headers. During the review period I have described several use cases for HTTP chunking that does require metadata (e.g. radio metadata,) as well as the preservation of chunk boundaries (e.g. event-based publish-subscribe channels.) Conceptually, a normal HTTP message has the following structure: message = start-line metadata* body? A chunked HTTP message can be described in a similar manner as: message = start-line metadata* ( metadata* chunk? )* The Beast message container is designed for the structure of the normal HTTP message, and assumes that chunks are appended the body. The chunk metadata does not fit into this model, and therefore Beast discards them by default. The repetitive structure of chunked messages is not accommodated by the Beast message container. This means that we have to make our own chunk decorator to generate chunk metadata, and our own custom parser to receive chunk metadata. The chunk decorator inserts metadata via a callback mechanism. Its design is poor for two reasons. First, the user must keep track of context to ensure that the correct metadata is applied to the correct chunks. Second, the user must write the metadata in correct HTTP syntax. The latter may sound trivial, but does anybody here know if " ;key=value\r\n" is correctly formatted? (notice the initial whitespace.) The answer is that it is correct according to RFC 2616, incorrect according to RFC 7230, and correct according to RFC 7230 errata. The best approach for handling this situation is to never insert whitespaces, but to always accept whitespaces from others. Users should not have to be bothered with such detail. Receiving chunk metadata rather than having them discarded by default requires a custom parser. The Beast message parser is a push parser (SAX) where callbacks are invoked for each parsed message part. This means that the user must keep track of which metadata belongs to which chunk. The user must also parse the metadata manually. Beast does not even check if the syntax is correct. The custom parser is also necessary if the user needs to receive individual chunks rather than having them appended to the body. I would have expected that a low-level API returns the next message part as soon as it is identified. The current collect-entire-message design is a higher-level design. In summary, 1. Beast should not discard anything by default, but should faithfully forward all information received. 2. Beast should handle all low-level syntax and not leave parts of it to the user. 3. Beast should not break the end-to-end principle unless allowed by the user. As a way forward I am going to sketch another simple message model that does encompass HTTP chunking. Conceptually it looks like this (I am going to ignore that, say, metadata may require different syntax depending on context) message = start-line section* section = metadata* body-part? Building an API around this model should offer read/write functions that handles partial bodies (whether chunked or progressively downloaded.) The current collect-entire-message API could be build on top of that. With this approach there is no need to mess around with custom parsers. This design does require state information to be able to select the proper syntax for both metadata and data. On another topic, the HTTP status codes can be roughly categorized as instructions (the 1xx and 2xx series) and errors (the 4xx and 5xx series.) I recommend that you create two error categories and pass status codes as error_code. Notice that I have no comment about whether or not they should be stored in the message container. What I really want to achieve is to enable the user to easily propagate HTTP errors in the same way that non-HTTP errors (including beast::http::error) are propagated. II. IMPLEMENTATION ------------------ I read the implementation to understand the design, so I did not look for QA issues. My overall impression is that much of the code is well-written, but there are parts that are convoluted and difficult to verify. There are too many goto statements for my taste; I do not mind the goto pattern used in the parser, but the state-machines in read_some_op and serializer looks more like micro-optimization. Furthermore, "case do_body + 1:" looks like an awful hack. The dynamic buffers are useful even by themselves. The code contains several foreign files with different licensing conditions: * include/beast/core/detail/base64.hpp * include/beast/zlib/zlib.hpp * include/beast/websocket/detail/utf8_checker.hpp * include/beast/http/detail/basic_parser.hpp All these include both the BSL and the original license. This should be checked by the Boost legal council. III. DOCUMENTATION ------------------ The documentation is generally well-written (and it looks like the DynamicBuffer section got a little help from Networking TS.) The Reference is missing some concepts, e.g. SyncReadStream. IV. NAMING ---------- The library name must change. "Boost.Beast" sounds like a bad joke. I do not have strong opinions about the new name. V. VERDICT ---------- I recommend to REJECT the library because the HTTP API provides the wrong low-level abstractions as explained in the design section above.

12 46

Follow-up to the follow-up of the CMake follow-up
by Peter Dimov 13 Jul '17

13 Jul '17

In branch feature/install-cmake-config-2 of the superproject, you can find an improved version of the award-winning work presented in my previous follow-up. This one integrates into the top-level "b2 install" and "b2 stage" and installs (or, respectively, stages) the CMake configuration files necessary for find_package to work. The instructions for trying it out are the same as last time: git clone --recursive --depth 64 -b feature/install-cmake-config-2 https://githib.com/boostorg/boost bootstrap b2 headers b2 install then add the installation prefix to CMAKE_PREFIX_PATH. An example CMakeLists.txt could look like this: cmake_minimum_required(VERSION 3.5) project(cmake_demo CXX) list(APPEND CMAKE_PREFIX_PATH C:/Boost) find_package(boost_filesystem 1.63.0 REQUIRED) find_package(boost_program_options 1.63.0 REQUIRED) # disable autolink add_definitions(-DBOOST_ALL_NO_LIB=1) # sample program add_executable(cmake_demo cmake_demo.cpp) target_link_libraries(cmake_demo boost::filesystem boost::program_options)

4 8

[review][beast] Review of Beast starts today : July 1 - July 10
by Michael Caisse 13 Jul '17

13 Jul '17

The formal review of Vinnie Falco’s Beast library will take place from July 1 through July 10, 2017. Please consider participating in this review. The success of Boost is partially a result of the quality review process which is conducted by the community. You are part of the Boost community. I will be grateful to receive a review based on whatever level of effort or time you can devote. Beast is a C++ header-only library serving as a foundation for writing interoperable networking libraries by providing low-level HTTP/1, WebSocket, and networking protocol vocabulary types and algorithms using the consistent asynchronous model of Boost.Asio. Beast is designed for: * Symmetry: Algorithms are role-agnostic; build clients, servers, or both. * Ease of Use: Boost.Asio users will immediately understand Beast. * Flexibility: Users make the important decisions such as buffer or thread management. * Performance: Build applications handling thousands of connections or more. * Basis for Further Abstraction: Components are well-suited for building upon. A branch has been made for the review. You can find Beast for review at these links: * Documentation : <http://vinniefalco.github.io/stage/beast/review/> * Source code : <https://github.com/vinniefalco/Beast/tree/review> * The FAQ answers real questions that come up : <http://vinniefalco.github.io/stage/beast/review/beast/design_choices/faq.ht…> A lightning talk from CppCon 2016 introducing Beast can be found here: <https://www.youtube.com/watch?v=uJZgRcvPFwI> Please provide in your review whatever information you think is valuable to understand your final choice of ACCEPT or REJECT including Beast as a Boost library. Please be explicit about your decision. Some other questions you might want to consider answering: - What is your evaluation of the design? - What is your evaluation of the implementation? - What is your evaluation of the documentation? - What is your evaluation of the potential usefulness of the library? - Did you try to use the library? With which compiler(s)? Did you have any problems? - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? - Are you knowledgeable about the problem domain? More information about the Boost Formal Review Process can be found here: <http://www.boost.org/community/reviews.html> Thank you for your effort in the Boost community. Happy coding - michael -- Michael Caisse Ciere Consulting ciere.com

24 107

Master branch now open for merging
by Marshall Clow 13 Jul '17

13 Jul '17

I will be uploading 1.65.0 beta1 this morning. Hint: It's the same as rc2 ;-) -- Marshall

2 1

[fiber] PR: Replace #include "Windows.h" by "windows.h"
by Frédéric Bron 13 Jul '17

13 Jul '17

Hi, A #include "Windows.h" causes troubles when cross-compiling on Linux in C++14 mode with MinGW-W64 with Posix threads. Can we change it to "windows.h"? Here is a PR: https://github.com/boostorg/fiber/pull/134 Thanks, Frédéric

2 2