Background: Websocket connections start with an HTTP UPGRADE request/response sequence. Often people want to customize this handshake so I added code in Beast to read and write HTTP messages. Upon reviewing the HTTP RFC (rfc2616 and rfc7230) it seemed to be logical that the HTTP message could be modeled as a single class. And there could be free functions implementing the algorithms to serialize and deserialize these messages in th HTTP/1 format. I took this as far as I could and I was pretty happy with the result. You have functions like read, async_read, write, and async_write which take beast::http::message objects as parameters. Example: http://vinniefalco.github.io/beast/beast/ref/http__read/overload3.html The innovation here is the "Body" template type which controls not only the type of container used to represent the body but also the algorithms used to transfer bytes from the message body on the wire into and out of that container. The Body concept, and its nested Reader and Writer concepts, are documented here: http://vinniefalco.github.io/beast/beast/ref/Body.html http://vinniefalco.github.io/beast/beast/ref/Reader.html http://vinniefalco.github.io/beast/beast/ref/Writer.html Previous efforts and HTTP implementations focused heavily on the representation and parsing of the headers. To my knowledge Beast is the first implementation to provide caller flexibility in the body representation. Now the Beast websocket implementation has a reasonable set of primitives it could use to perform the handshake, and users could get control of that process. The Problem: Unfortunately, while treating the message as an object that can be read/written atomically works great for example programs and trivial use-cases, it fails in the real world. These are issues with it: * The caller must commit to a Body representation before receiving the HTTP Header * No way to properly read and respond to Expect: 100-continue * An efficient HTTP relay function (one that does not buffer the entire body up front) cannot be implemented by the caller * The caller cannot reasonably set a timeout, for example what if the body is 2 gigabytes received over a slow connection? * The parser interface forces algorithms to perform an unnecessary buffer copy for body containers that are entirely stored in memory (e.g std::string or boost::asio::streambuf). These issues sparked quite a bit of discussion on GitHub: https://github.com/vinniefalco/Beast/issues/154 https://github.com/vinniefalco/Beast/issues/265 In October of 2016 these problems became apparent. I tried a lot of different ideas to improve the design to solve it, but none of them worked. Some of my co-workers looked at it but we didn't really find anything great. One user in particular, on GitHub, had strong ideas involving the concept of "completion conditions" to augment the read/parse algorithms but it didn't feel right to me. The Solution: After suffering from writer's block for 5 months inspiration hit and I tried a combination of interface changes that got the design to where it needs to be to solve these problems. * The Reader concept is refined into two types: "indirect" readers, and "direct" readers. This is described here: http://vinniefalco.github.io/stage/beast/beast/ref/Reader.html * The parser exposes a small number of internal states, informing callers of where it is at and giving the caller control over if and when the parser advances to the next state: https://github.com/vinniefalco/Beast/blob/http/include/beast/http/basic_pars... * The parser is rewritten to work only on linear buffers (allowing it to be optimized to become 20 times faster if SSE2 instructions are allowed). To facilitate this requirement, the flat_streambuf class is provided which works just like a basic_streambuf except it guarantees a single buffer. If the parser is given multiple buffers, it allocates memory to flatten it: https://github.com/vinniefalco/Beast/blob/http/include/beast/core/flat_strea... * New functions parse_some and async_parse_some allow the caller to perform incremental parsing. This allows for the implementation of relay functions, or reading just the header (for Expect: 100-continue): https://github.com/vinniefalco/Beast/blob/http/include/beast/http/parse.hpp#... With these changes callers can get control of the inner loop that reads the body off the socket or stream, so they can do whatever custom steps are necessary if they want. They can set timeouts, and write their own loop that calls parse_some repeatedly. I think it solves all of these issues. I've written some test code which demonstrates how direct and indirect body readers are implemented, and what calling code looks like. I've also written a prototype "relay" function (read a message from one socket and efficiently write it to another socket). And there's an example of how Expect: 100-continue might be implemented. These prototypes are located here: https://github.com/vinniefalco/Beast/blob/http/test/http/design.cpp#L30 How to Help? These are some controversial changes and I think they could use some polish and vetting before I merge it to the master branch. Maybe there is something I haven't thought of, or a use-case it doesn't address. Perhaps there is a better way, or perhaps there are tweaks to make it even more usable and understandable. If some kind soul(s) would look through it and offer some feedback, positive or negative, that would be enormously helpful. The formal review for Beast is now scheduled for the first week of July and like an expectant father I am anxious and would like everything to go just right! My only objective is to polish up the library and its documentation so it is in the best possible shape for review. Thanks for listening!