Vinnie Falco wrote:

Its possible that what you are really asking is how to read a message payload incrementally? One of the examples performs a similar operation: http://vinniefalco.github.io/beast/beast/more_examples/http_relay.html

Something like this should achieve your goal (note, untested):

This preempts one of my questions, I was going to ask for precisely that example, as it's referenced in the documentation as a use case here: http://vinniefalco.github.io/stage/beast/review/beast/using_http/parser_stre... as "* Receive a large body using a fixed-size buffer."

On Sun, Jul 2, 2017 at 6:33 AM, Bjorn Reese via Boost wrote:

I am trying to investigate how your message model handles trailing headers.

Trailers are simply inserted into the message fields when they are received.

On Sun, Jul 2, 2017 at 6:52 AM, Bjorn Reese via Boost wrote:

Can I differentiate between headers and trailers, or are they mixed together?

They are mixed together. You can look at the "Trailer" field after receiving the header to know which trailers have been promised. And then when the message is complete, inspect the headers again looking for those specific fields. If you absolutely need to distinguish them, for example to perform more robust error checking, reject trailers which were not promised or which are disallowed as per rfc7230, then you can subclass basic_parser and provide a suitable implementation of on_field. In this case you could store them in a separate container if desired.

On Sun, Jul 2, 2017 at 7:34 AM, Bjorn Reese via Boost wrote:

So chunk-ext fields are not passed to the user unless they write a parser to extract them?

If you want the chunk extensions you have to subclass basic_parser (not the same as writing a parser). Here's an example of what a user-defined subclass could start with: template class parser : public basic_parser> { using base_type = basic_parser>; message m_; boost::optional<typename Body::writer> wr_; public: using value_type = message; parser() = default; parser(parser const&) = delete; parser(parser&& other) = default; parser& operator=(parser const&) = delete; template explicit parser(Args&&... args) : m_(std::forward<Args>(args)) { } value_type const& get() const { return m_; } value_type& get() { return m_; } value_type release() { return std::move(m_); } private: friend class basic_parser; void on_request(verb method, string_view method_str, string_view target, int version, error_code& ec) { try { m_.target(target); if(method != verb::unknown) m_.method(method); else m_.method_string(method_str); ec.assign(0, ec.category()); } catch(std::bad_alloc const&) { ec = error::bad_alloc; } m_.version = version; } void on_response(int code, string_view reason, int version, error_code& ec) { m_.result(code); m_.version = version; try { m_.reason(reason); ec.assign(0, ec.category()); } catch(std::bad_alloc const&) { ec = error::bad_alloc; } } void on_field(field name, string_view name_string, string_view value, error_code& ec) { try { m_.insert(name, name_string, value); ec.assign(0, ec.category()); } catch(std::bad_alloc const&) { ec = error::bad_alloc; } } void on_header(error_code& ec) { ec.assign(0, ec.category()); } void on_body(boost::optional< std::uint64_t> const& content_length, error_code& ec) { wr_.emplace(m_, content_length, ec); } std::size_t on_data(string_view s, error_code& ec) { return wr_->put(boost::asio::buffer( s.data(), s.size()), ec); } void on_chunk(std::uint64_t size, string_view ext, error_code& ec) { ec.assign(0, ec.category()); } void on_complete(error_code& ec) { if(wr_) wr_->finish(ec); else ec.assign(0, ec.category()); } }; The function on_chunk() is called for each new chunk header, and gets passed the size of the chunk as well as the extension.

On Sun, Jul 2, 2017 at 9:18 AM, Bjorn Reese via Boost wrote:

On 07/02/2017 04:44 PM, Vinnie Falco via Boost wrote: Chunking is (abstractly speaking) similiar to WebSocket frames, so I am surprised at how different these solutions are.

Actually Beast treats them very similarly. Its not generally possible in Beast to see the WebSocket frame boundaries either (especially if you have compression on). It might look like you can see frame boundaries because of the function `websocket::stream::read_frame` but that function can cross boundaries or return less than a full frame. As per rfc6455 (WebSocket) interfaces presented to the application (`beast::websocket::stream` in this case) are free to reframe messages.

On Sun, Jul 2, 2017 at 9:49 AM, Bjorn Reese via Boost wrote:

But their APIs are quite different.

I'm not sure I agree that the claimed abstract similarity of the concept warrants efforts to make the interfaces "consistent." The APIs were not planned ahead of time, they were arrived at after months of reworking to handle all of the use cases. Some use-cases were made very easy, at the expense of some other less common use-cases requiring more effort (such as extracting chunk extensions).

Does Beast re-packet WebSocket frames...

Yes

or HTTP chunks?

Chunk-encoding is removed from incoming messages before presentation to the application layer. For outgoing messages, Beast is under control over how pieces are chunked. However, the Body::reader type has influence but not complete control (the specifications is intentionally vague). Users can set chunk extensions and trailers by using a ChunkDecorator http://vinniefalco.github.io/beast/beast/using_http/serializer_stream_operat... Thanks

On Sun, Jul 2, 2017 at 10:51 AM, Bjorn Reese via Boost wrote:

...

...

Does Beast re-packet WebSocket frames... ... Yes ... Under what conditions?

There are stream settings for auto-fragmentation of messages into frames of prescribed sizes. Also, when the permessage-deflate algorithm is enabled, Beast reframes messages to fit in the fixed compression and decompression buffers it uses.

The reason I ask is that Beast claims to provide a low-level API, so I would assume that users can have complete control over this.

I don't think that's a good assumption. From https://tools.ietf.org/html/rfc6455#section-1.2 "The WebSocket message does not necessarily correspond to a particular network layer framing, as a fragmented message may be coalesced or split by an intermediary." beast::websocket::stream is considered the first intermediary, and it exercises its privilege to reframe message octets. This is established practice, Autobhan websocket implementations also do this; they have a similar auto-fragment feature (which I copied).

...

...

or HTTP chunks?

The "low-level" claims refer to the implementation of the HTTP protocol. It doesn't imply that callers will have control over chunks. Intermediaries along the HTTP path are allowed to rechunk message payloads. Perhaps I have overlooked something. What is the use-case for application level control over chunking?

Does Beast concatenate chunks before passing them to the application layer, or does the application layer get the chunks in the same way as they were received?

The application sees the message body only after the chunked Transfer-Encoding has been removed.

Why does the user not have complete control? Under what conditions will Beast re-packet outgoing chunks?

Beast's implementation does not currently reframe serialized chunks, but there is no specification for how Beast will perform the chunked encoding. This could be changed in the future, if there was a compelling use-case. Thanks

On Tue, Jul 4, 2017 at 10:49 AM, Bjorn Reese via Boost wrote:

That was scary reading, because HTTPbis seemed to be unaware that such deprecation would break some of IETF's own standards, e.g. RFC 3507.

I agree :) That's why I made sure that Beast is capable of sending and receiving chunk-extensions.

Another use-case that is used in practice is for in-band meta-data. ... chunk extensions are used to carry the meta-data.

That makes sense, and that's a supported use-case. You can send the extensions and you can receive them.

Consider a device (e.g. a sensor), that publishes events (e.g. measurements) that are distributed unevenly and unpredictably over time. Each chunk contains a single event, so preserving the chunk boundary is important.

You're talking about a "perpetual message" of some kind, where the end of the message never arrives. I think that's kind of an abuse of HTTP. Isn't Websocket more suited for that? It supports message boundaries which are not rewritten by intermediates (unlike HTTP). With respect to detecting chunk boundaries on input, this is possible, but... I can't guarantee it on output. In fact I'm not even sure how you would convince Beast to keep a message in "perpetually sending" mode since that is a use-case not anticipated by rfc7230. I think the sane way to do what you want is to use Beast for sending the header, which works great, and then take responsibility for outputting the body chunks yourself. True, you'll have to be getting your hands dirty with the protocol details of HTTP but that's already assumed because you 1. want to know about chunks, 2. want to control their output, 3. want to send an infinitely long message. I'm not losing sleep that Beast doesn't make this use-case ridiculously easy.

...

The application sees the message body only after the chunked Transfer-Encoding has been removed. ... Can you elaborate with an example?

Let me rephrase that, beast::http::parser appends chunk bodies into the body container stored in the message (std::string for example).

From the caller's perspective they just see that the message body is growing. There's no guarantee that each increment of growth exactly corresponds to one chunk.

However, if you want to see chunks that is possible by subclassing basic_parser and handling it in the on_chunk callback, which is called after the chunk header is received and gets passed the size of the upcoming body as well as the chunk-extension. I suppose I could easily add an optional user-facing callback to beast::http::parser with the signature to allow users to easily detect chunk boundaries during parsing. It would be a handful of lines of code added here and there (similar to the on_header callback already present): https://github.com/vinniefalco/Beast/blob/78a065ba39836d91d7e70d93de7f9140f5... I haven't done it because I don't like to design theoretical features which are unlikely to be used. I have found that when I do such things, especially for features that I have little experience with (like chunk extensions) I end up discovering later that the interfaces don't satisfy real-world needs and then have to be changed.

Beast should not break the end-to-end principle unless explicitly allowed by the application.

I think if users demand control over outgoing chunks, its reasonable to ask that they manually output the chunked message body; Beast will take care of the header.

Is it possible to provide a HTTP 200 response alias for Beast? ... This is needed for SHOUTcast internet radio servers.

There are no provisions for this. Beast's parser is strict. Responses which don't start with a valid HTTP-version generate an error. That said, I recognize that sometimes in the real world if you want to get in to the club you have to pay the cover. I can add a simple customization point to basic_parser giving the derived class the opportunity to parse the status-line if it wants, to handle this case. However, it won't be for free - I'll kindly ask that the stakeholders (the people who intend to use the feature) participate in the design and code review in order that the result suits their needs. Its not my area of knowledge so if we want to get this right, those with more experience should weigh in. And by the way, thanks for pointing this out! This is exactly the sort of thing that Beast needs in order to mature into something great.

The chunk decorator uses boost::asio::null_buffers to overload the field trailer.

Given your desire to adapt Beast to Networking TS, do notice that null_buffers are not part of N4656 (they have been replaced by dedicated wait functions.)

Good point!! That will have to be addressed when Beast is ported. Right now the chunk decorator is pretty raw. Again, this is a case where a feature has been created with no stakeholders to vet the design. I wrote the minimum amount of interface necessary to make sure Beast supports the feature. With feedback from people actually using it in the field, it can be improved. Thanks

On Tue, Jul 4, 2017 at 4:31 PM, Gavin Lambert via Boost wrote:

The other common pattern (Comet style long poll) is to accept a request and parse the headers but then just keep the connection open indefinitely before finally sending a response.

That's a supported use case, call `write_header` or `async_write_header` on the `serializer` and then at your leisure call `write` or `async_write` on the `serializer` again to finish.

On Thu, Jul 6, 2017 at 12:41 PM, Bjorn Reese via Boost wrote:

When the user passes a buffer to Beast in chunked-mode, then serialization can convert it directly into a chunk. That ought to be very easy to do.

Something like that could be made to work if the caller is using `buffer_body`. That type of body represents caller supplied buffers, there could be a guarantee that each caller provided buffer will be output as a single chunk: https://github.com/vinniefalco/Beast/blob/89c416cde64de3265299ced216a6eef942...

...

you would convince Beast to keep a message in "perpetually sending" mode since that is a use-case not anticipated by rfc7230.

Are you implying that Beast has an upper limit for persistent connections? If not, what are you trying to tell me?

There's no upper limit, but I have not given consideration to a mode where a BodyReader never runs out of buffers. Perhaps it already works without change. But there are no tests for it and I have not tried it, so I cannot make a claim that it works.

...and likewise pass a chunk to the application as it is parsed. This preserves the chunk boundaries without any special facilities.

Beast does not buffer the entire chunk in memory so the application would only see pieces of it. The callback-based solution I provided, is the closest to what you're asking ("pass a chunk to the application").

...

I think if users demand control over outgoing chunks, its reasonable to ask that they manually output the chunked message body; Beast will take care of the header. ... Does this mean that the user has to manually insert chunk-size etc?

In the worst case, yes, but if there is a need for an easier way for users to manually produce chunked output, a simple adapter can be provided. Beast had one which I removed after I made significant improvements to the serialization interfaces, it can be brought back. This function takes a caller provided ConstBufferSequence and lazily transforms it into a new ConstBufferSequence (non-allocating) which performs a chunk-encoding: https://github.com/vinniefalco/Beast/blob/1e3543f63ed46bb1b2e698939f3cf0b132... Thanks

On 07/02/2017 08:11 PM, Vinnie Falco via Boost wrote:

To my understanding, chunk-extensions are a rare niche use-case with meaning only to applications using a custom interpretation at each end of the connection. In fact 5 years ago the IETF almost deprecated them: https://trac.ietf.org/trac/httpbis/ticket/343

That was scary reading, because HTTPbis seemed to be unaware that such deprecation would break some of IETF's own standards, e.g. RFC 3507.

On the other hand, I do not have significant expertise with HTTP servers; if a compelling use-case presents itself this is an aspect of the library which may be improved, in a backward-compatible way.

Chunk extensions were originally designed for per-chunk signatures. I do not know how extensively this is used. Another use-case that is used in practice is for in-band meta-data. Consider an Internet radio station that sends a constant stream of audio. When a new track is played this will be signaled by meta-data telling the track title, artist name, etc. Some audio codec formats embed this meta-data into the stream itself (e.g. MP3 ID3 tags), while others do not. In the latter case, chunk extensions are used to carry the meta-data.