Re: [boost] Boost.Buffers: network-agnostic buffer types and algorithms

pt., 3 lut 2023 o 02:02 Vinnie Falco napisał(a):

On Thu, Feb 2, 2023 at 1:58 PM Andrzej Krzemienski wrote:

...

Today the basics for networking seems like good candidates for vocabulary types. Is Boost.Buffer a candidate for such a vocabulary type?

In theory, yes. The intent is for Buffers to be more general than just networking.

...

First question, is this library to be consumed directly by user programs, or only by other libraries, such as Boost.Beast and Boost.Http.Proto?

User programs could use it when they want to express algorithms in terms of ranges of buffers. My personal use-case is Beast, HTTP, Websocket, and JSON but this could be extended to say, base64 encoding and decoding, calculating SHA-2 digest over a range of buffers, simplified front end to things like zlib. Example:

template< class DynamicBufferOut, class DynamicBufferIn > result< void > deflate( z_params&, DynamicBufferOut& DynamicBufferIn&, bool end_of_data);

...

Is it possible that the final program that uses Boost.Buffer will not use Boost.ASIO?

That is possible, yes.

...

For me, as a consumer of networking libraries, who can afford to download and build the entire set of Boost libraries, the biggest advantage would be that Buffers would be well documented and tested.

That is an advantage yes but in my opinion the advantage is that you get access to a rich and useful set of algorithms, containers, and concepts, without also depending on a particular network implementation.

...

I am not convinced by the argument with Boost.JSON not depending on ASIO.

Well, users definitely do not want to drag in Asio with their JSON.

...

We have (I think a still unresolved issue) of Boost-serializing Boost.Optional. Should the code responsible for the serialization of Boost.Optional be part of Boost.Serialization or Boost.Optional? But we will certainly not solve it by introducing yet another library.

I think that the cases of Boost.Serialize with boost::optional, and serialization of JSON using Boost.JSON and Boost.Buffers are two completely different cases. Boost.Serialize does not define a serialization format but rather a framework for allowing the user to define serialization. Boost.JSON is something else entirely; it is a library which implements a specific protocol for describing data using structured text.

If you look at Boost.JSON you see that it has json::parser and json::serializer. These classes work a buffer at a time and they support streaming; that is, that the entire input or output need not be presented at once in order to make progress. How does Boost.Buffers help here? Well.. it provides the means for a library such as JSON, which has algorithms to produce or consume a buffer at a time, to type-erase its algorithm using the buffers::source or buffers::sink interfaces:

/// An algorithm which consumes filled buffers struct sink { virtual ~sink() = 0;

/** Called when data is available */ virtual result<void> write( span< const_buffer const > ) = 0; };

/// An algorithm which produces filled buffers struct source { /** The results of reading from the source. */ struct results { error_code ec; std::size_t bytes = 0; bool more = false; };

virtual ~source() = 0;

/** Called when more data is required

This function is invoked when more data is required. The subclass should place zero or more bytes into the buffers referenced by `dest`, and return a `results` value with the members set appropriately. <br> Partial success is possible. */ virtual results read( span< mutable_buffer const > dest ) = 0; };

This is not really any different from libraries that support operator<<(std::ostream&), except that the vocabulary type is buffers::sink and buffers::source instead of std::istream and std::ostream. When Boost.JSON implements these APIs, it is not "supporting Boost.Serialization" it is merely providing a wrapper for the algorithms it already has (json::parser and json::serializer) to adapt them to a common, agreed-upon interface (the buffers::sink and the buffers::source).

Currently users complain that Boost is a huge bundle of libraries, and at some level this is true. But it is a bundle of libraries for a reason, because releasing the collection together as a unit allows us to ensure that all the components work together. But more importantly it allows a member library to leverage the advantages of having other member libraries available to it (Config, Variant2, mp11 come to mind). The point of implementing Boost.Buffers' sink and source interfaces in JSON is to let us leverage the benefits of the monolithic distribution model.

...

Boost.JSON doesn't need to depend on the implementation of the circular buffer.

Right, it wouldn't. It would only be implementing the sink and source abstract interfaces. This let's JSON participate in the growing ecosystem of Boost.Buffers enabled algorithms. I think JSON is the place for it, because the model can scale. Candidate libraries which would add value by implementing source and sink do so in the library itself. The model where a single library has to depend on an ever increasing number of other libraries in order to implement their wrappers is unsustainable.

Thank you. I am convinced that extracting the Boost.Buffers library in the way you described is a good idea. Regards, &rzej;

-- Regards, Vinnie

Follow me on GitHub: https://github.com/vinniefalco