Re: [boost] C++ Deflate (zlib-like) library

On Tue, Mar 10, 2020, 4:15 AM Vinnie Falco via Boost wrote:

On Mon, Mar 9, 2020 at 7:30 PM Gavin Lambert via Boost wrote:

...

That answer may be different depending on your context; for example Boost.Asio provides a number of buffer concepts and methods for manipulating them.

Which are good, in that they're templates and support all kinds of different buffer concepts (from basic vectors and strings through to dynamic multi-buffer streams), and you only pay for the complexity that you actually use.

They're also bad, in that they're templates and they're harder to consume generically without forcing your own code to also be a template.

But if you're in the context of Boost.Beast or some other app that uses Asio, it makes sense to use these buffer concepts rather than reinventing the wheel, especially with these things probably-maybe-eventually landing in the standard.

Yes now we're getting to the meat of it! You are proposing: Use Asio's BufferSequence concepts (or something substantially similar) in the high level ZLib interface (or a generic codec interface).

On the face of it this is not a bad idea, but the reality is that all codecs work a single buffer at a time. This is different from Asio or OS-level I/O, which support a scatter/gather interface where the buffers are all sent together. Thus, it seems to be that at the lowest level of a codec interface, it will be single-buffer oriented.

On top of that single-buffer interface, algorithms which work on a buffer sequence or range of buffers can be built. Asio's buffer sequence concepts are tragically connected to the asio::const_buffer and asio::mutable_buffer concrete types, which makes it necessary to have all of Asio as a dependency. Using these types out of the box for a low-level ZLib / codecc library doesn't seem like a great idea.

I did propose that Asio split out its buffer sequence concepts and types into a separate Boost library. This would allow Beast's HTTP parser to be in its own library, without requiring Asio. The experience with the large Beast library makes me now more in favor of smaller, more numerous libraries with fewer dependencies.

Perhaps something like this could be an improvement on ZLib's C API:

struct input_buffer { input_buffer( void const* data, std::size_t size);

bool is_empty() const noexcept; void const* data() const noexcept; std::size_t used() const noexcept; std::size_t remaining() const noexcept; void consume(std::size_t) noexcept; };

First questions regarding your API - why do we need all those functions? You compress/decompress in chunks so all you need is a const std::span for input and std::span for output and call decompress on a for loop. Since compression/decompression is not random access but streaming you can alternatively just accept an input iterator pair/range for input/output and cover tue streaming case without a for loop. With std::istream_iterator you can then handle std::stream (file, memory etc.) support. You can also chain for e.g. boost.spirit x3 parser and lazy read->decompress->parse/process Can't existing asio buffers be written to through span/iterators? struct output_buffer; // similar to input_buffer

Then we can express a compression function using these types, and get better safety:

// returns the number of bytes written to the output buffer std::size_t compress( output_buffer& out, input_buffer& in );

Whit iterators/ranges described above you wouldn't need to return std::size_t so return can be used for error codes instead. Regards, Domen A function to compress a ConstBufferSequence into a

MutableBufferSequence could be built from this one primitive, and a dependency on including

Thanks

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost