On 18/07/2024 21:58, Christian Mazakas via Boost wrote:
Actually, it might be nice to compare implementation notes. For Fiona, I create a buffer sequence for the user by over-allocating for each buffer and making an intrusive doubly-linked list. This means that the user gets a mutable buffer sequence without the need for any intermediate allocations.
Instead of over-allocating and wasting a page, I would put the link pointers at the end and slightly reduce the maximum size of the i/o buffer. This kinda is annoying to look at because the max buffer fill is no longer a power of two, but in terms of efficiency it's the right call.
It also enables nice constant time push/pop and slicing. I was heavily inspired by Rust and Asio when coming up with the buffer abstraction: https://github.com/cmazakas/fiona/blob/main/test/buffer_test.cpp
Surely for reading you want io_uring to tell you the buffers, and when you're done, you immediately push them back to io_uring? So no need to keep buffer lists except for the write buffers? There is a further optimisation trick for writes: if you allocate the write buffers out of a memory mapped anonymous inode, you can use sendfile offload. This isn't actually always a win surprisingly enough, if your write buffers are big then it's a big win, but if they're small, it usually isn't. I try to aim for write buffers of 8Mb and I try to fill them to max before sending them. Re: scatter-gather buffers, generally if somebody is doing more than one buffer at a time, they're working with wider data structures in the program that usually don't need to be in registered i/o buffers i.e. locked memory. I've found dropping support for registered i/o buffers if there is more than one struct iovec is generally pain free.
If you'd like, I'd really appreciate any implementation feedback on what's going on here. Not many seem to have the expertise in io_uring and this kind of stuff so it's rare when I get to really talk shop.
I can definitely agree on that!
Maybe I need to actually sit down and write some docs because I realize there's a lot to the codebase here and it's hard to convey everything just using email.
Generally if you've made these things before, the concepts and patterns all are familiar and it's easy to understand each other. If you've never done it before, yeah it can be impenetrable. Niall