[Asio] Making custom std::streambuf for async_read
At the moment I'm using async_read with boost::asio::streambuf and this works fine but I need a custom stream where every invoke to async_read appends the data to my stream. The stream should be based on std::streambuf because I want to keep the asio interface hidden to the user. My first idea was to search for the problem and I ended up with reading about MutableBufferSequence but I don't know if this is the right way and how to implement this in my streambuf object. Any help is higly appreciated.
On 9/08/2016 17:13, knarks@online.de wrote:
At the moment I'm using async_read with boost::asio::streambuf and this works fine but I need a custom stream where every invoke to async_read appends the data to my stream.
Asio streambuf already does this, so I'm not sure why you'd want to make a custom one on that basis.
The stream should be based on std::streambuf because I want to keep the asio interface hidden to the user.
While I understand the motivation, note that some of the performance of Asio relies on the compiler being able to inline things. There's certainly nothing stopping you from hiding it but you'll lose some of the benefits this way. Some of the special features such as strands and custom allocators also rely on the way that the templates interact. And Asio's streambuf is already based on std::streambuf; to make a custom one you'd end up completely duplicating it anyway.
My first idea was to search for the problem and I ended up with reading about MutableBufferSequence but I don't know if this is the right way and how to implement this in my streambuf object. Any help is higly appreciated.
Asio does support using many other data sequences as buffers, so you can just use something else other than streambuf if you wish to. http://www.boost.org/doc/libs/1_61_0/doc/html/boost_asio/overview/core/buffe...
Thank you for the response so far. My idea is to prevent unnecessary copy operations when receiving large files with tcp. For this reason I thought about implementing my own stream, which writes direct to file or memory, depending on the situation. I know that I can connect std::iostream to the asio streambuf but what about moving the received memory to a std::ofstream or std::stringstream without having the data twice in memory. Is that possible? Maybe I'm concerned too much about that problem and a simple copy operation to the std stream works in most cases.
On 9/08/2016 17:13, knarks@online.de wrote:
At the moment I'm using async_read with boost::asio::streambuf and this works fine but I need a custom stream where every invoke to async_read appends the data to my stream.
Asio streambuf already does this, so I'm not sure why you'd want to make a custom one on that basis.
The stream should be based on std::streambuf because I want to keep the asio interface hidden to the user.
While I understand the motivation, note that some of the performance of Asio relies on the compiler being able to inline things. There's certainly nothing stopping you from hiding it but you'll lose some of the benefits this way. Some of the special features such as strands and custom allocators also rely on the way that the templates interact.
And Asio's streambuf is already based on std::streambuf; to make a custom one you'd end up completely duplicating it anyway.
My first idea was to search for the problem and I ended up with reading about MutableBufferSequence but I don't know if this is the right way and how to implement this in my streambuf object. Any help is higly appreciated.
Asio does support using many other data sequences as buffers, so you can just use something else other than streambuf if you wish to.
http://www.boost.org/doc/libs/1_61_0/doc/html/boost_asio/overview/core/buffe...
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On 11/08/2016 07:16, knarks@online.de wrote:
Thank you for the response so far. My idea is to prevent unnecessary copy operations when receiving large files with tcp. For this reason I thought about implementing my own stream, which writes direct to file or memory, depending on the situation.
I know that I can connect std::iostream to the asio streambuf but what about moving the received memory to a std::ofstream or std::stringstream without having the data twice in memory. Is that possible?
ASIO does provide posix::stream_descriptor and windows::stream_handle, which can wrap native file descriptors and handles (resp.) and provide async operations on them. Thus you can read from the network into one asio streambuf and then pass the same streambuf to the file write operation. This is probably as close to zero-copy as you can get. Though other than scalability and blocking concerns there's not much difference between this and just doing a blocking write to a regular fstream -- at the point you're hitting disk the cost of copying memory is negligible. For writing direct to memory, you can simply supply the target memory as a buffer to the initial read (making sure to keep track of the origin and size correctly as data blocks incrementally arrive, remembering that short reads are likely). This will quite literally be zero-copy (bar any copies that occur inside the network stack); no special streams required.
ASIO does provide posix::stream_descriptor and windows::stream_handle, which can wrap native file descriptors and handles (resp.) and provide async operations on them. Thus you can read from the network into one asio streambuf and then pass the same streambuf to the file write operation. This is probably as close to zero-copy as you can get.
Though other than scalability and blocking concerns there's not much difference between this and just doing a blocking write to a regular fstream -- at the point you're hitting disk the cost of copying memory is negligible.
For writing direct to memory, you can simply supply the target memory as a buffer to the initial read (making sure to keep track of the origin and size correctly as data blocks incrementally arrive, remembering that short reads are likely). This will quite literally be zero-copy (bar any copies that occur inside the network stack); no special streams required.
I'm using my library for HTTP handling and receiving HTTP POST requests. When the transfer is chunked then every call to the complete handler can append a small bit of the stream to a stringstream or a file with the standard c++ features. This would be the most portable way than using stream_descriptor or stream_handle.
participants (2)
-
Gavin Lambert -
knarks＠online.de