Is the async_read more reliable than async_read_some?
Hi, I have been using asio::async_read with protobuf for some time, it is relative easy to use protobuf function SerializeToOstream to attach a packet header and send the packet header along with to protobuf message, with fixed packet header size and message size in the header, I was able to use async_read. But I have to change to use msgpack which I could not find a way like protobuf to attach a packet header to msgpack message. The only way I could do is to use async_read_some, but I am very concerning the remark in the async_read_some document. Appreciate your comments. "The read operation may not read all of the requested number of bytes. Consider using the async_read function if you need to ensure that the requested amount of data is read before the asynchronous operation completes." Thank you. - JH
AMDG On 12/07/2018 02:44 AM, hh h via Boost wrote:
I have been using asio::async_read with protobuf for some time, it is relative easy to use protobuf function SerializeToOstream to attach a packet header and send the packet header along with to protobuf message, with fixed packet header size and message size in the header, I was able to use async_read.
But I have to change to use msgpack which I could not find a way like protobuf to attach a packet header to msgpack message. The only way I could do is to use async_read_some, but I am very concerning the remark in the async_read_some document. Appreciate your comments.
How do you identify the end of the message? Can you use one of the overloads of async_read that takes a completion_condition?
"The read operation may not read all of the requested number of bytes. Consider using the async_read function if you need to ensure that the requested amount of data is read before the asynchronous operation completes."
aynyc_read is essentially implemented by calling async_read_some repeatedly, until it fills the requested size. In Christ, Steven Watanabe
aynyc_read is essentially implemented by calling async_read_some repeatedly, until it fills the requested size.
That is exactly the question I am going to ask, if you don't know the package size to read, you have to call sync_read_some with max_buffer_size, how does the sync_read_some function know the end of the message? Obviously, that the end of the message length is smaller than the max_buffer_size. Is some timeout defined in sync_read_some to identify the end of the message? Thanks Steven On 12/8/18, Steven Watanabe via Boost <boost@lists.boost.org> wrote:
AMDG
On 12/07/2018 02:44 AM, hh h via Boost wrote:
I have been using asio::async_read with protobuf for some time, it is relative easy to use protobuf function SerializeToOstream to attach a packet header and send the packet header along with to protobuf message, with fixed packet header size and message size in the header, I was able to use async_read.
But I have to change to use msgpack which I could not find a way like protobuf to attach a packet header to msgpack message. The only way I could do is to use async_read_some, but I am very concerning the remark in the async_read_some document. Appreciate your comments.
How do you identify the end of the message? Can you use one of the overloads of async_read that takes a completion_condition?
"The read operation may not read all of the requested number of bytes. Consider using the async_read function if you need to ensure that the requested amount of data is read before the asynchronous operation completes."
aynyc_read is essentially implemented by calling async_read_some repeatedly, until it fills the requested size.
In Christ, Steven Watanabe
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
AMDG On 12/07/2018 05:05 PM, hh h wrote:
aynyc_read is essentially implemented by calling async_read_some repeatedly, until it fills the requested size.
That is exactly the question I am going to ask, if you don't know the package size to read, you have to call sync_read_some with max_buffer_size, how does the sync_read_some function know the end of the message? Obviously, that the end of the message length is smaller than the max_buffer_size. Is some timeout defined in sync_read_some to identify the end of the message?
The way to identify the end of the message depends on how the end of the message is represented. If it uses some kind of delimiter, then async_read_until will work. If you signal the end of the message by closing the connection, then you can just read until EOF (which is indicated by the error_code). In Christ, Steven Watanabe
"If it uses some kind of delimiter, then async_read_until will work" What kind of delimiter can I use to make async_read_until will work? The server to call async_read_some will be connected by thousands devices, can the async_read_some cope with thousands different messages without causing problems? I've just tried to run async_write using boost streambuf to pack packet header and msgpack data together in one message data, but the async_read in server site could only work a couple of minutes than it crashed even I fixed all sizes in header file. It seems I can only use the async_read_some for a stable server implementation. On 12/8/18, Steven Watanabe via Boost <boost@lists.boost.org> wrote:
AMDG
On 12/07/2018 05:05 PM, hh h wrote:
aynyc_read is essentially implemented by calling async_read_some repeatedly, until it fills the requested size.
That is exactly the question I am going to ask, if you don't know the package size to read, you have to call sync_read_some with max_buffer_size, how does the sync_read_some function know the end of the message? Obviously, that the end of the message length is smaller than the max_buffer_size. Is some timeout defined in sync_read_some to identify the end of the message?
The way to identify the end of the message depends on how the end of the message is represented. If it uses some kind of delimiter, then async_read_until will work. If you signal the end of the message by closing the connection, then you can just read until EOF (which is indicated by the error_code).
In Christ, Steven Watanabe
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
AMDG On 12/08/2018 04:51 PM, hh h wrote:
"If it uses some kind of delimiter, then async_read_until will work"
What kind of delimiter can I use to make async_read_until will work?
I have no idea. If you don't know already then your protocol is probably not designed to work that way. If you're designing your own protocol, then sending the size first is probably simpler to implement.
The server to call async_read_some will be connected by thousands devices, can the async_read_some cope with thousands different messages without causing problems?
I've just tried to run async_write using boost streambuf to pack packet header and msgpack data together in one message data, but the async_read in server site could only work a couple of minutes than it crashed even I fixed all sizes in header file. It seems I can only use the async_read_some for a stable server implementation.
How does it crash? There's no fundamental reason why prepending your own header with the size wouldn't work. The only likely issue that I see with this setup is that async_read will receive the entire message and buffer it in memory before it signals completion, which might be a problem if the messages are large. In Christ, Steven Watanabe
How does it crash? There's no fundamental reason why prepending your own header with the size wouldn't work.
I sent 4 bytes header and 221 bytes messages repeatedly (every 2 seconds) calling from async_write of a sender, the receiver works correctly and perfectly at async_read for a couple minutes, then it crashed in the statement of readMessageBody for 221 bytes message at async_read(socket_, boost::asio::buffer(buffer, size), ....), where the size = 221n I don't think it is related to my program, either some bugs in async_read or libssl. Here is my debug: Read body size = 221 Program received signal SIGSEGV, Segmentation fault. 0x00007ffff69d4c01 in __GI___libc_free (mem=0x5555557f7770) at malloc.c:3123 3123 malloc.c: No such file or directory. (gdb) backtrace #0 0x00007ffff69d4c01 in __GI___libc_free (mem=0x5555557f7770) at malloc.c:3123 #1 0x00007ffff796a8fc in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #2 0x00007ffff796a139 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #3 0x00007ffff796f9aa in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #4 0x00007ffff79797d5 in SSL_read () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #5 0x00005555555665b1 in boost::asio::ssl::detail::engine::do_read ( this=0x5555557d0fe0, data=0x5555557f7765, length=221) at /usr/include/boost/asio/ssl/detail/impl/engine.ipp:309 ........ (gdb) print (char *)mem $2 = 0x5555557f7770 "n\226" Thanks Steven and Cliff. On 12/9/18, Steven Watanabe via Boost <boost@lists.boost.org> wrote:
AMDG
On 12/08/2018 04:51 PM, hh h wrote:
"If it uses some kind of delimiter, then async_read_until will work"
What kind of delimiter can I use to make async_read_until will work?
I have no idea. If you don't know already then your protocol is probably not designed to work that way. If you're designing your own protocol, then sending the size first is probably simpler to implement.
The server to call async_read_some will be connected by thousands devices, can the async_read_some cope with thousands different messages without causing problems?
I've just tried to run async_write using boost streambuf to pack packet header and msgpack data together in one message data, but the async_read in server site could only work a couple of minutes than it crashed even I fixed all sizes in header file. It seems I can only use the async_read_some for a stable server implementation.
How does it crash? There's no fundamental reason why prepending your own header with the size wouldn't work. The only likely issue that I see with this setup is that async_read will receive the entire message and buffer it in memory before it signals completion, which might be a problem if the messages are large.
In Christ, Steven Watanabe
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
AMDG On 12/08/2018 06:50 PM, hh h wrote:
How does it crash? There's no fundamental reason why prepending your own header with the size wouldn't work.
I sent 4 bytes header and 221 bytes messages repeatedly (every 2 seconds) calling from async_write of a sender, the receiver works correctly and perfectly at async_read for a couple minutes, then it crashed in the statement of readMessageBody for 221 bytes message at async_read(socket_, boost::asio::buffer(buffer, size), ....), where the size = 221n I don't think it is related to my program, either some bugs in async_read or libssl. Here is my debug:
Read body size = 221
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff69d4c01 in __GI___libc_free (mem=0x5555557f7770) at malloc.c:3123 3123 malloc.c: No such file or directory.
(gdb) backtrace #0 0x00007ffff69d4c01 in __GI___libc_free (mem=0x5555557f7770) at malloc.c:3123 #1 0x00007ffff796a8fc in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #2 0x00007ffff796a139 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #3 0x00007ffff796f9aa in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #4 0x00007ffff79797d5 in SSL_read () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #5 0x00005555555665b1 in boost::asio::ssl::detail::engine::do_read ( this=0x5555557d0fe0, data=0x5555557f7765, length=221) at /usr/include/boost/asio/ssl/detail/impl/engine.ipp:309
........
(gdb) print (char *)mem $2 = 0x5555557f7770 "n\226"
This looks like some kind of memory corruption, possibly a double free. Have you tried valgrind or addrsan? In Christ, Steven Watanabe
Yes, I am doing the debug to further chasing it down... Thanks Steven. On 12/9/18, Steven Watanabe via Boost <boost@lists.boost.org> wrote:
AMDG
On 12/08/2018 06:50 PM, hh h wrote:
How does it crash? There's no fundamental reason why prepending your own header with the size wouldn't work.
I sent 4 bytes header and 221 bytes messages repeatedly (every 2 seconds) calling from async_write of a sender, the receiver works correctly and perfectly at async_read for a couple minutes, then it crashed in the statement of readMessageBody for 221 bytes message at async_read(socket_, boost::asio::buffer(buffer, size), ....), where the size = 221n I don't think it is related to my program, either some bugs in async_read or libssl. Here is my debug:
Read body size = 221
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff69d4c01 in __GI___libc_free (mem=0x5555557f7770) at malloc.c:3123 3123 malloc.c: No such file or directory.
(gdb) backtrace #0 0x00007ffff69d4c01 in __GI___libc_free (mem=0x5555557f7770) at malloc.c:3123 #1 0x00007ffff796a8fc in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #2 0x00007ffff796a139 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #3 0x00007ffff796f9aa in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #4 0x00007ffff79797d5 in SSL_read () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #5 0x00005555555665b1 in boost::asio::ssl::detail::engine::do_read ( this=0x5555557d0fe0, data=0x5555557f7765, length=221) at /usr/include/boost/asio/ssl/detail/impl/engine.ipp:309
........
(gdb) print (char *)mem $2 = 0x5555557f7770 "n\226"
This looks like some kind of memory corruption, possibly a double free. Have you tried valgrind or addrsan?
In Christ, Steven Watanabe
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
What kind of delimiter can I use to make async_read_until will work?
The server to call async_read_some will be connected by thousands devices, can the async_read_some cope with thousands different messages without causing problems?
I've just tried to run async_write using boost streambuf to pack packet header and msgpack data together in one message data, but the async_read in server site could only work a couple of minutes than it crashed even I fixed all sizes in header file. It seems I can only use the async_read_some for a stable server implementation.
Describe your wire protocol, then we (those reading this message thread) can give you some ideas. There are many ways to do the "message framing" (i.e. determining the beginning and end of a message), but it all depends on how the wire protocol is defined. I assume you know that TCP / IP is a stream oriented protocol. There is no direct correlation between how many bytes you send and how many (at one time) will be received on the other end. That's why Asio provides many different ways to consume the data on read. Cliff
participants (3)
-
Cliff Green -
hh h -
Steven Watanabe