I meant I read data multiple times (accumulating incoming data using std::string::append(charBuffer) ) until readBytes >= BUFFER_SIZE which, sometimes, happens to interrupt (read_some() returns value less then BUFFER_SIZE when not all 8004 bytes was read). Now everything is clear to me. Thanks everyone!

Then messages of length not multiple of 128 (BUFFER_SIZE) will not be read - tested it, anyway, after last socket::read_some() (with readBytes > 0 && < BUFFER_SIZE) next socket::read_some() never ends.

On 05/11/2011 09:36 AM, Andrew Holden wrote:

On Wednesday, May 11, 2011 9:57 AM, Slav wrote:

...

Then messages of length not multiple of 128 (BUFFER_SIZE) will not be read - tested it, anyway, after last socket::read_some() (with readBytes> 0&& < BUFFER_SIZE) next socket::read_some() never ends.

It would probably help to understand that TCP has no concept of a "message". Anything you write to a socket is appended to a stream of *bytes*.

Alternatively, we could say that TCP, in fact, does have a well-defined concept of messages: they are all exactly one byte long.

Several subsystems on both the sending and receiving computers

... and sometimes boxes in the middle ...

will have the option of splitting and combining adjacent buffers with no consideration to how big each individual write was.

I've done a little protocol stuff with ASIO now and I must say it's a lot of fun and I can't go back to doing it any other way.

On further thought, I think I see the problem (and I apologize for the bad recommendation in my last email). Your sender somehow needs to communicate the message size or flag the end of the message. A partial list of options includes:

The pattern I encounter over and over again (often at multiple levels in a protocol) is: class protocol_layer_context { vector<uint8> buffer; void on_received_data(vector<uint8> & rx_bytes) { buffer.append(rx_bytes); // perhaps a virtual override. size_t msg_len = this->parse_len_from_start_of_buffer(); if (buffer.size() <= msg_len) { vector<uint8> msg_buf = consume_data_from_front_of_buffer(buffer, msg_len); // perhaps a virtual override this->process_complete_message(msg_buf); } // post another ASIO read request this->request_more_data(); } ... But there are some important issues with this naive pseudocode: 1. It can result in recopying the data a bunch of times for every protocol layer, killing performance. 2. It's susceptible to a denial-of-service (DoS). Bad guy can send trick you into allocating all your memory. 3. Sometimes the length of a message is stated at the beginning of the message, sometimes it isn't known until the end. 4. No processing happens on the message until it's completely read, but some protocols really need the receiving endpoint to process it incrementally. 5. Error handling 6. Optimal threading 7. Etc. We find bugs in exactly this logic all the darn time. Often the data being received is untrusted and possibly malicious. Real-world protocol implementations will commonly crash under fragmentation fuzzing, sometimes resulting in exploitable security holes. In a sense, this is the general refactoring problem of 'incrementalizing' a parsing function by moving all its state from stack variables into a longer-lived context object. We've seen it done successfully with coroutines, but that's not a commonly accepted solution because, frankly, the native C/C++ runtimes have not yet given coroutines the love (i.e., portability and performance guarantees) they really deserved. If someone figured out how to leverage generic techniques to handle just the unidirectional message delimiting problem in a bulletproof way I think it would make a really great boost library. - Marsh

As I raised this question I thought that TCP itself has the ability to determin the message size - but it does not (I previously widely used ENet (secure data transmission based on UDP) - may be that's why I was misleaded). Important usage "application protocol" build upon "message size prefix" protocol (if such a small thing can be called "protocol") could be the C++ objects' serialization using the boost::serialization, so the whole picture of protocol stack would looks like this: Ethernet -> IP -> TCP -> "message size prefix" -> "boost::serialization"

On 05/12/2011 04:28 PM, Jerry wrote:

Ted Byers writes:

...

It seems to me to be an obvious thing to do, but my question to you is "Do you know of anyone who has done it?" (in some kind of open source project)

What about: http://cpp-netlib.github.com/

Looks interesting. This page in particular looks like it's getting close to what I was talking about: http://cpp-netlib.github.com/0.9.0/message.html I realize the project is new and the docs may not be complete, but every other page seems to be about its HTTP implementation. Even the generic basic_message class presumes a headers/body structure. HTTP is often thought of as a half-duplex message/response protocol because it (mostly) stateless and originally closed the connection after every response. I was interested more in a general facility for a common low-level protocol buffering pattern.

Apparently it will be submitted for review someday: http://comments.gmane.org/gmane.comp.lib.boost.user/67431

Cool. - Marsh

On 05/12/2011 04:59 PM, Ted Byers wrote:

what I haven't found, yet, is a way to compare the IP info in the TCP packest with the IP info in the HTTP headers.

Sometimes a proxy will add something, but usually there aren't any IP addresses in HTTP headers.

That is in particualr. Mre generally, I am looking for an online resource for learning network programming in general and security related network proramming in particular.

That's interesting. There are resources about secure programming, and securing networks, but I don't see much new stuff about basic network programming. They are probably casualties of the trend to make all communications run over HTTP(s). I don't recall ever seeing a book or online resource saying "here's how to accept data from the network and process it in the most scalable and secure way using C or C++". On the crypto side of things I recommend:

http://www.amazon.com/Cryptography-Engineering-Principles-Practical-Applicat...

I tweeted your question: https://twitter.com/marshray/status/68810041234432000 Got this recommendation, doesn't seem to be too related to network programming though. Perhaps we'll get more.

http://www.amazon.com/Memory-Programming-Concept-Frantisek-Franek/dp/0521520...

- Marsh

It must be reflected in documentation. Isn't it?

"The read_some operation may not read all of the requested number of bytes." is pretty clear because incoming message isn't bound to the BUFFER_SIZE which is just the place where to srore intermediate data. "Consider using the read function if you need to ensure that the requested amount of data is read before the blocking operation completes." sounds like "read until my buffer will be fulled regardless of incoming message length" - not like "read until the logical message will be fully read and use my buffer as something where data will be put - regargingly it's size". And, by the way, there could come empty messages - just without any data. TCP contains info about message length - it is duplication to prefix all messages with it's length.

On Wednesday, May 11, 2011 11:12 AM, Slav wrote:

And, by the way, there could come empty messages - just without any data.

TCP does not have a concept of an empty write.

TCP contains info about message length - it is duplication to prefix all messages with it's length.

Where did you read that? TCP has NO concept of "messages". As such, it has no concept of "message length". It is only a stream of bytes. The sending computer can easily combine the buffers from two consecutive write calls into a single packet, or split the buffer from a single write call into multiple packets, or both. In either case, ALL information about the size of the original write call(s), the number of write calls, and anything else that you hope will provide a clue about "messages" will be lost. Likewise, the receiving computer can and will freely combine and split packets into whatever buffers it sees fit, with similar effects on any "message boundaries". The only thing that will remain is the sequence of bytes. Do not try to search for TCP options to change this behavior. The closest you can come is options that will *usually* keep the message boundaries. This means that your program will *usually* not crash. If you wish to preserve message boundaries, then you MUST provide your own message framing, just as you would if writing to a file. If you prefix each message with its length, then you can use the read function to ensure you get the whole message in one call, as you will know the message length. This will also be effective at ensuring you don't have the beginning of the next message at the end of your buffer.

On Thursday, May 12, 2011 10:36 AM, Slav wrote:

Yeah - I was really mistaken. Thanks for correcting me.

I reimplemented the reading using message length prefix and now everything works fine.

Left just one question: socket has "receive_buffer_size" option which is by default equals 8192 - does it mean that message of length (if it will come at once) will be truncated? Or I still could read it with multiple "async_read" calles (collecting it into a buffer using data length prefix)?

TCP will not alter the byte stream, also meaning it will not drop bytes. If the receive buffer fills, then it will tell the other machine it is sending data too fast and needs to slow down. It will also have the other machine resend the data that couldn't fit in the receive buffer. Your programs (on both ends) will not need to address this issue; the operating system will handle it for you. That said, experimenting with the receive buffer size may improve performance, but will have no effect on correctness. Don't assume more is better. If you make the buffers too big, you'll just increase overhead and latency.