Re: [Boost-users] [asio] Sending large packages

Nicola Michael Gutberlet

4 Jan 2010 4 Jan '10

9:53 a.m.

Doesn't anyone have any ideas? Did anyone receive this mail? Best regards and a happy new year ;-) _____ From: Nicola Michael Gutberlet [mailto:nicola.gutberlet@hhi.fraunhofer.de] Sent: 15 December 2009 17:28 To: 'boost-users@lists.boost.org' Subject: RE: [asio] Sending large packages Some testing showed that the maximum package (load) size of boost::asio::write/read() for a TCP socket seems to be 1007 kB _____ From: Nicola Michael Gutberlet [mailto:nicola.gutberlet@hhi.fraunhofer.de] Sent: 15 December 2009 17:11 To: 'boost-users@lists.boost.org' Subject: [asio] Sending large packages Dear all, Is there a well-defined way of sending large packages over TCP? Or do I have to split the packages on my own? For explanation: I want to send images (size around 6MB). Best regards, Nicola

Attachments:

attachment.html (text/html — 5.9 KB)

Show replies by date

Igor R

4 Jan 4 Jan

10:41 a.m.

New subject: [asio] Sending large packages

...

Doesn’t anyone have any ideas? Did anyone receive this mail?

Best regards and a happy new year ;-)

________________________________

From: Nicola Michael Gutberlet [mailto:nicola.gutberlet@hhi.fraunhofer.de] Sent: 15 December 2009 17:28 To: 'boost-users@lists.boost.org' Subject: RE: [asio] Sending large packages

Some testing showed that the maximum package (load) size of boost::asio::write/read() for a TCP socket seems to be 1007 kB

I guess it's not an asio limitation, but the OS one - you just exceeded the maximum message size for the underlying protocol. On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.

Jason Felice

3:30 p.m.

New subject: [asio] Sending large packages

On Mon, Jan 4, 2010 at 5:41 AM, Igor R wrote:

...

...
Doesn’t anyone have any ideas? Did anyone receive this mail?

I guess it's not an asio limitation, but the OS one - you just exceeded the maximum message size for the underlying protocol. On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.

I've several times encountered a faulty assumption about how TCP works, actually in multiple code bases I've maintained. I'm not sure if this might be the same faulty assumption, but it sounds like it might be. TCP networking is a stream-based protocol and it makes NO guarantees that the receiving end's read sizes are coincident with the sending send's write sizes. In fact, it has a few mechanisms which will actively thwart this, including Nagle's algorithm (see http://en.wikipedia.org/wiki/Nagle%27s_algorithm ). All TCP networks have an MTU, and packets larger than this will be sliced up for delivery (without affecting the data contained within or the guaranteed order of delivery). There is some handshaking to discover the minimum MTU along a route, though this is intended to allow client systems to adapt for maximum throughput, it is not a guarantee that slicing won't happen. It's more like a guarantee that it won't happen frequently. Additionally, loaded routers may well choose to split transmit units or collect and concatenate them in order to meet quality of service and throughput requirements. Unfortunately, though, making this assumption will apparently work often enough with small packet sizes to lull some developers into believing they've confirmed that read and write sizes are coincident. IF you are making this assumption, you need to redesign your wire protocol to include within it delineation of the data, such as sending the data size up front, or using some sort of demarcation or end-of-data marker. You then need to collect the reads on the receiving side until you've reached the demarcation or expected data size.

Oliver Seiler

5:05 p.m.

New subject: [asio] Sending large packages

On Mon, Jan 4, 2010 at 7:30 AM, Jason Felice wrote:

...

On Mon, Jan 4, 2010 at 5:41 AM, Igor R wrote:

...
...
Doesn’t anyone have any ideas? Did anyone receive this mail?

I guess it's not an asio limitation, but the OS one - you just exceeded the maximum message size for the underlying protocol. On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.

[...] All TCP networks have an MTU, and packets larger than this will be sliced up for delivery (without affecting the data contained within or the guaranteed order of delivery).

And herein lies what the original poster is hitting. The MTU is way smaller than the ~1MB limit the poster sees, so that isn't likely the problem (and would typically only be an issue if using UDP). My guess is the TCP read/write buffers have been exceeded. Original poster should include more information: - OS that this is on, - The actual error codes being encountered. If you're exceeding the TCP buffer sizes and trying to write in one chunk, you may be hitting an error like ENOBUFS (no_buffer_space in boost:error_code). Just a guess though.

Nicola Michael Gutberlet

5 Jan 5 Jan

10:27 a.m.

New subject: [asio] Sending large packages

Dear all, Thank you for your replies. I wanted to tell you the error message thrown when trying to send larger packages but it seems to be solved. I don't know why or how... Anyway, it is sensible to check the maximum message size for the socket/OS/whatever. How do I get the information of the maximum size of messages I can write/read over my TCP socket? I noticed the option mentioned by Jason Felice "SO_MAX_MSG_SIZE" but the TCP socket doesn't support it. I tried to use an own template: boost::asio::detail::socket_option::integer cOptionMaxMessageSize; socket->get_option( cOptionMaxMessageSize ); But this also did not work - resulting in the option value -1. Best regards, Nicola -----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Oliver Seiler Sent: 04 January 2010 18:05 To: boost-users@lists.boost.org Subject: Re: [Boost-users] [asio] Sending large packages On Mon, Jan 4, 2010 at 7:30 AM, Jason Felice wrote:

...

On Mon, Jan 4, 2010 at 5:41 AM, Igor R wrote:

...
...
Doesnt anyone have any ideas? Did anyone receive this mail?

I guess it's not an asio limitation, but the OS one - you just exceeded the maximum message size for the underlying protocol. On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.

[...] All TCP networks have an MTU, and packets larger than this will be sliced up for delivery (without affecting the data contained within or the guaranteed order of delivery).

Matheus Araújo Aguiar

10:55 a.m.

New subject: [asio] Sending large packages

On Tue, Jan 5, 2010 at 8:27 AM, Nicola Michael Gutberlet < nicola.gutberlet@hhi.fraunhofer.de> wrote:

...

Dear all,

Thank you for your replies. I wanted to tell you the error message thrown when trying to send larger packages but it seems to be solved. I don't know why or how...

Anyway, it is sensible to check the maximum message size for the socket/OS/whatever. How do I get the information of the maximum size of messages I can write/read over my TCP socket? I noticed the option mentioned by Jason Felice "SO_MAX_MSG_SIZE" but the TCP socket doesn't support it. I tried to use an own template:

boost::asio::detail::socket_option::integer cOptionMaxMessageSize; socket->get_option( cOptionMaxMessageSize );

But this also did not work - resulting in the option value -1.

Hi Nicola, Note that in the code you posted, you wrote the option as "SO*L*_SOCKET, SO_MAX_MSG_SIZE". You have an extra "L" there which might be causing your value -1. Also, remember that as stated before, this option goes for microsoft windows. There is no maximum size you can send through a socket, although there is a MTU(Maximum Transfer Unity) and you should read more about it if you are not familiarized with that concept. As said before, TCP being a stream protocol, you have to ensure in a way or another (by putting the size in front of data, or using an "END OF STREAM" mark) that your read get all the bytes you expect. Hope that helps!

...

Best regards, Nicola

-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Oliver Seiler Sent: 04 January 2010 18:05 To: boost-users@lists.boost.org Subject: Re: [Boost-users] [asio] Sending large packages

On Mon, Jan 4, 2010 at 7:30 AM, Jason Felice wrote:

...
On Mon, Jan 4, 2010 at 5:41 AM, Igor R wrote:

...
...
Doesn’t anyone have any ideas? Did anyone receive this mail?

I guess it's not an asio limitation, but the OS one - you just exceeded the maximum message size for the underlying protocol. On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.

[...] All TCP networks have an MTU, and packets larger than this will be sliced up for delivery (without affecting the data contained within or the guaranteed order of delivery).

And herein lies what the original poster is hitting. The MTU is way smaller than the ~1MB limit the poster sees, so that isn't likely the problem (and would typically only be an issue if using UDP).

My guess is the TCP read/write buffers have been exceeded. Original poster should include more information: - OS that this is on, - The actual error codes being encountered.

If you're exceeding the TCP buffer sizes and trying to write in one chunk, you may be hitting an error like ENOBUFS (no_buffer_space in boost:error_code). Just a guess though. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

-- Matheus Araújo Aguiar Computer Scientist matheus.pit@gmail.com

Matheus Araújo Aguiar

10:59 a.m.

New subject: [asio] Sending large packages

2010/1/5 Matheus Araújo Aguiar

...

On Tue, Jan 5, 2010 at 8:27 AM, Nicola Michael Gutberlet < nicola.gutberlet@hhi.fraunhofer.de> wrote:

...
Dear all,

Thank you for your replies. I wanted to tell you the error message thrown when trying to send larger packages but it seems to be solved. I don't know why or how...

Anyway, it is sensible to check the maximum message size for the socket/OS/whatever. How do I get the information of the maximum size of messages I can write/read over my TCP socket? I noticed the option mentioned by Jason Felice "SO_MAX_MSG_SIZE" but the TCP socket doesn't support it. I tried to use an own template:

boost::asio::detail::socket_option::integer cOptionMaxMessageSize; socket->get_option( cOptionMaxMessageSize );

But this also did not work - resulting in the option value -1.

Hi Nicola, Note that in the code you posted, you wrote the option as "SO*L*_SOCKET, SO_MAX_MSG_SIZE". You have an extra "L" there which might be causing your value -1. Also, remember that as stated

Sorry for that, the option is not written wrong, i confused things about that.

...

before, this option goes for microsoft windows. There is no maximum size you can send through a socket, although there is a MTU(Maximum Transfer Unity) and you should read more about it if you are not familiarized with that concept. As said before, TCP being a stream protocol, you have to ensure in a way or another (by putting the size in front of data, or using an "END OF STREAM" mark) that your read get all the bytes you expect.

Hope that helps!

...
Best regards, Nicola

-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Oliver Seiler Sent: 04 January 2010 18:05 To: boost-users@lists.boost.org Subject: Re: [Boost-users] [asio] Sending large packages

On Mon, Jan 4, 2010 at 7:30 AM, Jason Felice wrote:

...
On Mon, Jan 4, 2010 at 5:41 AM, Igor R wrote:

...
...
Doesn’t anyone have any ideas? Did anyone receive this mail?

I guess it's not an asio limitation, but the OS one - you just exceeded the maximum message size for the underlying protocol. On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.

[...] All TCP networks have an MTU, and packets larger than this will be sliced up for delivery (without affecting the data contained within or the guaranteed order of delivery).

And herein lies what the original poster is hitting. The MTU is way smaller than the ~1MB limit the poster sees, so that isn't likely the problem (and would typically only be an issue if using UDP).

My guess is the TCP read/write buffers have been exceeded. Original poster should include more information: - OS that this is on, - The actual error codes being encountered.

If you're exceeding the TCP buffer sizes and trying to write in one chunk, you may be hitting an error like ENOBUFS (no_buffer_space in boost:error_code). Just a guess though. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

-- Matheus Araújo Aguiar Computer Scientist matheus.pit@gmail.com

-- Matheus Araújo Aguiar Computer Scientist matheus.pit@gmail.com

Tim Couper

6 Jan 6 Jan

5:14 p.m.

New subject: Boost-python wrapping a vector of vectors?

I'm trying to boost-python a vector-of-vectors, like class A { public A(const std::vector>& my_array); }; and would intuitively write the wrapper: BOOST_PYTHON_MODULE(foo) { using namespace boost::python class_<A>("A") .def(init(std::vector()) ; but get the error "a call to a constructor cannot appear in a constant expression" Can anyone supply any guidance as to how to wrap this? Thanks Tim

Simon Pickles

7:17 p.m.

New subject: Boost-python wrapping a vector of vectors?

Tim Couper wrote:

...

Can anyone supply any guidance as to how to wrap this? Thanks

Try asking on cplusplus-sig@python.org - mailing list for 'Development of Python/C++ integration' Si

...

------------------------------------------------------------------------

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Tim Couper

10:25 p.m.

New subject: Boost-python wrapping a vector of vectors?

Ah thanks :-) Tim On 06/01/2010 19:17, Simon Pickles wrote:

...

Tim Couper wrote:

...
Can anyone supply any guidance as to how to wrap this? Thanks

Try asking on cplusplus-sig@python.org - mailing list for 'Development of Python/C++ integration'

Si

...
------------------------------------------------------------------------

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

No virus found in this incoming message. Checked by AVG - www.avg.com Version: 9.0.725 / Virus Database: 270.14.127/2603 - Release Date: 01/06/10 07:35:00

5526

Age (days ago)

5528

Last active (days ago)

List overview

Download

9 comments

7 participants

participants (7)

Igor R
Jason Felice
Matheus Araújo Aguiar
Nicola Michael Gutberlet
Oliver Seiler
Simon Pickles
Tim Couper