Re: [Boost-users] [asio] Sending large packages
Doesn't anyone have any ideas? Did anyone receive this mail? Best regards and a happy new year ;-) _____ From: Nicola Michael Gutberlet [mailto:nicola.gutberlet@hhi.fraunhofer.de] Sent: 15 December 2009 17:28 To: 'boost-users@lists.boost.org' Subject: RE: [asio] Sending large packages Some testing showed that the maximum package (load) size of boost::asio::write/read() for a TCP socket seems to be 1007 kB _____ From: Nicola Michael Gutberlet [mailto:nicola.gutberlet@hhi.fraunhofer.de] Sent: 15 December 2009 17:11 To: 'boost-users@lists.boost.org' Subject: [asio] Sending large packages Dear all, Is there a well-defined way of sending large packages over TCP? Or do I have to split the packages on my own? For explanation: I want to send images (size around 6MB). Best regards, Nicola
Doesn’t anyone have any ideas? Did anyone receive this mail?
Best regards and a happy new year ;-)
________________________________
From: Nicola Michael Gutberlet [mailto:nicola.gutberlet@hhi.fraunhofer.de] Sent: 15 December 2009 17:28 To: 'boost-users@lists.boost.org' Subject: RE: [asio] Sending large packages
Some testing showed that the maximum package (load) size of boost::asio::write/read() for a TCP socket seems to be 1007 kB
I guess it's not an asio limitation, but the OS one - you just exceeded the maximum message size for the underlying protocol. On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.
On Mon, Jan 4, 2010 at 5:41 AM, Igor R <boost.lists@gmail.com> wrote:
Doesn’t anyone have any ideas? Did anyone receive this mail?
I guess it's not an asio limitation, but the OS one - you just exceeded the maximum message size for the underlying protocol. On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.
I've several times encountered a faulty assumption about how TCP works, actually in multiple code bases I've maintained. I'm not sure if this might be the same faulty assumption, but it sounds like it might be. TCP networking is a stream-based protocol and it makes NO guarantees that the receiving end's read sizes are coincident with the sending send's write sizes. In fact, it has a few mechanisms which will actively thwart this, including Nagle's algorithm (see http://en.wikipedia.org/wiki/Nagle%27s_algorithm ). All TCP networks have an MTU, and packets larger than this will be sliced up for delivery (without affecting the data contained within or the guaranteed order of delivery). There is some handshaking to discover the minimum MTU along a route, though this is intended to allow client systems to adapt for maximum throughput, it is not a guarantee that slicing won't happen. It's more like a guarantee that it won't happen frequently. Additionally, loaded routers may well choose to split transmit units or collect and concatenate them in order to meet quality of service and throughput requirements. Unfortunately, though, making this assumption will apparently work often enough with small packet sizes to lull some developers into believing they've confirmed that read and write sizes are coincident. IF you are making this assumption, you need to redesign your wire protocol to include within it delineation of the data, such as sending the data size up front, or using some sort of demarcation or end-of-data marker. You then need to collect the reads on the receiving side until you've reached the demarcation or expected data size.
On Mon, Jan 4, 2010 at 7:30 AM, Jason Felice <jason.m.felice@gmail.com> wrote:
On Mon, Jan 4, 2010 at 5:41 AM, Igor R <boost.lists@gmail.com> wrote:
Doesn’t anyone have any ideas? Did anyone receive this mail?
I guess it's not an asio limitation, but the OS one - you just exceeded the maximum message size for the underlying protocol. On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.
[...] All TCP networks have an MTU, and packets larger than this will be sliced up for delivery (without affecting the data contained within or the guaranteed order of delivery).
And herein lies what the original poster is hitting. The MTU is way smaller than the ~1MB limit the poster sees, so that isn't likely the problem (and would typically only be an issue if using UDP). My guess is the TCP read/write buffers have been exceeded. Original poster should include more information: - OS that this is on, - The actual error codes being encountered. If you're exceeding the TCP buffer sizes and trying to write in one chunk, you may be hitting an error like ENOBUFS (no_buffer_space in boost:error_code). Just a guess though.
Dear all, Thank you for your replies. I wanted to tell you the error message thrown when trying to send larger packages but it seems to be solved. I don't know why or how... Anyway, it is sensible to check the maximum message size for the socket/OS/whatever. How do I get the information of the maximum size of messages I can write/read over my TCP socket? I noticed the option mentioned by Jason Felice "SO_MAX_MSG_SIZE" but the TCP socket doesn't support it. I tried to use an own template: boost::asio::detail::socket_option::integer<SOL_SOCKET, SO_MAX_MSG_SIZE> cOptionMaxMessageSize; socket->get_option( cOptionMaxMessageSize ); But this also did not work - resulting in the option value -1. Best regards, Nicola -----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Oliver Seiler Sent: 04 January 2010 18:05 To: boost-users@lists.boost.org Subject: Re: [Boost-users] [asio] Sending large packages On Mon, Jan 4, 2010 at 7:30 AM, Jason Felice <jason.m.felice@gmail.com> wrote:
On Mon, Jan 4, 2010 at 5:41 AM, Igor R <boost.lists@gmail.com> wrote:
Doesnt anyone have any ideas? Did anyone receive this mail?
I guess it's not an asio limitation, but the OS one - you just exceeded the maximum message size for the underlying protocol. On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.
[...] All TCP networks have an MTU, and packets larger than this will be sliced up for delivery (without affecting the data contained within or the guaranteed order of delivery).
And herein lies what the original poster is hitting. The MTU is way smaller than the ~1MB limit the poster sees, so that isn't likely the problem (and would typically only be an issue if using UDP). My guess is the TCP read/write buffers have been exceeded. Original poster should include more information: - OS that this is on, - The actual error codes being encountered. If you're exceeding the TCP buffer sizes and trying to write in one chunk, you may be hitting an error like ENOBUFS (no_buffer_space in boost:error_code). Just a guess though. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On Tue, Jan 5, 2010 at 8:27 AM, Nicola Michael Gutberlet < nicola.gutberlet@hhi.fraunhofer.de> wrote:
Dear all,
Thank you for your replies. I wanted to tell you the error message thrown when trying to send larger packages but it seems to be solved. I don't know why or how...
Anyway, it is sensible to check the maximum message size for the socket/OS/whatever. How do I get the information of the maximum size of messages I can write/read over my TCP socket? I noticed the option mentioned by Jason Felice "SO_MAX_MSG_SIZE" but the TCP socket doesn't support it. I tried to use an own template:
boost::asio::detail::socket_option::integer<SOL_SOCKET, SO_MAX_MSG_SIZE> cOptionMaxMessageSize; socket->get_option( cOptionMaxMessageSize );
But this also did not work - resulting in the option value -1.
Hi Nicola, Note that in the code you posted, you wrote the option as "SO*L*_SOCKET, SO_MAX_MSG_SIZE". You have an extra "L" there which might be causing your value -1. Also, remember that as stated before, this option goes for microsoft windows. There is no maximum size you can send through a socket, although there is a MTU(Maximum Transfer Unity) and you should read more about it if you are not familiarized with that concept. As said before, TCP being a stream protocol, you have to ensure in a way or another (by putting the size in front of data, or using an "END OF STREAM" mark) that your read get all the bytes you expect. Hope that helps!
Best regards, Nicola
-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Oliver Seiler Sent: 04 January 2010 18:05 To: boost-users@lists.boost.org Subject: Re: [Boost-users] [asio] Sending large packages
On Mon, Jan 4, 2010 at 7:30 AM, Jason Felice <jason.m.felice@gmail.com> wrote:
On Mon, Jan 4, 2010 at 5:41 AM, Igor R <boost.lists@gmail.com> wrote:
Doesn’t anyone have any ideas? Did anyone receive this mail?
I guess it's not an asio limitation, but the OS one - you just exceeded the maximum message size for the underlying protocol. On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.
[...] All TCP networks have an MTU, and packets larger than this will be sliced up for delivery (without affecting the data contained within or the guaranteed order of delivery).
And herein lies what the original poster is hitting. The MTU is way smaller than the ~1MB limit the poster sees, so that isn't likely the problem (and would typically only be an issue if using UDP).
My guess is the TCP read/write buffers have been exceeded. Original poster should include more information: - OS that this is on, - The actual error codes being encountered.
If you're exceeding the TCP buffer sizes and trying to write in one chunk, you may be hitting an error like ENOBUFS (no_buffer_space in boost:error_code). Just a guess though. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- Matheus Araújo Aguiar Computer Scientist matheus.pit@gmail.com
2010/1/5 Matheus Araújo Aguiar <matheus.pit@gmail.com>
On Tue, Jan 5, 2010 at 8:27 AM, Nicola Michael Gutberlet < nicola.gutberlet@hhi.fraunhofer.de> wrote:
Dear all,
Thank you for your replies. I wanted to tell you the error message thrown when trying to send larger packages but it seems to be solved. I don't know why or how...
Anyway, it is sensible to check the maximum message size for the socket/OS/whatever. How do I get the information of the maximum size of messages I can write/read over my TCP socket? I noticed the option mentioned by Jason Felice "SO_MAX_MSG_SIZE" but the TCP socket doesn't support it. I tried to use an own template:
boost::asio::detail::socket_option::integer<SOL_SOCKET, SO_MAX_MSG_SIZE> cOptionMaxMessageSize; socket->get_option( cOptionMaxMessageSize );
But this also did not work - resulting in the option value -1.
Hi Nicola, Note that in the code you posted, you wrote the option as "SO*L*_SOCKET, SO_MAX_MSG_SIZE". You have an extra "L" there which might be causing your value -1. Also, remember that as stated
Sorry for that, the option is not written wrong, i confused things about that.
before, this option goes for microsoft windows. There is no maximum size you can send through a socket, although there is a MTU(Maximum Transfer Unity) and you should read more about it if you are not familiarized with that concept. As said before, TCP being a stream protocol, you have to ensure in a way or another (by putting the size in front of data, or using an "END OF STREAM" mark) that your read get all the bytes you expect.
Hope that helps!
Best regards, Nicola
-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Oliver Seiler Sent: 04 January 2010 18:05 To: boost-users@lists.boost.org Subject: Re: [Boost-users] [asio] Sending large packages
On Mon, Jan 4, 2010 at 7:30 AM, Jason Felice <jason.m.felice@gmail.com> wrote:
On Mon, Jan 4, 2010 at 5:41 AM, Igor R <boost.lists@gmail.com> wrote:
Doesn’t anyone have any ideas? Did anyone receive this mail?
I guess it's not an asio limitation, but the OS one - you just exceeded the maximum message size for the underlying protocol. On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.
[...] All TCP networks have an MTU, and packets larger than this will be sliced up for delivery (without affecting the data contained within or the guaranteed order of delivery).
And herein lies what the original poster is hitting. The MTU is way smaller than the ~1MB limit the poster sees, so that isn't likely the problem (and would typically only be an issue if using UDP).
My guess is the TCP read/write buffers have been exceeded. Original poster should include more information: - OS that this is on, - The actual error codes being encountered.
If you're exceeding the TCP buffer sizes and trying to write in one chunk, you may be hitting an error like ENOBUFS (no_buffer_space in boost:error_code). Just a guess though. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- Matheus Araújo Aguiar Computer Scientist matheus.pit@gmail.com
-- Matheus Araújo Aguiar Computer Scientist matheus.pit@gmail.com
I'm trying to boost-python a vector-of-vectors, like class A { public A(const std::vector<std::vector double >>& my_array); }; and would intuitively write the wrapper: BOOST_PYTHON_MODULE(foo) { using namespace boost::python class_<A>("A") .def(init(std::vector<std::vector<double> >()) ; but get the error "a call to a constructor cannot appear in a constant expression" Can anyone supply any guidance as to how to wrap this? Thanks Tim
Tim Couper wrote:
Can anyone supply any guidance as to how to wrap this? Thanks
Try asking on cplusplus-sig@python.org - mailing list for 'Development of Python/C++ integration' Si
------------------------------------------------------------------------
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Ah thanks :-) Tim On 06/01/2010 19:17, Simon Pickles wrote:
Tim Couper wrote:
Can anyone supply any guidance as to how to wrap this? Thanks
Try asking on cplusplus-sig@python.org - mailing list for 'Development of Python/C++ integration'
Si
------------------------------------------------------------------------
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 9.0.725 / Virus Database: 270.14.127/2603 - Release Date: 01/06/10 07:35:00
participants (7)
-
Igor R -
Jason Felice -
Matheus Araújo Aguiar -
Nicola Michael Gutberlet -
Oliver Seiler -
Simon Pickles -
Tim Couper