Re: [Boost-users] Buffering and ASIO

Good day. I'm trying to read video data transferred by HTTP. I use the following function in order to read HTTP header: boost::asio::async_read_until(m_socket, m_headerBuffer, "\r\n\r\n", boost::bind(&CAXISParser::read_header, this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
, where m_headerBuffer is asio::streambuf object. During header parsing i can get video sample length. Now I want to read all video data in one function call, for example:
boost::asio::async_read(m_socket, boost::asio::buffer(pSample->GetBody(), m_currentSampleLength), boost::bind(&CAXISParser::read_body, this, pSample, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
, where pSample is object with buffer for video data. There is problem that async_read_until captures > some data from video data, and that call's behavior is incorrect. How can i fight that problem? For example, how can i figure out number of bytes left in streambuf and copy them to my buffer, and then read rest with async_read? I'm will very appreciate for any advices.
It's not clear to me exactly what you're asking here. From what I can tell, you have created a buffer of size m_currentSampleLength, and asked ASIO to call CAXISParser::read_body when it has received this many bytes. Is that what you intended to do? Is that what boost is doing? Or is the problem something else?
-----Scott.
Hello, Scott. Sorry for bad english, i'm will try explain. Yep, there is problem. When i'm read HTTP header with acync_read_until, it consume header and some bytes from video data (i think). After i'm calculate video data length from header and trying read data with async_read. But it works incorrect as previous call async_read_until captures some bytes from video data and video data readed with async_read have wrong offset. How can i understand, the problem that async_read_until and async_read used different buffers for reading (asio::streambuf and custom buffer), and some video data consumed by streambuf. Question: how to get video data captured with async_read_until and read rest from tcp buffer?There is is code sample:void read_header(PSample pSample, const boost::system::error_code& error, size_t bytes_transferred){ // Calculate data length with regex boost::asio::const_buffer data = m_headerBuffer.data(); const char* beg = boost::asio::buffer_cast<const char*>(data); const char* end = beg + bytes_transferred; CalculateDataLength(beg, end); // Clear buffer std::vector<char> vec(bytes_transferred); m_headerBuffer.sgetn(&vec[0], bytes_transferred); ReadBody();} Before ReadBody call, i reject streambuf content with sgetn. But i think there ismore data in streambuf that is video data. How can i get it? Below non working sample, but it describe my wishes: // Add before read bodysize_t restBytes = m_headerBuffer.size() - bytes_transferred;if (0 != restBytes){ m_headerBuffer.sgetn(pSample->GetBody(), restBytes);} Then in ReadBody i can to read rest video data with single async_readcall. There is implementation sample: void ReadBody(NMMSS::PSample& pSample){1)//boost::asio::async_read(m_socket, m_headerBuffer.prepare(m_currentSampleLength),2)//boost::asio::async_read_until(m_socket, m_headerBuffer, "\r\n",3)//boost::asio::async_read(m_socket, boost::asio::buffer(pSample->GetBody(), m_currentSampleLength), boost::bind(&CAXISParser::read_body, this, pSample, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));} I trying use 3 variants there:1) Trying read to buffer all video data (dont understand why it doesn't work)2) Very slow variant, as video data is big.3) Preferrable variant P.S. One question more: how can i correctly reject data consumed by streambuf? Now i used the following strategy: streambuf buf; ..... std::vector<char> vec(size); buf.sgetn(&vec[0], size); Is there another strategy for that? For example, commit and consume strategy? I dont need streambuf content in some cases. Best regards, Vadim

IIUC, you're trying to mix and match calls to async_read_until (with a streambuf) and calls to async_read (with a buffer parameter), but having problems with "extra data past the length data". If you keep the same design, you will need to first extract the "extra data" out of the streambuf, place it in to the buffer you want to use, then call async_read or async_read_at with the partially filled buffer. While I haven't done anything similar, I can't think of a reason this won't work. However, I think it would be easier to use async_read_until calls only, and keep all data in the streambuf. Create a match_condition function object, track the state of the HTTP transfer in the function object, and then return a true condition when the full buffer has been received. This encapsulates all of your "buffer filling logic" into one small place and I think is the simplest code of everything I'm writing in this e-mail. If you cannot use a streambuf for the full transfer for whatever reason, then I would not use async_read_until and instead use async_read or async_read_at with a single buffer for everything. It shouldn't be hard to manage the "two states" of the transfer ("need length header", "have length header, getting rest of data"). Your handler will be called multiple times (most likely) while in the second state (and it's possible it will get called multiple times while in the first state, depending on how congested the TCP incoming buffers are). I think your design of trying to mix the two models is more complicated than it needs to be. Cliff

Good day. I'm developing an application that transfer video data and there is problem with transfer big data chunk (~600 Kb) over UDP. Data processing consumes 50% CPU and performance is not good ( i need transfer 25 chunks in sec, but really there is about 18). Below is code snippet: ....... // Get video data chunk void Receive(ISample* pSample) { // Divide chunk on RTP packets IPacketIterator* iter = m_accessor->GetIterator(pSample); SendDatagram(iter); } // Send datagram void SendDatagram(IPacketIterator* iter) { if (iter->IsValid()) { IProtocolPacket* packet = iter->Next(); std::vector<asio::const_buffer> buffs(2); buffs.push_back(asio::const_buffer(packet->GetHeader(), packet->GetHeaderSize()); buffs.push_back(asio::const_buffer(packet->GetBody(), packet->GetBodySize()); m_socket.async_send_to(buffs, m_endpoint, boost::bind(&CChannel::handle_send, this, iter, placeholders::error, placeholders::bytes_transferred); } else { // Query next chunk ........ } } void handle_send(IPacketIterator* iter, const boost::system::error_code& error, size_t bytes_trans) { if (!error) SendDatagram(iter); } Anyone can help me with that problem? Best regards, Vadim

Try the following to see if the (probably small) reduction in memory allocation / deallocation helps (I really doubt it will help much - I think there's something else that is causing problems, since you're seeing some major performance problems that probably have nothing to do with the code sample you posted): Change: IProtocolPacket* packet = iter->Next(); std::vector<asio::const_buffer> buffs(2); buffs.push_back(asio::const_buffer(packet->GetHeader(), packet->GetHeaderSize()); buffs.push_back(asio::const_buffer(packet->GetBody(), packet->GetBodySize()); To: IProtocolPacket* packet = iter->Next(); boost::array<asio::const_buffer, 2> buffs = { asio::buffer(packet->GetHeader(), packet->GetHeaderSize()), asio::buffer(packet->GetBody(), packet->GetBodySize()) }; (Warning - uncompiled and untested.) --- Hmmm - I see something that could be suspicious: "packet" is a local variable (short lifetime) - are the pointers obtained from packet valid (point to good memory) until the handle_send function is called? Asio does not own the data that the buffer objects wrap. Otherwise, you might want to post your io_service run code to see if there's something suspicious there. Cliff

Hello Vadim, This is a piece of code from an application that inadvertently does what your application appears to be doing: receives a stream of images over a HTTP connection. We had the same problems and you have in bold the solution we found: we call in_avail() on the read-buffer of the stream that tells us the amount of data already read but left in the buffer and then we readsome to get the actual data. void onConnect(boost::system::error_code ErrorCode,unsigned int fps) { boost::asio::streambuf response; boost::asio::read_until(*Socket, response, "\r\n"); std::istream response_stream(&response); response_stream >> http_version; response_stream >> status_code; std::getline(response_stream, status_message); if (!response_stream || http_version.substr(0, 5) != "HTTP/") { THROW_EXCEPTION(logic_error("Invalid HTTP response")); } // Read the response headers, which are terminated by a blank line. boost::asio::read_until(*Socket, response, "\r\n\r\n"); while (std::getline(response_stream, header) && header != "\r") { LOG_DEBUG(log,"Got http header: "<<header); } //Get the number of bytes we have received after the HTTP header size_t avail = response_stream.rdbuf()->in_avail(); //Read that data into the buffer response_stream.readsome(Buffer.data(), avail ); LOG_DEBUG(log,"Connection succeded. Already received "<<avail); //Call the asynchronous receive handler as if an read operation was completed with avail bytes of data onReadComplete(ErrorCode,avail ,fps); } void onReadComplete(const boost::system::error_code& ErrorCode,size_t bytesTransfered,unsigned int fps) { if (!ErrorCode) { boost::asio::async_read( *Socket, boost::asio::buffer(Buffer), boost::asio::transfer_all(), boost::bind( & onReadComplete, this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred, fps ) ); } } Best regards, ---- Emil Mieilica IT Manager Office and mailing : Calea Bucuresti no 3A, Otopeni, Ilfov, Romania Tel: +40 21 350 40 57 ; +40 21 350 40 54 ; +40 21 350 40 55 ; +40 21 350 40 56 ; +4031.620.0212; +4031.620.0213; +4743.26.4142 Fax: +40 21 350 15 80 E-mail: emilm@mbtelecom.ro<mailto:emilm@mbtelecom.ro> office@mbtelecom.ro<mailto:office@mbtelecom.ro> soft@mbtelecom.ro<mailto:soft@mbtelecom.ro> Web: http://www.mbtelecom.com<http://www.mbtelecom.com/> From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Vadim Shmelev Sent: Monday, December 15, 2008 1:55 PM To: boost-users@lists.boost.org Subject: Re: [Boost-users] Buffering and ASIO
Good day. I'm trying to read video data transferred by HTTP. I use the following function in order to read HTTP header: [sip]
participants (3)
-
Cliff Green
-
Emil Mieilica
-
Vadim Shmelev