On Tue, 28 Dec 2004 21:02:37 -0800 "Robert Ramey" <ramey@rrsd.com> wrote:

However, I don't think the concept of transmission protocal should be mixed into the library - which is already very complex.

I do not think Scott is talking about transmission protocol specifically.

I believe you could easily achive what you want to accomplish by serializaiton to a memory buffer (e.g string_stream) and transmitting this. On the other end the inverse of this process would occur.

If I understand Scott correctly, the problem still exists, if you want to use the lib in that way. Assume an object whose serialization is something like 4K. If reading from a file, or even a TCP stream with the socket in blocking mode, then you just keep reading until you get all the data. However, for a socket in non-blocking mode, you will typically use select or poll or some other notification mechanism to be told when data is available. You will then go read as much as is currently available, and then return to other tasks until more data is ready. Let's say that data is slow, and to ready the entire 4K of data, it takes 10 different "notifications" and 10 different "read" operations. I think Scott is saying that operator>> is insufficient because it can not do a partial read of what is there... it wants to snarf all 4K. I could be missing the boat, but this is the usual problem with serialization methods, when using them with sockets. For this to work, the operator>>() has to know that there is no more data (i.e., correctly interpret return code of read when the fd is in non-blocking mode), and keep its current state so that the next call to operator>>() will continue where the last call left off. I do not see this as a protocol issue but as supporting non blocking reads where you can get the data in many small chunks. Then again, it is possible that the serialization library already does support this in some way...

----- Original Message ----- From: "Robert Ramey" <ramey@rrsd.com> To: <boost@lists.boost.org> Sent: Wednesday, December 29, 2004 7:16 PM Subject: [boost] Re: Re: Serialization and async messaging <snip>

...

I could be missing the boat, but this is the usual problem with serialization methods, when using them with sockets. For this to work, the operator>>() has to know that there is no more data (i.e., correctly interpret return code of read when the fd is in non-blocking mode), and keep its current state so that the next call to operator>>() will continue where the last call left off.

I do not see this as a protocol issue but as supporting non blocking reads where you can get the data in many small chunks.

Then again, it is possible that the serialization library already does support this in some way...

Not as far as I can see. I would say that one should serialize the data in the chunk size you want and not attempt to break up the chunks.

Hmmmm. My background in network messaging gives me a certain POV and now I can see that serialization (a la persistence) gives quite another. Any attempts to arrange "chunks" at the level we are talking about will be subverted by the network (some router somewhere will decide to fragment/coalesce/...). So we can never give significance to a block received, i.e. the first byte may not be the first byte of a serialized object and the last byte may not be the last byte of a serialized object. Thanks for the feedback. My original question was fairly "open" (not deeply researched ;-) I suspected that operator>> was not really going to be appropriate in an async environment but held out some hope that experts would have found a completely different way to apply your library. Scott.

...

Assume an object whose serialization is something like 4K. If reading from a file, or even a TCP stream with the socket in blocking mode, then you just keep reading until you get all the data. However, for a socket in non-blocking mode, you will typically use select or poll or some other notification mechanism to be told when data is available. You will then go read as much as is currently available, and then return to other tasks until more data is ready. Let's say that data is slow, and to ready the entire 4K of data, it takes 10 different "notifications" and 10 different "read" operations.

I don't think this is that big of a problem. This is usual situation in marshaling implementation that parser/builder/marshaler class need to have completed data block. And the usual solution is to use some kind of

----- Original Message ----- From: "Gennadiy Rozental" <gennadiy.rozental@thomson.com> To: <boost@lists.boost.org> Sent: Wednesday, December 29, 2004 7:40 PM Subject: [boost] Re: Re: Serialization and async messaging <snip> protocol

level envelope that allows detect end. So what you do is you collect you pieces in buffer and then process it altogether. This is a case whether you sending object by object or (most probably) whole I don't know ... message. In all practical cases it does not incur any significant time/space overhead (IOW buffer size, envelope and message units could be configured so it's efficient).

Understand this and can envision a successful implementation. It would seem to involve some duplication though, i.e. the generation and detection of the envelopes is similar to what occurs in serialization already. It was this duplication of (apparently equivalent) code that I was hoping to avoid in the first place. If the serialization format (e.g. XML) already satisfies your requirement for "some kind of protocol level envelope that allows detect end" then it seem a little tragic to ignore that potential? OTOH the envelope technique can be applied to all sorts of things and that might be enough to satisfy the aesthete/engineer. Cheers.

----- Original Message ----- From: "Caleb Epstein" <caleb.epstein@gmail.com> To: <boost@lists.boost.org> Sent: Thursday, December 30, 2004 4:10 AM Subject: Re: [boost] Re: Serialization and async messaging

...

I think Scott is saying that operator>> is insufficient because it can not do a partial read of what is there... it wants to snarf all 4K.

I think at least one person here is mixing pickles and milk, as my old Latin teacher used to say.

Firstly, the serialization library doesn't use operator>> or operator<< at all. If you're using these operators on some sort of socket iostream and you don't want blocking to occur, thats your first mistake.

When reading/writing data with sockets, its up to you to use an appropriate protocol that can detect "message" boundaries so you know when you have a complete block of data to work with. See for example the BEEP protocol (http://beepcore.org/) for a nice specification and implementation.

Don't tightly couple this communications layer to the application layer. The two layers should just pass "complete" blocks of data (e.g. serialized objects) between each other.

At the core of the comments I have read, I believe that a common problem has been identified. The separation of application-level communications from the send+recv of blocks over a socket is the abstract, while "operator>>... wants to snarf all 4K" is the dirty guts of it? Thanks for the reference. First review has me resigned to another round of research. Cheers.

From: "Robert Ramey" <ramey@rrsd.com> To: <boost@lists.boost.org> Sent: Wednesday, December 29, 2004 6:02 PM Subject: [boost] Re: Serialization and async messaging <snip>

...

IMHO the input function (operator>>) needs to be "re-entrant" if serialization can be used in an async environment. Repeated calls should be perfectly acceptable. Ultimately it would return an indication that a complete object has indeed been loaded. Some "state" needs to be held somewhere. My best guess is that this is not the case but I cant find anything conclusive (without wading deeper into code).

Can anyone comment?

I'm not sure I'm understanding exactly what you want to do - but that deters me not at all from making a comment.

I envisioned the library would be useful for marshalling data accross space (transmitting/recieving between programs) as well as across time (persistence - the most common application). So I would be pleased to see someone apply it in this way.

However, I don't think the concept of transmission protocal should be mixed into the library - which is already very complex.

I believe you could easily achive what you want to accomplish by serializaiton to a memory buffer (e.g string_stream) and transmitting

If you mean here that it would be unwise to couple the serialization to TCP then yes, I understand that. this.

On the other end the inverse of this process would occur.

On the transmit side your suggestion seems completely workable. On the receive side you have the problem (i.e. in async environment) of not knowing whether the data available at any point will be enough to complete the call to operator>>. You cant make a call that is going to "block" in some way.

...

ps: Its a thoroughly amazing library.

Have you concluded this from actually using the library or from just reading the documentation?

The documentation, the functionality that has been achieved and my struggles to achieve less. I hadnt even considered trying to recover objects including pointers to objects that... I have done some analogous work in the area of async messaging, i.e. lots of marshalling inside async frameworks. I have recently tried to extend this to persistence (or serialization) as supporting two distinct bodies of code seems ill-directed. The option that I was exploring here was to apply your serialization back the other way; to async messaging. Cheers.

----- Original Message ----- From: "Robert Ramey" <ramey@rrsd.com> To: <boost@lists.boost.org> Sent: Thursday, December 30, 2004 8:34 AM Subject: [boost] Re: Re: Serialization and async messaging

...

On the transmit side your suggestion seems completely workable. On the receive side you have the problem (i.e. in async environment) of not knowing whether the data available at any point will be enough to complete the call to operator>>. You cant make a call that is going to "block" in some way.

I have to say I just can't see this understand this. I envision such a process as functioning in the following way.

transmitting program ============== serialize to a string. we now have its total length. transmit a string using what ever method syncronous/asyncronous whatever.

recieving program ============ retrieve a string - using what ever method async, sync or whatever. when a complete string is retrieved/reassembled or whatever de-serialize to the original structure.

Honestly, I can't see any thing about this that is less than optimal.

Aha. After Gennadiy's version of this I can now see it for what it is. Yes, the simple response is that this will work. A pedantic response is that it feels like duplication; why implement another layer (the envelopes) just to detect completion of objects, ignoring the potential in the existing serialization format (e.g. XML). A more useful response would point out the higher memory requirements, two scanning phases and increased incidence of copying. A comparative analysis is probably impossible. Given opportunity I would probably explore the non-envelope version buts thats curiosity and the sex appeal of "doing tricky stuff that results in less lines". Commercially the envelope version seems like a winner. Thanks.

Robert Ramey wrote:

Scott Woods wrote:

If you're really a gluton for punishment consider the following:

make you sockets/tcp/whatever implementation with a i/ostream interface.

The punishment level may be reduced somewhat by using Jonathan's IOStream library. Jeff