Is there a marshaling library in boost and is there an interest in such a functionality? Is the serializing library any use to marshall calls? I've not seen any mails concerning marshaling recently, though I might have missed one. Thanks, Hans I'm thinking something along the lines of the following. Although it may not be possible, I think it is (maybe the function registry is not necessary). int func( int i, int j ) { using namespace std; cout << "i: " << i << " j:" << k << endl; return i + j; } int main() { using m = boost::marshall; using std; int funcid = 1; m::register_function( funcid, func ); // Using same principles as in boost::function for type safety. m::call x(funcid, 1, 2); string str = x.serialize(); m::call y( str ); try { cout << y() << endl; } catch( m::invalid_function x ) { cout << x.what() << endl; } return 0; } // ------ // Should output: // i: 1 j: 2 // 3
Hi Hans, If there isn't one in the works already, I'd be interested in helping throw one together. I have some related code in the signal network library that I am working on, namely serializing values of parameters of a function call into a string, sending the string over an asio socket, and then unserializing on the other end and doing a function call. I started isolating the code which can be used to help with that aspect of the marshalling, and using it to throw together a simple marshalling framework. I'll send it out tonight or tomorrow ehen I finish the one or two more things that it needs to do anything usable. Maybe that will spark some ideas on where it could go. Stjepan On 4/13/07, Hans Larsen <hans@poltras.com> wrote:
Is there a marshaling library in boost and is there an interest in such a functionality? Is the serializing library any use to marshall calls?
I've not seen any mails concerning marshaling recently, though I might have missed one.
Thanks, Hans
Hi, Here is one that the author is working towards Boosting it... http://channel.sourceforge.net/#boost Maybe a few of you interested parties could get together to develop it faster... Thanks Shams -- "Hans Larsen" <hans@poltras.com> wrote in message news:7D55A7CC-F228-4BAA-A3F3-210D84D79901@poltras.com...
Is there a marshaling library in boost and is there an interest in such a functionality? Is the serializing library any use to marshall calls?
I've not seen any mails concerning marshaling recently, though I might have missed one.
Thanks, Hans
I'm thinking something along the lines of the following. Although it may not be possible, I think it is (maybe the function registry is not necessary).
int func( int i, int j ) { using namespace std; cout << "i: " << i << " j:" << k << endl; return i + j; }
int main() { using m = boost::marshall; using std;
int funcid = 1; m::register_function( funcid, func ); // Using same principles as in boost::function for type safety. m::call x(funcid, 1, 2); string str = x.serialize(); m::call y( str ); try { cout << y() << endl; } catch( m::invalid_function x ) { cout << x.what() << endl; } return 0; }
// ------ // Should output: // i: 1 j: 2 // 3 _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Well, I threw together a little hack of a marshalling wanna-be framework... Files attached. Implementation is ..eh.., and the results of the call (either returned or in the parameters) are not marshalled back at all. The usage example goes something like: marshall::registry<int> reg; reg.set<void (int, int)>(2, func2); // register function reg.set<void (int)>(1, func1); // register function reg.set<void ()>(0, func0); // register function // embed the function id into the call as well as two parameters std::string call2_5_6 = marshall::serialize<int, void (int, int)>(2, 5, 6); // embed just two parameters std::string call_5_6 = marshall::serialize<void (int, int)>(5, 6); // embed just one parameter std::string call_5 = marshall::serialize<void (int)>(7); // embed nothing std::string call = marshall::serialize<void ()>(); reg(call2_5_6); // call via the embedded function id; results in func2(5,6) reg(2, call_5_6); // call by specifying the id; func2(5,6) reg(1, call_5); // call by specifying the id; func1(5) reg(1, call_5_6); // no checking to see if extra parameters are left over; func1(5) reg(0, call); // call by specifying the id; func0() ...not sure if anything beyond this will work with the current implementation. Anyway, If Channel can already do this, cool... If not and someone wants to lead getting a marshalling thing together, or finishing the Boostification of Channel, I can help a little but I need to focus on the signal network library for GSoC. In any case, I'd be interested in getting some marshalling into boost because that allows for distributed signal networks that can not only pass signals from one computer to another but also execute RPCs. Stjepan On 4/16/07, Shams <shams@orcon.net.nz> wrote:
Hi,
Here is one that the author is working towards Boosting it...
http://channel.sourceforge.net/#boost
Maybe a few of you interested parties could get together to develop it faster...
Thanks Shams
--
"Hans Larsen" <hans@poltras.com> wrote in message news:7D55A7CC-F228-4BAA-A3F3-210D84D79901@poltras.com...
Is there a marshaling library in boost and is there an interest in such a functionality? Is the serializing library any use to marshall calls?
I've not seen any mails concerning marshaling recently, though I might have missed one.
Thanks, Hans
I'm thinking something along the lines of the following. Although it may not be possible, I think it is (maybe the function registry is not necessary).
int func( int i, int j ) { using namespace std; cout << "i: " << i << " j:" << k << endl; return i + j; }
int main() { using m = boost::marshall; using std;
int funcid = 1; m::register_function( funcid, func ); // Using same principles as in boost::function for type safety. m::call x(funcid, 1, 2); string str = x.serialize(); m::call y( str ); try { cout << y() << endl; } catch( m::invalid_function x ) { cout << x.what() << endl; } return 0; }
// ------ // Should output: // i: 1 j: 2 // 3 _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
----- Original Message ----- From: "Stjepan Rajko" <stipe@asu.edu>
The usage example goes something like:
marshall::registry<int> reg;
reg.set<void (int, int)>(2, func2); // register function reg.set<void (int)>(1, func1); // register function reg.set<void ()>(0, func0); // register function
Just a bullet-point review of your example of marshalling in case I'm missing something; * a register of return type+parameter list, keyed on small, unique integer, * generate portable image (string) of a call, i.e. "marshall", * call the registered function using one of several available conventions.
Anyway, If Channel can already do this, cool... If not and someone wants to lead getting a marshalling thing together, or finishing the Boostification of Channel, I can help a little but I need to focus on the signal network library for GSoC.
I would also like to see marshalling in Boost. An early $0.02 worth is that persistence, serialization, async sockets and state machines are all subtly related and that huge benefits might be realized from exploring this. Oh and that marshalling is just a combination of some of the above items.
In any case, I'd be interested in getting some marshalling into boost because that allows for distributed signal networks that can not only pass signals from one computer to another but also execute RPCs.
That would be inherent, i.e. the ability to execute RPCs is a freebie once distributed signaling is working? I dont mean literally, more that an RPC would be achieved through an exchange of a particular set of signals. Scott
Hi Scott, On 4/16/07, Scott Woods <scottw@qbik.com> wrote:
Just a bullet-point review of your example of marshalling in case I'm missing something; * a register of return type+parameter list, keyed on small, unique integer, * generate portable image (string) of a call, i.e. "marshall", * call the registered function using one of several available conventions.
Yep, you got it. That's it so far :-) The key is templated so theoretically it could be something other than an int (I used int because that was in the original example by Hans).
In any case, I'd be interested in getting some marshalling into boost because that allows for distributed signal networks that can not only pass signals from one computer to another but also execute RPCs.
That would be inherent, i.e. the ability to execute RPCs is a freebie once distributed signaling is working? I dont mean literally, more that an RPC would be achieved through an exchange of a particular set of signals.
Yeah, what I have right now in the signal network library prototype is a little pair of objects which serve as a trans-socket signal. One of them receives a signal / function call locally, serializes the arguments via Boost.Serialization into a string, and sends the string through an asio socket. On the other end, the other object receives the packet, de-serializes it, and generates an identical signal/function call on that side. Marshalling would work the same way except the packet would also carry the function id/key, and on the other end a function would be called from a function registry. Neither of these mechanisms handle returning of the data from the function call right now, and even if they did I'm sure there's a lot of other marshalling/RPC issues to be considered... not an expert. Stjepan
On 17-Apr-07, at 1:07 AM, Stjepan Rajko wrote:
Hi Scott,
On 4/16/07, Scott Woods <scottw@qbik.com> wrote:
Just a bullet-point review of your example of marshalling in case I'm missing something; * a register of return type+parameter list, keyed on small, unique integer, * generate portable image (string) of a call, i.e. "marshall", * call the registered function using one of several available conventions.
Yep, you got it. That's it so far :-) The key is templated so theoretically it could be something other than an int (I used int because that was in the original example by Hans).
I like the idea of having a type as an identity mean. We'll have to make it type-safe during serialization though, but it's a good idea.
In any case, I'd be interested in getting some marshalling into boost because that allows for distributed signal networks that can not only pass signals from one computer to another but also execute RPCs.
That would be inherent, i.e. the ability to execute RPCs is a freebie once distributed signaling is working? I dont mean literally, more that an RPC would be achieved through an exchange of a particular set of signals.
Yeah, what I have right now in the signal network library prototype is a little pair of objects which serve as a trans-socket signal. One of them receives a signal / function call locally, serializes the arguments via Boost.Serialization into a string, and sends the string through an asio socket. On the other end, the other object receives the packet, de-serializes it, and generates an identical signal/function call on that side.
That's exactly what I had in mind.
Marshalling would work the same way except the packet would also carry the function id/key, and on the other end a function would be called from a function registry.
Neither of these mechanisms handle returning of the data from the function call right now, and even if they did I'm sure there's a lot of other marshalling/RPC issues to be considered... not an expert.
Basically, I see marshalling in current implementations as a four steps process (although it's more like 2 step mirrored...): 1. The call and all its parameter is made into a structured sequence of symbols (serializing into binary or XML) so that it can be reconstructed (basically, a function [F: Any -> Something] which can be reversed). This may imply a way to identify the function to call (in a type-safe manner[1]). 2. The sequence is sent over a stream of communication.[3] 3. The call is made, and the return parameters (function return considered as a parameter of type reference in this case) are serialized again. [2] 4. The return parameters are sent back over the stream of communication and back to the callee. It may really be interesting if objects were deleted during the calls. Also, thread safety has a funny eerie aura around those things... [1] The problem is much more complex than it seems, and may be left to the user: how can we be sure that function X will be called if we ask to call function X. Registering a function does not ensure type- safety over two different codes. Maybe we'll need to identify the types further, but how can we put an id on type "CEmployee"...? BTW, basically nothing prevents a COM object from being mis-called if interface and GUID doesn't concur, meaning you may register a GUID but use it with an I... different, and I think CORBA has same problem. We may want to say it's up to user to register functions correctly. [2] We'll have to find a way to identify in, out and in-out parameters (return parameter included). I'm working on a way right now. [3] Network security in mind. I've seen enough SAMBA and WinDCE buff overflow vulnerability to make me "vomit in my own mouth". What I offer to solve in a short term is only point 1. Point 2 require at least a way to communication over a network (using Signals) or inter-processes (I read once about Boost.Interprocess... what's the status?). Once a serialization is made, sending it over a stream is not really difficult. Waiting the return value is not necessarly a problem. Basically, if we can save the marshall and load it then call it from another executable, then the networking or interprocessing is of little challenge. Example of the kind of nasty case we're facing: char* func0( int& i, int n ) { i = i + n; return "Hello World!"; } int main() { marshall::registry<int> reg; reg.set< char*(int&, int) >(0, func0); int k = 8; // Putting the call id aside from param list may be a good way to improve readability. // !!! {a} marshall::call c1 = reg[0]( k, 1 ); // This should print "Hello World!9\n"... cout << c1() << k << endl; // !!! {b} return 0; } Basically this is a two step process (the call is not actually made when we create the variable, but only when we call it). Final solution may be more along the lines of: // !!! {a} marshall::call c1 = reg[0]; // This should print "Hello World!9\n"... cout << c1(k,1) << k << endl; // !!! {b} Then c1(...) will then send over network, wait for answer, et cetera. I will put together some code this week, so we can have a prototype started. Hans PS: No use reinventing the wheel, I will check around CORBA for the network communication part. I was more thinking along the "serialize params and returns" as a simple step toward the final solution than the whole yard all at once. We might use their (CORBA) protocol if it's perfect for our needs. We might even want to template the network protocol so that we can call Web Services instead... I'm still dreaming :)
Stjepan _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/ listinfo.cgi/boost
Hello Hans, On 4/16/07, Hans Larsen <hans@poltras.com> wrote:
On 17-Apr-07, at 1:07 AM, Stjepan Rajko wrote:
Yeah, what I have right now in the signal network library prototype is a little pair of objects which serve as a trans-socket signal. One of them receives a signal / function call locally, serializes the arguments via Boost.Serialization into a string, and sends the string through an asio socket. On the other end, the other object receives the packet, de-serializes it, and generates an identical signal/function call on that side.
That's exactly what I had in mind.
Cool. The code is available from: at http://dancinghacker.com/code/signet/ Feel free to use it :) The current code (which allows you to send a specific function call over the socket) could be used with the marshalling library sketch by making the "reg(call2_5_6)" or "reg(2, call_5_6)" call over the network. While this would work it's inefficient because it serializes/deserializes twice (once to construct/deconstruct the call2_5_6 string, for example, and once to construct/deconstruct the network packet). So, I'll work on separating out the socketing functionality so it can be used separately by either the current over-a-socket signal sender/receiver pair, or the RPC sender/receiver pair. Also, the current receiver is synchronous which is not very useful - I still need to add an asynch version.
<snip> [2] We'll have to find a way to identify in, out and in-out parameters (return parameter included). I'm working on a way right now.
Just an idea, but perhaps as a default behavior we can use the parameter types to deduce what can be used to return a value? For example, any parameter that is a non-const reference or pointer to non-const automatically gets marshalled back. We can then offer functionallity to override this default behavior (i.e., don't marshall anything back, or only marshall something specific). Stjepan
Hello, Yes Channel already support transparent distributed message passing among different processes (among different machines thru ASIO and in the same machines thru Interprocess shared memory queues), there are tutorial samples for both scenarios. The messages are marshaled/demarshaled using Boost.Serialization. However Channel is for asynchronous message passing, so it is a one-way, no-return call. To implement request-response style RPC, the caller should pack a response -id/name in the request message and wait on the response-id/name; and the callee at the other end can return the result with the response-id/name. Regards, Yigong On 4/16/07, Stjepan Rajko <stipe@asu.edu> wrote:
Well, I threw together a little hack of a marshalling wanna-be framework... Files attached. Implementation is ..eh.., and the ....... Anyway, If Channel can already do this, cool... If not and someone wants to lead getting a marshalling thing together,
----- Original Message ----- From: "Yigong Liu" <yigongliu@gmail.com> To: <boost@lists.boost.org> Hi Yigong, Hans, Stjepan... Good to hear about all the "marshalling" activity out there. Sounds like we have been treading similar paths. I have grumbled about the tools and techniques available in this area for years. Recently I was cornered into a lot of distributed software development and re-discovered how hard it all is. I now have a wish-list for tools and techniques I might consider for marshalling in the future. Here are a few; * single definition of external representation (i.e. serialization) for persistence and network messaging purposes * multiple encodings (e.g. readable, fast or secure) * purely async software at the lower levels, i.e. only an application thread may block on a sync RPC * multiple program and threading models (e.g. simple server like an FTP server vs a complex program that initiates and accepts multiple connections with different encodings and protocols) * formalized protocols, i.e. an RPC protocol might have CALL, RETURN and EXCEPTION entries. * object-based message routing (e.g. a (local) object originates a message and a (possibly remote) object is the target) The scope of this list get daunting quickly. What is the scope for you guys?
----- Original Message ----- From: "Scott Woods" <scottw@qbik.com> To: <boost@lists.boost.org>
Hi Yigong, Hans, Stjepan...
Good to hear about all the "marshalling" activity out there. Sounds like we have been treading similar paths.
I have grumbled about the tools and techniques available in this area for years. Recently I was cornered into a lot of distributed software development and re-discovered how hard it all is. I now have a wish-list for tools and techniques I might consider for marshalling in the future. Here are a few;
* single definition of external representation (i.e. serialization) for persistence and network messaging purposes * multiple encodings (e.g. readable, fast or secure) * purely async software at the lower levels, i.e. only an application thread may block on a sync RPC * multiple program and threading models (e.g. simple server like an FTP server vs a complex program that initiates and accepts multiple connections with different encodings and protocols) * formalized protocols, i.e. an RPC protocol might have CALL, RETURN and EXCEPTION entries. * object-based message routing (e.g. a (local) object originates a message and a (possibly remote) object is the target)
Oops and; * integration with standard protocols and encodings
Hi everyone, On 4/17/07, Scott Woods <scottw@qbik.com> wrote:
I now have a wish-list for tools and techniques I might consider for marshalling in the future. Here are a few;
* single definition of external representation (i.e. serialization) for persistence and network messaging purposes * multiple encodings (e.g. readable, fast or secure) * purely async software at the lower levels, i.e. only an application thread may block on a sync RPC * multiple program and threading models (e.g. simple server like an FTP server vs a complex program that initiates and accepts multiple connections with different encodings and protocols) * formalized protocols, i.e. an RPC protocol might have CALL, RETURN and EXCEPTION entries. * object-based message routing (e.g. a (local) object originates a message and a (possibly remote) object is the target) * integration with standard protocols and encodings
The scope of this list get daunting quickly. What is the scope for you guys?
My personal needs are pretty basic - just making a remote call and getting the results back would cut it. In terms of designing a marshalling framework, I'd propose separating out as much functionality from threading, network, shared memory, and other issues, and sticking to the actual function call and execution (for starts). What I'm hoping is that things can be designed so that other Boost libraries can transparently address these issues through existing functionality. For example, the readable/fast encodings can be used by allowing the choice of either binary or xml (or text) serialization archives offered by Boost.Serialization. A secure channel can be established by using an SSL asio connection, etc. Your wish list encompasses a lot of things to keep in mind - let's see how many we can map to existing Boost functionality. Cheers, Stjepan
----- Original Message ----- From: "Stjepan Rajko" <stipe@asu.edu> To: <boost@lists.boost.org> Sent: Thursday, April 19, 2007 10:53 AM Subject: Re: [boost] Marshalling library
The scope of this list get daunting quickly. What is the scope for you guys?
My personal needs are pretty basic - just making a remote call and getting the results back would cut it. In terms of designing a marshalling framework, I'd propose separating out as much functionality from threading, network, shared memory, and other issues, and sticking to the actual function call and execution (for starts).
While I understand your motives, I suggest that there are some preliminary iterations of developing marshalling that may be leapfrogged, e.g. a sub-system dedicated to delivering RPC might move through the issues in a blinkered fashion. For example an initiative that targets RPC but considers threading and object-based message routing as enabling technologies would be a different ballgame to the one you are considering?
What I'm hoping is that things can be designed so that other Boost libraries can transparently address these issues through existing functionality. For example, the readable/fast encodings can be used by allowing the choice of either binary or xml (or text) serialization archives offered by Boost.Serialization. A secure channel can be established by using an SSL asio connection, etc.
I like the picture but suspect there will be trouble along the road :-) Without some notion of an overall framework there could be many deadends. There are already deadends in other Boost threads and on other forums. Dont mean to rain on this discussion. Love this area of software probably cos its proven so intractable. Cheers.
----- Original Message ----- From: "Stjepan Rajko" <stipe@asu.edu> To: <boost@lists.boost.org> Sent: Thursday, April 19, 2007 10:53 AM Subject: Re: [boost] Marshalling library
The scope of this list get daunting quickly. What is the scope for you guys?
My personal needs are pretty basic - just making a remote call and getting the results back would cut it. In terms of designing a marshalling framework, I'd propose separating out as much functionality from threading, network, shared memory, and other issues, and sticking to the actual function call and execution (for starts).
What I'm hoping is that things can be designed so that other Boost libraries can transparently address these issues through existing functionality. For example, the readable/fast encodings can be used by allowing the choice of either binary or xml (or text) serialization archives offered by Boost.Serialization. A secure channel can be established by using an SSL asio connection, etc.
How's this for a 1st cut breakdown; * asynchronous socket manager (definitely a thread or threads, might be in asio) * asynchronous RPC manager (probably a thread) with a sync interface for application access * synchronous application thread(s) Something like this would deliver what you are after, if in a more cumbersome way than what you might have been thinking. Maybe some magic is created if it also caters to needs such as the following; * deriving from the base socket manager to create an optimally fast FTP server * ability to communicate with two RPC servers, one running a high-speed proprietary encoding (Boost!) to a local server the other running good old Sun RPC (is that XDR encoding?) to some remote legacy box, the location and associated encoding being transparent to the caller. * ability to switch the local server encoding between fast and readable (i.e. debug vs release) * facility to develop a monitoring and control tool for a socket manager. * ability to instantiate both a base socket manager and the dedicated FTP server in the one executable, each one taking ownership of an accessible interface or port range. While these requirements might seem contrived, the circumstances are not so unrealistic. If you tackle the intiial goal (RPC) but deliver a collection of types and objects that provides for the needs above then I really think you would have something. Its the decomposition into the useful pieces; that's the intractable bit. If you get it right then its easy to reshuffle into new arrangements such as those above. Hope that explains my position a bit better. The solution is not obvious to me and thats probably been apparent in my messages. Cheers.
Hello, I have worked some on the RPC/marshal implementation. FWIW, I'm attaching the new code - it builds on MSVC 8.0 and apple's branch of GCC 4.0.1, BUT it doesn't run too well. The network code is thrown together and there are network/race condition issues. Also, the heap gets messed up along the way even when the RPCs get executed correctly. *sigh* Anyway, before I go into working this out bottom up and adding docs, it would be better for me to get a better idea of what the end result should be. So, here is how the framework functions now. 1. The server-side components: 1.1. The registry - this contains a collection of functions mapped by key: // make a registry where the functions are keyed by int: marshal::registry<int> reg; 1.2. The function - each function gets added to the registry so it can be accessed: int add2( int i, int j ) { return i+j; } ... reg.set<int (int, int)>(2, add2); // register function 1.3. registry_server - this will service calls from an individual client, and is associated with a particular registry. // Make a server that uses a simple connection protocol marshal::server<marshal::registry<int>, marshal::simple_acceptor> server(reg, 1097); The simple_acceptor just grants a connection to anyone that asks for it. The Acceptor is a template parameter, so different connection/authentication protocols could be used (although they should probably be in a different library, since they are not necessarily specific to RPC). 1.4. The server - the server will accept connections, and make a new registry_server object which will serve the new socket 2. Client-side components 2.1. The client marshal::client<marshal::registry<int>, marshal::simple_connector> client( ip::tcp::endpoint(ip::address::from_string("127.0.0.1"), 1097)); Negotiates a connection to the server and can service calls to the registry on the server side. 2.2. The call marshal::call<int (int, int)> call_5_6(5,6); A call now stores the parameters only, and (possibly) storage for the return value of the function. 2.3. Making the call client(2, call_5_6, marshal::call_options( marshal::call_options::return_only, marshal::call_options::async)); A call is given to the client, with a function id, the parameters stored in the marshal::call class, and call options. Call options specify: - what to marshal back (nothing, just an acknowledgement that the function has completed, just the return value, or (not implemented yet) values that can be returned through parameters) - sync (don't return until the function call has completed) vs. async (return immediately) 2.4 The results // it was an async call, wait until it has completed: while(!call_5_6.has_completed()) {} // now we can get the return value because we asked for it BOOST_CHECK(call_5_6.return_val() == 11); The test file included in the code has only a few more examples. My main questions are: * right now, the entire server-side code is running in only one thread. Should there be one thread per client? One thread per function call? Is there a "best" solution or should there be options on this? * the above schemes separates the call into - the id - the parameters - the options (what to marshal back, sync vs. async, (exception reporting to come)) It is possible that the implementation of a sync call will be substantially different then an async call (for example, an async call might use futures to return the result). Should the sync/async choice be placed with the marshal::call class (i.e., make a marshal::async_call and marshal::sync_call). I can see both advantages and disadvantages to this. * right now, the network protocol is rather clumsy - there's one or two header packets and a marshal packet in each direction (and some acknowledgement packets that I threw in to try to fix things but it didn't seem to help much)... The advantage of this approach is that the header can say how big the marshal packet is (which can vary widely, so any preallocated buffer might be too small), but changing things so that there is only one packet would probably simplify things a whole lot. I'm still getting familiar with Boost.Asio... if anyone has any thoughts on what to do I'd appreciate it. OK, that's all for now... when I grab some time I'll try to document this thing a little better and hopefully fix it in the process. Please let me know what you think... Thanks, Stjepan On 4/19/07, Scott Woods <scottw@qbik.com> wrote:
----- Original Message ----- From: "Stjepan Rajko" <stipe@asu.edu> To: <boost@lists.boost.org> Sent: Thursday, April 19, 2007 10:53 AM Subject: Re: [boost] Marshalling library
The scope of this list get daunting quickly. What is the scope for you guys?
My personal needs are pretty basic - just making a remote call and getting the results back would cut it. In terms of designing a marshalling framework, I'd propose separating out as much functionality from threading, network, shared memory, and other issues, and sticking to the actual function call and execution (for starts).
What I'm hoping is that things can be designed so that other Boost libraries can transparently address these issues through existing functionality. For example, the readable/fast encodings can be used by allowing the choice of either binary or xml (or text) serialization archives offered by Boost.Serialization. A secure channel can be established by using an SSL asio connection, etc.
How's this for a 1st cut breakdown;
* asynchronous socket manager (definitely a thread or threads, might be in asio) * asynchronous RPC manager (probably a thread) with a sync interface for application access * synchronous application thread(s)
Something like this would deliver what you are after, if in a more cumbersome way than what you might have been thinking. Maybe some magic is created if it also caters to needs such as the following;
* deriving from the base socket manager to create an optimally fast FTP server * ability to communicate with two RPC servers, one running a high-speed proprietary encoding (Boost!) to a local server the other running good old Sun RPC (is that XDR encoding?) to some remote legacy box, the location and associated encoding being transparent to the caller. * ability to switch the local server encoding between fast and readable (i.e. debug vs release) * facility to develop a monitoring and control tool for a socket manager. * ability to instantiate both a base socket manager and the dedicated FTP server in the one executable, each one taking ownership of an accessible interface or port range.
While these requirements might seem contrived, the circumstances are not so unrealistic. If you tackle the intiial goal (RPC) but deliver a collection of types and objects that provides for the needs above then I really think you would have something. Its the decomposition into the useful pieces; that's the intractable bit. If you get it right then its easy to reshuffle into new arrangements such as those above.
Hope that explains my position a bit better. The solution is not obvious to me and thats probably been apparent in my messages.
Cheers.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Hi Stjepan, Gave your code a quick look. Think I follow the flow. There are gaps in my understanding because I don't know Boost as well as you do. ----- Original Message ----- From: "Stjepan Rajko" <stipe@asu.edu> To: <boost@lists.boost.org> Sent: Monday, April 23, 2007 6:03 PM Subject: Re: [boost] Marshalling library
My main questions are:
* right now, the entire server-side code is running in only one thread. Should there be one thread per client? One thread per function call? Is there a "best" solution or should there be options on this?
The threading architecture of the server will be specific to each server. I don't think you can arbitrate on that one without coming unstuck. A minimal server will have at least one thread dedicated to the socket activity and at least one application thread. Maybe some worked examples involving typical arrangements?
* the above schemes separates the call into - the id - the parameters - the options (what to marshal back, sync vs. async, (exception reporting to come)) It is possible that the implementation of a sync call will be substantially different then an async call (for example, an async call might use futures to return the result). Should the sync/async choice be placed with the marshal::call class (i.e., make a marshal::async_call and marshal::sync_call). I can see both advantages and disadvantages to this.
If you are adhering to the RPC model I think there should be no async options (even though the underlying network activity is async). The OK packet returned for an invocation that has no return type is a synchronization, i.e. the caller knows that the function has indeed completed. While this may seem unnecessarily limiting its cleaner and more consistent (with RPC). If the user wants concurrency of server activity then they should be using an async model for their application code as well as using an async messaging lib. Bending RPC just makes it ugly and you weaken/lose the metaphor.
* right now, the network protocol is rather clumsy - there's one or two header packets and a marshal packet in each direction (and some acknowledgement packets that I threw in to try to fix things but it didn't seem to help much)... The advantage of this approach is that the header can say how big the marshal packet is (which can vary widely, so any preallocated buffer might be too small), but changing things so that there is only one packet would probably simplify things a whole lot. I'm still getting familiar with Boost.Asio... if anyone has any thoughts on what to do I'd appreciate it.
Sorry, this area is a mine field. I have my own solutions to framing of messages in the presence of fragmentation and nagling. Don't know that asio or serialization really tackles these and others directly.
OK, that's all for now... when I grab some time I'll try to document this thing a little better and hopefully fix it in the process.
Please let me know what you think...
I think you are focused on RPC (with an async twist), which was your stated intention. This means that the re-usability of the resulting socket code is at best a secondary goal. My intentions are almost perfectly inverted but I'll continue to eavesdrop and offer feedback when I can.
Thanks,
Good luck.
Hi Scott! On 4/23/07, Scott Woods <scottw@qbik.com> wrote:
The threading architecture of the server will be specific to each server. I don't think you can arbitrate on that one without coming unstuck. A minimal server will have at least one thread dedicated to the socket activity and at least one application thread.
Maybe some worked examples involving typical arrangements?
OK - I'll shoot for providing one thread for the socket activity and one application thread as a minimum, plus (eventually) additional application / function threads on request (i.e., a client could request from the server a separate application thread, and/or a separate thread for a function call).
If you are adhering to the RPC model I think there should be no async options (even though the underlying network activity is async). The OK packet returned for an invocation that has no return type is a synchronization, i.e. the caller knows that the function has indeed completed.
While this may seem unnecessarily limiting its cleaner and more consistent (with RPC). If the user wants concurrency of server activity then they should be using an async model for their application code as well as using an async messaging lib.
Bending RPC just makes it ugly and you weaken/lose the metaphor.
I agree that an async call is bound to be messy with what's available now - but a good futures implementation might provide an elegant mechanism on providing the call results when they come back. I'll focus on the sync functionality for now.
* right now, the network protocol is rather clumsy - there's one or <snip>>
Sorry, this area is a mine field. I have my own solutions to framing of messages in the presence of fragmentation and nagling. Don't know that asio or serialization really tackles these and others directly.
I fixed the protocol I was using to have a fixed size header, so I know exactly what to expect when and in what size... this made all the difference. I needed a good nights's sleep to figure out what I needed to do :-)
I think you are focused on RPC (with an async twist), which was your stated intention. This means that the re-usability of the resulting socket code is at best a secondary goal. My intentions are almost perfectly inverted but I'll continue to eavesdrop and offer feedback when I can.
As far as re-usability of the socket code, the server and client components take a template parameter that specifies the underlying connection/authentication mechanism. I implemented a very primitive pair of classes (simple_connector and simple_acceptor) which are trivial mappings onto the asio functionality - the connector asks acceptor for a socket and the acceptor sets it up, then the socket connection is used between the client and a dedicated registry_server. Making this a template parameter is meant to allow easy inclusion of other authentication schemes - I will try to add something that's a little more involved as an example (for example, an acceptor that only grants sockets to clients with certain IP addresses, or a very simple password mechanism). Anything more along these lines is probably beyond me though :( However, any socket code that was developed here could be reused for other purposes, as long as the required interface was the same. Your feedback is much appreciated! Stjepan
participants (5)
-
Hans Larsen -
Scott Woods -
Shams -
Stjepan Rajko -
Yigong Liu