[async] messages threads and networks
Hi Boost, Is anyone interested in a framework for asynchronous programming? This is based on the thinking contained in; * SDL (http://www.sdl-forum.org/SDL/index.htm) * Active Objects (http://www.cs.wustl.edu/~schmidt/PDF/Act-Obj.pdf) SDL is the ITU standard for specification of signaling systems. AO is a pattern for concurrent programming. These have been combined to create an environment where objects exchange messages, possibly between threads or even over networks. Objects are created simply; id = create<object_type>(); and once created, anything can be sent to it; send( anything, id ); The framework takes care of threading, network addresses, queueing, encoding and optimal I/O, as appropriate. Threads are equally simple to create; p = start<message_processor>(); and objects can be assigned to specific threads at creation time; id2 = create<object_type>( p ); A network server can be created as simply as; listen = create<socket_listen>(); server = create<socket_server<my_server> >(); service operational( "loopback:5500", server ); send( operational, listen ); Connecting is equally simple; connect = create<socket_connect>(); client = create<my_client>(); service operational( "localhost:5500", client ); send( operational, connect );
From that point on, client and server can exchange messages using the send primitive without any knowledge that messages are actually travelling across a network. There is in fact no difference between inter-thread messaging and inter-processes (i.e. for the application).
Feel free to poke around in the software at; http://groups.google.co.nz/group/pact-serialization/web I am interested in a Boost review of this library but suspect it is too large. It also overlaps with several existing boost components. But good to get further opinions. Cheers, Scott
----- Original Message ----- From: "Scott Woods" <scott@wdesignnz.co.nz> To: <boost@lists.boost.org> Sent: Saturday, April 24, 2010 2:26 AM Subject: [boost] [async] messages threads and networks
Hi Boost,
Is anyone interested in a framework for asynchronous programming? This is based on the thinking contained in;
* SDL (http://www.sdl-forum.org/SDL/index.htm) * Active Objects (http://www.cs.wustl.edu/~schmidt/PDF/Act-Obj.pdf)
SDL is the ITU standard for specification of signaling systems. AO is a pattern for concurrent programming. These have been combined to create an environment where objects exchange messages, possibly between threads or even over networks. Objects are created simply;
id = create<object_type>();
and once created, anything can be sent to it;
send( anything, id );
The framework takes care of threading, network addresses, queueing, encoding and optimal I/O, as appropriate. Threads are equally simple to create;
p = start<message_processor>();
and objects can be assigned to specific threads at creation time;
id2 = create<object_type>( p );
A network server can be created as simply as;
listen = create<socket_listen>(); server = create<socket_server<my_server> >();
service operational( "loopback:5500", server ); send( operational, listen );
Connecting is equally simple;
connect = create<socket_connect>(); client = create<my_client>();
service operational( "localhost:5500", client ); send( operational, connect );
From that point on, client and server can exchange messages using the send primitive without any knowledge that messages are actually travelling across a network. There is in fact no difference between inter-thread messaging and inter-processes (i.e. for the application).
Feel free to poke around in the software at;
http://groups.google.co.nz/group/pact-serialization/web
I am interested in a Boost review of this library but suspect it is too large. It also overlaps with several existing boost components. But good to get further opinions.
I have read your last posts about your mega library. I'm interested on the kind of services the library could provide. I don't think people would spent time to inspect code without the Boost license that reinvent a lot of Boost libraries. Is there more documentation that the one contained on the link? Could you show how the objects read the messages? Note that async is already used by a library under developement https://svn.boost.org/trac/boost/wiki/LibrariesUnderConstruction#Boost.Async A last point, to request interest on a library maybe the best could be to put "Request for interest" in the title, and show some motivating use cases. Best, _____________________ Vicente Juan Botet Escribá http://viboes.blogspot.com/
I am interested in a Boost review of this library but suspect it is too large. It also overlaps with several existing boost components. But good to get further opinions.
I have read your last posts about your mega library. I'm interested on the kind of services the library could provide.
Perhaps MegaMessaging.lib?
I don't think people would spent time to inspect code without the Boost license that reinvent a lot of Boost libraries.
Sorry this is a little confusing. I can understand that Boosters will be less inclined to review a library that replicates some existing Boost components. Even if there is differentiation in the implementations I am resigned to the natural reluctance. But it is the sum of the parts that I have hopes for. I am willing to substitute reinventions with the appropriate Boost component as necessary. But you also mention the lack of a Boost license. Are you suggesting that all code offered for review must first hand over all rights? With some effort I can see justification for such a position but it leaves me (and others like me) with a difficult question.
Is there more documentation that the one contained on the link?
Yes but they are internal and extremely abstract. Peer reviews have suggested that these docs are probably not useful to users of the library. This library is an internal development. It is/will be used for product development. All the materials that you have seen were created at the expense of that development. And in the full knowledge that organizations such as Boost are less than likely to accept/adopt/integrate.
Could you show how the objects read the messages?
I'm not entirely sure what you are asking for. If I give a quick description perhaps you can highlight the section of interest? Objects are read by calls to codec_base::decode (lhc/codec_base.h). This is a virtual member that invokes the parsing implementation for whichever encoding type has been selected. Those implementations are generally a combination of lexical analysis (manually defined state tables) and actual parsing (simple stack manipulation for simple encodings, BNF tools for XML). Some related files are; * lhc/lean_text.h - declaration of Lean Text codec incl. output of that encoding. * lhc/word_parser.h - declaration of simple object parser * lhc_parser/word_markup.h - declaration of Word Markup (XML) codec incl output You can see how the lex and parse activity is managed in codec<T>::decode (lhc/codec.h). The whole process is made much more complex by the requirement for encoding independence. The solution is based on a stack that deals with (top to bottom) I/O of objects, buffering strategy and block device operations. The proper use of codecs is demonstrated in load (lhc/codec_object.h). But maybe your question is about how messages are detected on network connections, routed and dispatched? There is a thread object (lhc/socket_processor.h) that is dedicated to the handling of winsock FD_xxxx events and client winsock calls (listen, connect). It arranges for creation of session objects (lhc/socket_transport.h) that manage actual I/O. These objects are the network equivalent of the objects created by load for reading of flies. Both socket_transport and load (actually file_transfer::read_existing) ultimately call codec_base::decode. The dispatching of messages once they reach an object is demonstrated in the samples, e.g. sample-t_trip-inter-process-client\trip_controller.cpp. Does this roadmap help to focus your question?
Note that async is already used by a library under developement https://svn.boost.org/trac/boost/wiki/LibrariesUnderConstruction#Boost.Async
Ah OK. Will check this out (pun unintended).
A last point, to request interest on a library maybe the best could be to put "Request for interest" in the title, and show some motivating use cases.
Yes. Thanks for this. I will try out a few ideas while also considering your other points. The level of interest so far is a concern. Not unexpected but silly to ignore. Cheers, Scott
On Sat, Apr 24, 2010 at 2:28 PM, Scott Woods <scott@wdesignnz.co.nz> wrote:
/* snip */
This pact library looks interesting from a cursory read of the 'about' page. Going to look through it and make comments. On reading http://groups.google.com/group/pact-serialization/web/a-shortened-tour: With this example: """ #include "pact/cursor.h" struct person { string name; int number; }; you will need the following transform functions (probably in the same header file): inline void operator<<( cursor &parent, const person &p ) { cursor &c = parent.output( 2 ); c << p.name; c << p.number; } inline void operator>>( cursor &parent, person &p ) { cursor &c = parent.input( 2 ); c >> p.name; c >> p.number; } Somewhere in your application, probably in person.cpp, you will also need the following code fragments: #include "pact/type.h" COMPLEX_TYPE( person, "Organization.Product.Person" ) { COMPLEX_MEMBER( "Name", name ); COMPLEX_MEMBER( "Number", number ); } """ If you were using boost, all of that could have been replaced with something like this: """ #include <boost/fusion/include/adapt_struct.hpp> struct person { string name; int number; }; BOOST_FUSION_ADAPT_STRUCT( person, (string,name) (int,number) ) // Probably something to register a name for it too, wrap it all up together """ And that is all in the header, no CPP usage needed. Your system would just register to handle the fusion types and you can suddenly everything from structs, C static arrays, stl static containers (including boost::array), and much more without needing to specialize for any of them. Also noticed that your enum registrations require a single CPP side along with the header side, would it not be a lot easier to just register in the header, like how fusion does it? You could do something like use Boost.MPL to save the enums and their values in a type declaration instead, needing no operator<</>> overloads or anything, and you handle the enum registration type. Based on this sentence: """ During the "load" function Pact automatically detects the non-existence of an expected file and instead of failing with an error, it writes the current value of the object into the indicated file. That's right; a function that appears to be an input function also performs output. """ Would load not be better served by renaming to something consistent, like load_or_create (bad name) or something better (sync_to_filename?)? Heh, nice names for the encodings: """ * Lean Text – text, readable, portable * Neat Bytes – binary, portable * Byte Blast – binary, non-portable * Word Markup – text, readable with appropriate tools, portable, symbolic. """ The http://groups.google.com/group/pact-serialization/web/the-extended-journey link does not exist. Based on the description page, the Async I/O work and messaging looks interesting. Have not played with any actual source though, and do not see and code examples page (maybe that extended journey missing page?). The type system could easily be vastly simplified with a few Boost libraries. Will look at it later when I get more time, looks interesting though, but definitely need more information, speed, a lot more code examples, etc...
On Sat, Apr 24, 2010 at 7:46 PM, OvermindDL1 <overminddl1@gmail.com> wrote:
The http://groups.google.com/group/pact-serialization/web/the-extended-journey link does not exist.
Ah, it seems that the latest version returns a 404, but the next-to-last version comes up: http://groups.google.com/group/pact-serialization/web/the-extended-journey?v...
Hi,
#include <boost/fusion/include/adapt_struct.hpp>
struct person { string name; int number; };
BOOST_FUSION_ADAPT_STRUCT( person, (string,name) (int,number) ) // Probably something to register a name for it too, wrap it all up together """
And that is all in the header, no CPP usage needed. Your system would just register to handle the fusion types and you can suddenly everything from structs, C static arrays, stl static containers (including boost::array), and much more without needing to specialize for any of them.
This basically looks appealing. I have one question though. I jumped through a few hoops so that registration does not need to mention the type - it is auto-generated from the member name. I would be happy to leverage the work of Fusion but sad to lose the correctness guarantee offered by auto-generating the type. Can Fusion offer that as well? As the types get more complicated the chances of a registration error (i.e. an incorrectly associated type ) move from "possible" to "expected".
Also noticed that your enum registrations require a single CPP side along with the header side, would it not be a lot easier to just register in the header, like how fusion does it? You could do something like use Boost.MPL to save the enums and their values in a type declaration instead, needing no operator<</>> overloads or anything, and you handle the enum registration type.
Again, this sounds good. From the words you use it sounds like the Fusion registrations are a compile-time facility? Pact registrations create information in a runtime database. That database is searched/iterated by parts of the library (such as the one that generates an XML encoding), based on a type id. There is no compile-time type information available to that part of the library. The type database also needs to be something that can be sent around a network. Are MPL/Fusion-based solutions still a good direction given these requirements?
Based on this sentence: """ During the "load" function Pact automatically detects the non-existence of an expected file and instead of failing with an error, it writes the current value of the object into the indicated file. That's right; a function that appears to be an input function also performs output. """ Would load not be better served by renaming to something consistent, like load_or_create (bad name) or something better (sync_to_filename?)?
Understand your point. The full name is of course "pact::load". Being a verb in its own namespace I was hoping it could define its own specific meaning. Happy to concede to a majority opinion that I was misdirected.
Heh, nice names for the encodings: """ * Lean Text – text, readable, portable * Neat Bytes – binary, portable * Byte Blast – binary, non-portable * Word Markup – text, readable with appropriate tools, portable, symbolic. """
Well - a bit cutesy but alternatives were unremarkable. Check the last section of the "extended journey" (Why the silly encoding names) for an attempt at absolution.
The http://groups.google.com/group/pact-serialization/web/the-extended-journey link does not exist.
Very unfortunate. I have had a few problems with links in Google Groups. They seem to be a symptom of not being signed in, i.e. do you have a Google identity? This is the subject of the first entry in the group discussions. Quite why it should make any difference is beyond me. There is nothing in any of the GG documentation that suggests non-signed-in (unsigned?) viewing gets a lower class of service. I did discover a crude fix so I will look at this again.
Based on the description page, the Async I/O work and messaging looks interesting. Have not played with any actual source though, and do not see and code examples page (maybe that extended journey missing page?). The type system could easily be vastly simplified with a few Boost libraries.
Will look at it later when I get more time, looks interesting though, but definitely need more information, speed, a lot more code examples,
All the documentation you have seen has been written specifically to present the library to parties such as Boost (i.e. you). From initial responses I'm guessing that the time and energy consumed by that effort is unlikely to result in anything. Thats a lot of effort that could have been directed into other development. Apologies for the small rant but requests for documentation are endless. I deliberately compiled a complete list of supporting documents and then chopped it down to the set achievable writing at 2am to avoid family obligations. I would love to finish the full set but there must be a proper opportunity. More speed? Always interested in making software faster but where is it slow? I also enjoy code examples but have you downloaded the samples solution yet? You are asking for more code beyond that provided in the samples? Thanks, Scott
On Sat, Apr 24, 2010 at 9:31 PM, Scott Woods <scott@wdesignnz.co.nz> wrote:
#include <boost/fusion/include/adapt_struct.hpp>
struct person { string name; int number; };
BOOST_FUSION_ADAPT_STRUCT( person, (string,name) (int,number) ) // Probably something to register a name for it too, wrap it all up together """
And that is all in the header, no CPP usage needed. Your system would just register to handle the fusion types and you can suddenly everything from structs, C static arrays, stl static containers (including boost::array), and much more without needing to specialize for any of them.
This basically looks appealing. I have one question though. I jumped through a few hoops so that registration does not need to mention the type - it is auto-generated from the member name. I would be happy to leverage the work of Fusion but sad to lose the correctness guarantee offered by auto-generating the type. Can Fusion offer that as well?
As the types get more complicated the chances of a registration error (i.e. an incorrectly associated type ) move from "possible" to "expected".
Fusion is completely compile-time in how it can assign 'views' to types, but your type registration is independent, hence why making your own macro that handle that and the fusion view adaptation would be best, it would negate the need for your operator>>/<< functions is all it is, would vastly simplify code and reduce chance of errors. On Sat, Apr 24, 2010 at 9:31 PM, Scott Woods <scott@wdesignnz.co.nz> wrote:
Also noticed that your enum registrations require a single CPP side along with the header side, would it not be a lot easier to just register in the header, like how fusion does it? You could do something like use Boost.MPL to save the enums and their values in a type declaration instead, needing no operator<</>> overloads or anything, and you handle the enum registration type.
Again, this sounds good. From the words you use it sounds like the Fusion registrations are a compile-time facility? Pact registrations create information in a runtime database. That database is searched/iterated by parts of the library (such as the one that generates an XML encoding), based on a type id. There is no compile-time type information available to that part of the library. The type database also needs to be something that can be sent around a network. Are MPL/Fusion-based solutions still a good direction given these requirements?
See above response, you could easily make the values saved in the types at compile-time, and still use your type system to determine what is what, would provide a speed-boost at run-time while simplifying code and reducing the chance of errors. On Sat, Apr 24, 2010 at 9:31 PM, Scott Woods <scott@wdesignnz.co.nz> wrote:
The http://groups.google.com/group/pact-serialization/web/the-extended-journey link does not exist.
Very unfortunate. I have had a few problems with links in Google Groups. They seem to be a symptom of not being signed in, i.e. do you have a Google identity? This is the subject of the first entry in the group discussions.
Quite why it should make any difference is beyond me. There is nothing in any of the GG documentation that suggests non-signed-in (unsigned?) viewing gets a lower class of service.
I did discover a crude fix so I will look at this again.
I was logged in, unsure why it shows 404 for the most recent version, but I was able to view the second to most recent easily as shown by my next sent email after that first one. On Sat, Apr 24, 2010 at 9:31 PM, Scott Woods <scott@wdesignnz.co.nz> wrote:
Based on the description page, the Async I/O work and messaging looks interesting. Have not played with any actual source though, and do not see and code examples page (maybe that extended journey missing page?). The type system could easily be vastly simplified with a few Boost libraries.
Will look at it later when I get more time, looks interesting though, but definitely need more information, speed, a lot more code examples,
All the documentation you have seen has been written specifically to present the library to parties such as Boost (i.e. you). From initial responses I'm guessing that the time and energy consumed by that effort is unlikely to result in anything. Thats a lot of effort that could have been directed into other development.
Once I was able to view the missing page (at least the older version of it) it clarified many things and became sufficient in my opinion, just need to make sure links work. :) On Sat, Apr 24, 2010 at 9:31 PM, Scott Woods <scott@wdesignnz.co.nz> wrote:
Apologies for the small rant but requests for documentation are endless. I deliberately compiled a complete list of supporting documents and then chopped it down to the set achievable writing at 2am to avoid family obligations. I would love to finish the full set but there must be a proper opportunity.
Only had a request for documentation because that massive extended tutorial page was missing, once I was able to view it, it all worked much better. On Sat, Apr 24, 2010 at 9:31 PM, Scott Woods <scott@wdesignnz.co.nz> wrote:
More speed? Always interested in making software faster but where is it slow?
I also enjoy code examples but have you downloaded the samples solution yet? You are asking for more code beyond that provided in the samples?
I was not asking if it was slow, just asking if you had speed reports and comparisons, through-put tests for network (perhaps in comparison to other libs, Raknet for example, or Erlang too for massage passing tests?), etc... Cannot try to compile it right now, my main computer is down, and Visual Studio does not work on PocketPC/Arm.
This basically looks appealing. I have one question though. I jumped through a few hoops so that registration does not need to mention the type - it is auto-generated from the member name. I would be happy to leverage the work of Fusion but sad to lose the correctness guarantee offered by auto-generating the type. Can Fusion offer that as well?
As the types get more complicated the chances of a registration error (i.e. an incorrectly associated type ) move from "possible" to "expected".
Fusion is completely compile-time in how it can assign 'views' to types, but your type registration is independent, hence why making your own macro that handle that and the fusion view adaptation would be best, it would negate the need for your operator>>/<< functions is all it is, would vastly simplify code and reduce chance of errors.
Yes, I can see real benefit. Always preferred to have Boost but didnt want to impose prerequisite on potential users. Perhaps I was wrong.
More speed? Always interested in making software faster but where is it slow?
I also enjoy code examples but have you downloaded the samples solution yet? You are asking for more code beyond that provided in the samples?
I was not asking if it was slow, just asking if you had speed reports and comparisons, through-put tests for network (perhaps in comparison to other libs, Raknet for example, or Erlang too for massage passing tests?), etc...
Ah yes. I spent substantial time on performance testing. Would love to carry out comparison testing. Time and technical resources are the only limiting factors. If you are truly interested in statistics I can send some results to you directly. Reviewing the numbers was both interesting and confusing. One informal performance metric is the number of times a message can be sent between two points; * same thread, same process = 220k/second * different threads, same process = 120k/second * different threads, different process, same PC = 11k/second * different threads, different process, different PC = 1100/second (54Mb wireless)
Cannot try to compile it right now, my main computer is down, and Visual Studio does not work on PocketPC/Arm.
Time for an iPhone? OK it doesn't run there either but just having one would make you feel better. Cheers, Scott
----- Original Message ----- From: "OvermindDL1" <overminddl1@gmail.com> To: <boost@lists.boost.org> Sent: Sunday, April 25, 2010 3:46 AM Subject: Re: [boost] [async] messages threads and networks
On Sat, Apr 24, 2010 at 2:28 PM, Scott Woods <scott@wdesignnz.co.nz> wrote:
/* snip */
This pact library looks interesting from a cursory read of the 'about' page. Going to look through it and make comments.
On reading http://groups.google.com/group/pact-serialization/web/a-shortened-tour:
The http://groups.google.com/group/pact-serialization/web/the-extended-journey link does not exist.
The first time I connect to this page I got something as "To be written". Later on I got the same 404 error and I was connected to gmail. Now I'm able to read it. I'll post after reading it. Best, Vicente
Disturbing but thanks for info. If it's currently working I'm wary of touching it. Also noticed your mail address. Are you aware that the Boost mailing list has known problems with gmail-based descriptions. At least, I was having problems on a previous subscription and that was the feedback from the list admin. And that's the last off-topic post from me :-)
The http://groups.google.com/group/pact-serialization/web/the-extended-journey link does not exist.
The first time I connect to this page I got something as "To be written". Later on I got the same 404 error and I was connected to gmail. Now I'm able to read it. I'll post after reading it.
Best, Vicente
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
----- Original Message ----- From: "Scott Woods" <scott@wdesignnz.co.nz> To: <boost@lists.boost.org> Sent: Saturday, April 24, 2010 10:28 PM Subject: Re: [boost] [async] messages threads and networks
I am interested in a Boost review of this library but suspect it is too large. It also overlaps with several existing boost components. But good to get further opinions.
I have read your last posts about your mega library. I'm interested on the kind of services the library could provide.
Perhaps MegaMessaging.lib?
I don't think people would spent time to inspect code without the Boost license that reinvent a lot of Boost libraries.
Sorry this is a little confusing. I can understand that Boosters will be less inclined to review a library that replicates some existing Boost components. Even if there is differentiation in the implementations I am resigned to the natural reluctance.
But it is the sum of the parts that I have hopes for. I am willing to substitute reinventions with the appropriate Boost component as necessary.
Scott, using Boost to implement your own library will help to get more people interested in your library (at least for inclusion in Boost). This probes that you know which parts od Boost can be reused without reinventing the wheel, the suers needs to lears less things, as he possible already knows the reused parts.
But you also mention the lack of a Boost license. Are you suggesting that all code offered for review must first hand over all rights? With some effort I can see justification for such a position but it leaves me (and others like me) with a difficult question.
No this is not absolutely necesary. Reviewing a lot of code that is not released with BSL could be a waste of time if at the end you decide to not propose your library to Boost. The fact you release it under BSL is already a good sign that you want to make it part od Boost.
Is there more documentation that the one contained on the link?
Yes but they are internal and extremely abstract. Peer reviews have suggested that these docs are probably not useful to users of the library.
This question was related to the missing link "The Extended Journey" as the communication and networking part was in ;-) Anyway, is there a Reference page of all the files classes functions ... available?
This library is an internal development. It is/will be used for product development. All the materials that you have seen were created at the expense of that development. And in the full knowledge that organizations such as Boost are less than likely to accept/adopt/integrate.
Could you show how the objects read the messages?
I'm not entirely sure what you are asking for. If I give a quick description perhaps you can highlight the section of interest?
I see now how this works in the "The Extended Journey" link.
Does this roadmap help to focus your question? <snip> Yes it helps.
A last point, to request interest on a library maybe the best could be to put "Request for interest" in the title, and show some motivating use cases.
Yes. Thanks for this. I will try out a few ideas while also considering your other points. The level of interest so far is a concern. Not unexpected but silly to ignore.
It is up to you to make interest. This is part of the work to get a library on Boost. My experience is that this part is even harder than implementing your library, but the feedback will make your library much better. Another thing that helps could be to been able to build and test your library with Boost.Build (bjam) so we can evaluate the portability of your library (If I have understood you have only tested using a windows platform with MSVC). Note that a library could start to be considered as portable when it is implemented on at least two platforms (Win|Linux) with two compilers (gcc|msvc). In addition there are some Boosters that have no access to MSVC, so this people will be unable to run the example programs. Best, _____________________ Vicente Juan Botet Escribá http://viboes.blogspot.com/
[snip]
Scott, using Boost to implement your own library will help to get more people interested in your library (at least for inclusion in Boost). This probes that you know which parts od Boost can be reused without reinventing the wheel, the suers needs to lears less things, as he possible already knows the reused parts.
I appreciate the value of what you say. It's perhaps interesting to note that I have been developing this library since before most of the "parts ... Boost" have existed, i.e. initially at least there was no actual avoidance of Boost components. Also, in some cases I had goals that were simply not included in Boost implementations. Boost.Serialization does not have runtime encoding selection. Boost.Serialization will never deploy seamlessly inside an async messaging system without additional framing work. Apparent re-inventions are not always re-inventions. Lastly there is also a broader issue. My collection of parts cooperate inside a framework, i.e. my unique id mechanism has requirements imposed by the runtime type system and my discriminated variant has requirements from the encoding layer of my I/O stack. Boost does not have anything like such a framework. In fact Boost components have a strong "generic" flavor about them. They tend to shrug off any requirements coming from a specific architectural view. I have observed the review process for several parts of Boost that seemed relevant to my library. In some cases I have offered input from the perspective of my library, including input on singletons, statechart (i.e. fsm), logging and I suspect, asio. Except for singletons that input is not reflected in Boost components. I think all parties involved behaved perfectly reasonably and that the resulting components are strong, generic software tools. But they do not meet the requirements of my framework. The process of Boost-ifying my library should be interesting.
But you also mention the lack of a Boost license. Are you suggesting that all code offered for review must first hand over all rights? With some effort I can see justification for such a position but it leaves me (and others like me) with a difficult question.
No this is not absolutely necesary. Reviewing a lot of code that is not released with BSL could be a waste of time if at the end you decide to not propose your library to Boost. The fact you release it under BSL is already a good sign that you want to make it part od Boost.
Thanks Vicente, this makes good sense. [snip]
It is up to you to make interest. This is part of the work to get a library on Boost. My experience is that this part is even harder than implementing your library, but the feedback will make your library much better.
Understood. I'm still considering the use cases.
Another thing that helps could be to been able to build and test your library with Boost.Build (bjam) so we can evaluate the portability of your library (If I have understood you have only tested using a windows platform with MSVC). Note that a library could start to be considered as portable when it is implemented on at least two platforms (Win|Linux) with two compilers (gcc|msvc). In addition there are some Boosters that have no access to MSVC, so this people will be unable to run the example programs.
Also understood. Cheers, Scott
Hi Sohail,
Also, in some cases I had goals that were simply not included in Boost implementations. Boost.Serialization does not have runtime encoding selection.
The polymorphic archives do allow this.
They assist you towards that goal. But considering the two libraries somehow equivalent in this area would be misleading. In the docs it describes the motivation/solution according to polymorphic archives. It describes the construction of sets of archive implementations, e.g. you can build an application with text and wide text or text and binary, etc. It also shows the selection of a specific archive (e.g. text); boost::archive::polymorphic_text_oarchive oa(ss); boost::archive::polymorphic_oarchive & oa_interface = oa; How does the user of the application or even the developer select a different encoding? Given the example it would take a recompilation. The Pact library selection is innately runtime strings; load( object, "file", "Neat Bytes" ); While changing the line above would also require a recompilation there is an obvious potential to pass strings into an application from command line arguments, environment variables or configuration data. The Pact library implements a "codec_factory" that takes care of all of that. This might seem trivial but there are pitfalls and some further issues. It's not enough to say that Boost.Serialization has the potential to do something similar (which is still unclear). Until it is a part of Boost.Serialization you cannot rely on any /standard/ collection of encodings and /standard/ naming of encodings. It would be interesting to get a survey of all uses of boost::archive::polymorphic. I'm sure there will be a variety of deployments - different sets of archives, different runtime selector types (enums, strings) and different values for those types. Now imagine trying to get these deployments to agree to a Boost /standard/ and you might see what I am trying to explain. Any software component developed in Pact can use any encoding available at any site. All the I/O primitives have the necessary selection parameters, the parameter type is fixed and the basic set of encodings available are standard. Using Boost you would have to ship a DLL with your components. And the user of your component may have to accept this arrangement for multiple components. The last point is how Boost.Serialization handles UDTs/ADTs. My understanding of this area is sketchy - it appears to rely on pointers to methods for archiving of such objects. The polymorphic interface passes a bald object through to the archive implementation and I'm guessing, onto a registered method pointer. The Pact library converts all objects including UDTs into a generic form before they enter the encoding machinery and there are no registered per-type method pointers. Once an object enters the "Pact world" it is type-less.Objects can travel between threads and across networks. As they pass between processes they can be serialized and de-serialized multiple times and in several different formats (i.e. archives/encodings). Achieving the same mobility with Boost.Serialization is at the very least difficult. Thanks for highlighting the issue, Scott
Scott Woods wrote:
Hi Sohail,
Also, in some cases I had goals that were simply not included in Boost implementations. Boost.Serialization does not have runtime encoding selection.
The polymorphic archives do allow this.
It also shows the selection of a specific archive (e.g. text);
boost::archive::polymorphic_text_oarchive oa(ss); boost::archive::polymorphic_oarchive & oa_interface = oa; How does the user of the application or even the developer select a different encoding? Given the example it would take a recompilation.
nope. string encoding; polymorphic_oarchve * poa; if( encoding = "Neat Bytes") poa = boost::archive::polymorphic_text_oarchive oa(ss); else poa = boost::archive::polymorphic_binary_oarchive oa(ss); .. *poa << ....
The last point is how Boost.Serialization handles UDTs/ADTs. My understanding of this area is sketchy - it appears to rely on pointers to methods for archiving of such objects.
nope - uses templates with type parameters.
Achieving the same mobility with Boost.Serialization is at the very least difficult.
Well, it would require a careful study of the documentation and understanding of how to use it. Robert Ramey
On 4/25/2010 6:48 PM, Robert Ramey wrote:
polymorphic_oarchve * poa; if( encoding = "Neat Bytes") poa = boost::archive::polymorphic_text_oarchive oa(ss); else poa = boost::archive::polymorphic_binary_oarchive oa(ss); .. *poa<< ....
I think a couple of new statements are missing, but that is how I would choose at run time.
encoding? Given the example it would take a recompilation.
nope.
Well "given the example" - it would. You also make no mention of my related point that there is no standard archive registry.
string encoding;
polymorphic_oarchve * poa; if( encoding = "Neat Bytes") poa = boost::archive::polymorphic_text_oarchive oa(ss); else poa = boost::archive::polymorphic_binary_oarchive oa(ss); .. *poa << ....
Fair enough; but you have presented different code. I made the same point about a Pact version of a similar operation. It was never about whether users of Boost.Serialization could add the code you have added. It was about the fact that there is no Boost.ArchiveFactory.
The last point is how Boost.Serialization handles UDTs/ADTs. My understanding of this area is sketchy - it appears to rely on pointers to methods for archiving of such objects.
nope - uses templates with type parameters.
Apologies. But not sure that that finalizes anything.
Achieving the same mobility with Boost.Serialization is at the very least difficult.
Well, it would require a careful study of the documentation and understanding of how to use it.
OK. I recently spent about a month attempting to determine which was the better approach to serialization, given a set of 5 different solutions. The harder I looked the further I got from an answer. At the important levels they are apples and oranges. When someone chooses a serialization solution out of Protobuf, Boost.Serialization and s11n they are choosing much more than a function that converts an application type to a sequence of bytes. They are choosing a model of operation for the related serialization activity. My response to Sohail was an attempt at saying that Pact deliberately targets runtime selection of encoding and the associated mobility of encoded data. This is within the "model of operation" for Pact.- out of the box. If I am wrong to suggest that Boost.Serialization had different design goals and consequently does not deliver this (/out of the box/) then I am happy to retract. Cheers, Scott
Scott Woods wrote:
encoding? Given the example it would take a recompilation.
nope.
Well "given the example" - it would.
You also make no mention of my related point that there is no standard archive registry.
Honestly, I don't know what this would be and what utility it might have. Archive instatiations are local objects only.
I made the same point about a Pact version of a similar operation. It was never about whether users of Boost.Serialization could add the code you have added. It was about the fact that there is no Boost.ArchiveFactory.
I don't seen the need for any archive factory. In nine years no one has asked for such a thing either. Either because no one needs one, or making one is such trivial task.
At the important levels they are apples and oranges. When someone chooses a serialization solution out of Protobuf, Boost.Serialization and s11n they are choosing much more than a function that converts an application type to a sequence of bytes.
Hmm - I would say just the opposite. A serialization library derives its utility from it's clearly defined narrow purpose. It is for this reason that it can be used in a variety of contexts. It has been used for persistence, marshalling (e.g. see MPI library) and probably others. It can be used with any stream type. Stream types can use any encoding and (almost) any protocol.
They are choosing a model of operation for the related serialization activity.
Not here - they can plug in the other parts according to their requirements.
My response to Sohail was an attempt at saying that Pact deliberately targets runtime selection of encoding and the associated mobility of encoded data. This is within the "model of operation" for Pact.- out of the box. If I am wrong to suggest that Boost.Serialization had different design goals and consequently does not deliver this (/out of the box/) then I am happy to retract.
I think your documentation should focus more on what your library does rather than what you believe other libraries can't do. I did read through the links you posted and didn't think I could say much that would be constructive. But now that I've been sucked into the discussion, I would make the following comment. I think that your proposal suffers from a common failing that is shares with many proposals received here. It is focused on solving a fairly specific programming problem. As the work progresses, more and more "features" have to be added to make it's application wider. This often seen in commercial products which continually get bigger and harder to use. This turns out to be the exact wrong approach. A good library is simple to explain and use. When a feature is needed, it should be obvious how to compose the particular library with another one. For an example of a good library - look at iterators: a) iterator_facade b) iterator_adaptor examples filter_iterator counting_iterator .. and finally, iterators_adaptors are composable so that one can "stick" them together without re-writting anything. If I want to add some special facility (e.g. insert a space every 4 characters) I can make an iterator which does that - and compose with all the others. That's power!!! Another example - *nix text processsing tools. After 30 years grep is probably the single most frequently invoked program on the planet !!! And it's composable with all the other *nix tools. Compare this with microsoft word a behemouth which I would have to read the manual/help for a day just get the equivalent of "grep -i Ramey <my_file" | count to how many time I've mentioned my own name in this post. The classic example: algebra - four operations + ( and ) and a high school student can express a calculation which would have take euclid pages to describe. That's Power!!! That is what we should strive for. Those should be our inspiration. I realize this thinking is out of style and that I sort of got carried away. Sorry, but I feel better now. Robert Ramey
Hi,
I made the same point about a Pact version of a similar operation. It was never about whether users of Boost.Serialization could add the code you have added. It was about the fact that there is no Boost.ArchiveFactory.
I don't seen the need for any archive factory. In nine years no one has asked for such a thing either. Either because no one needs one, or making one is such trivial task.
At the important levels they are apples and oranges. When someone chooses a serialization solution out of Protobuf, Boost.Serialization and s11n they are choosing much more than a function that converts an application type to a sequence of bytes.
Hmm - I would say just the opposite. A serialization library derives its utility from it's clearly defined narrow purpose. It is for this reason that it can be used in a variety of contexts. It has been used for persistence, marshalling (e.g. see MPI library) and probably others. It can be used with any stream type. Stream types can use any encoding and (almost) any protocol.
I respect the noble calling. For some of the solutions mentioned I think that approach has been to the expense of initiatives involving combinations of components. For example, with a certain approach to the decoding phase of serialization there is no need for the framing machinery that has been a subject on several support forums including Boost. That may be as part of network messaging or an attempt to store a sequence of mixed-type data items in a file. You mention the MPI library. From the section "Separating structure from the content"; <quote> When communicating data types over MPI that are not fundamental to MPI (such as strings, lists, and user-defined data types), Boost.MPI must first serialize these data types into a buffer and then communicate them; the receiver then copies the results into a buffer before deserializing into an object on the other end. For some data types, this overhead can be eliminated by using is_mpi_datatype. However, variable-length data types such as strings and lists cannot be MPI data types. </quote> Why is this special handling so common around network messaging and files of data sequences? Isn't there merit in trying to fix this at the root cause? <excerpts>
I think your documentation should focus more on what your library does rather than what you believe other libraries can't do.
I think that your proposal suffers from a common failing that is shares with many proposals received here.
This turns out to be the exact wrong approach.
For an example of a good library - look at iterators: all the others. That's power!!!
After 30 years grep is probably the single most frequently invoked program on the planet !!!
have take euclid pages to describe. That's Power!!!
That is what we should strive for. Those should be our inspiration. I realize this thinking is out of style and that I sort of got carried away. Sorry, but I feel better now. </excerpts>
I should hope so! :-) Love the passion. Still there after nine years of Boost.Serialization? My own passions are actually parsing technologies and network protocols. Because of years in the telephony/networking field I was forced (yeah; kicking and biting) to adopt SDL. In the end I had to concede it was a good thing. For a certain flavour of work it's now the only hammer I keep in the toolbox. Pact is an implementation of SDL (with a dusting of Active Objects). It's been the vehicle for indulging my own coding passions for several years. It's only recently achieved the completeness that is hopefully apparent from documentation and samples. It is a minimal implementation of an international standard, not a behemoth staggering under the weight of a bloated feature set. SDL is not an iterator or a grep or a VM supporting a half-dozen algebraic opcodes. As far as my failure to incorporate Boost components goes; that is both a little sad and a little embarrassing. In some cases potential components did not yet exist and in other cases I provided feedback hoping to move designs in directions indicated by SDL. In the case of Spirit I was deeply motivated to deploy. But along with it's blindingly clever use of overloading it also accepts a constrained set of grammars and has an input sytem not amenable to change. Authors of Spirit should not take insult from this as eventually Bison did not allow the necessary changes either. The point of that last paragraph is that if I had committed to implementing Pact with only Boost components I would still be discussing the proper feature set of Boost serialization (imagine that, for the last 9 years). And the feature sets of other components. And there would be no Pact. Genuine thanks, Scott
participants (5)
-
OvermindDL1 -
Robert Ramey -
Scott Woods -
Sohail Somani -
vicente.botet