Robert Ramey said: (by the date of Sat, 7 Oct 2006 17:12:18 -0700)

...

Jeremy Pack wrote:

...

It is, basically, a library for building plugins.

Note the serialization library contains code which handles essentially same problem domain. Its under extended_type_info. It even has its own documentation. However some of the functionality is mixed into the serialization library.

It has the following features:

no common base class required for objects created by factory registration by GUID usage of rtti optional object don't have to be default constructible class definitons can be DLLs, - static templates refer to DLLS as necessary only if the derived class GUID is explicitly referred to.

So I have examined the extended_type_info docs, first a note about few typos there, then a real question :) 1. a typo? I can't understand this sentence in Miscellaneous/extended_type_info/features: "The first is a global type table. it and has with an entry for each type used." 2. missing A,B,C ;) in Miscellaneous/void_cast: "void_cast_register(); void_cast_register(); automatically derives the fact that A can be upcast to C and vice-versa." 3. "case studies"/shared_ptr revisited: "... of serializing a moderatly compiicated class structure. " ^^^^ ;) 4. please emphasize your example, by adding a subsection "Example" in the doc about extended_type_info. I mean, add subsection title just before: "The test program test_no_rtti implements this function in terms of the extended_type_info...." I had clicked it ( http://www.boost.org/libs/serialization/test/test_no_rtti.cpp ) the first time when I was reading from top to bottom, but later I couldn't find it, as I did not remember where I clicked :) 5. if I use extended_type_info with rtti, will my binary files be portable between various platforms? (I assume that without rtti they will be) 6. a bit related question: does binary serialization take care of endianess and different sizes of fundamentals across various platforms? (size in bytes) Type ia32 PowerPC amd64 ----------------------------------- long 4 4 8 unsigned long 4 4 8 long double 12 8 16 7. the real question: is it possible to use extended_type_info for the class factory purposes without actually using serialization, for example like that below (with or without rtti). It's just your example, slightly modified: class polymorphic_base { friend class boost::serialization::access; template<class Archive> void serialize(Archive & /* ar */, const unsigned int /* file_version */){ } public: virtual const char * get_key() const = 0; virtual ~polymorphic_base(){}; }; BOOST_IS_ABSTRACT(polymorphic_base) BOOST_CLASS_TYPE_INFO( polymorphic_base, extended_type_info_no_rtti<polymorphic_base> ) // note: types which use ...no_rtti MUST be exported BOOST_CLASS_EXPORT(polymorphic_base) class polymorphic_derived1 : public polymorphic_base { friend class boost::serialization::access; template<class Archive> void serialize(Archive &ar, const unsigned int /* file_version */){ ar & BOOST_SERIALIZATION_BASE_OBJECT_NVP(polymorphic_base); } public: virtual const char * get_key() const ; }; BOOST_CLASS_TYPE_INFO( polymorphic_derived1, extended_type_info_no_rtti<polymorphic_derived1> ) BOOST_CLASS_EXPORT(polymorphic_derived1) const char * polymorphic_derived1::get_key() const { const boost::serialization::extended_type_info *eti = boost::serialization::type_info_implementation<polymorphic_derived1> ::type::get_instance(); return eti->get_key(); } int main( int /* argc */, char* /* argv */[] ) { // the most important line: // this function takes 'const char* class_name' (or GUID ?) as an argument. polymorphic_base* p=please_somehow__create_an_instance_of("polymorphic_derived1"); } -- Janek Kozicki |

Robert Ramey said: (by the date of Sun, 8 Oct 2006 22:13:26 -0700)

The documentation contains a portable binary archive as part of an illustration on how to derive from an existing archives. Portable binary archives are not as fast as native binary archives - but faster than text based archives.

thanks. I'm still digging through the docs, so haven't read that yet.

...

7. the real question: is it possible to use extended_type_info for the class factory purposes without actually using serialization, for example like that below (with or without rtti). It's just your example, slightly modified: ... int main( int /* argc */, char* /* argv */[] ) { // the most important line: // this function takes 'const char* class_name' (or GUID ?) as an argument. polymorphic_base* p=please_somehow__create_an_instance_of("polymorphic_derived1"); }

A ha - this is the missiing magic. The equivalent of this code is currently buried in iserialer.hpp. If you think about it - it HAS to be there in order for serialization of exported pointers to function.

Ok, I'll have a look at it. I had similar experience with serialization for yade. First - the factory was part of serialization, and looked like very difficult to separate. But later we managed to separate this library. Hovewer yade's serialization is much simpler. We've never thought about exported pointers to function :) So I understand that the case here is even more difficult.

I believe it could be extracted from this location and moved to the extended_type_info code. I don't think it's trivial however. But I believe that the end result would be an (almost) complete solution for dynamically loading DLLS automatically whenever an exported class is required.

So I think that this should be the goal to pursue if we want to add an extensions/plugin library into boost. (also to Jeremy) Think of it. With two separate libraries (one for serialization and one for plugins) the user will have to register each class twice. Serialization already registers classes with GUID (Global Unique IDentifier) so plugins/extensions library should take advantage of that. -- Janek Kozicki |

Martin Bonner wrote:

...

From Janek Kozicki: So I think that this should be the goal to pursue if we want to add an extensions/plugin library into boost.

Think of it. With two separate libraries (one for serialization and one for plugins) the user will have to register each class twice. Serialization already registers classes with GUID (Global Unique IDentifier) so plugins/extensions library should take advantage of that.

I'm not convinced. It seems to me that the registration for plugins and for serialization is rather different.

Serialization: - When you see this GUID, this is how to create the class.

Plugins: - This is another object with an IFoo interface.

Almost but not quite. This is another object with an IFo interface. But which object. There can be many different types. So each type is identified by a GUID - same one used by the serialization system. So the magic missing function is BOOST_CLASS_EXPORT(IFoo_derived_1, IFOO_ONE) IFoo *ifptr = static_cast<IFoo *>(extended_type_info::create_object(IFOO_ONE)) This is basically how microsoft COM works. This system would be simple, fast, powerful and easy to implement. It would be upto the programmer to ensure that the static_cast is correct.

Perhaps I'm misunderstanding what you mean by plugin, but I think of something like Internet Explorer, or Photoshop, where the application has a bunch of IFoo pointers, and it iterates through all of them to extend its menu. When the menu is clicked it calls the appropriate function through the IFoo pointer.

This is exactly the case that I have in mind.

I don't really see how the two relate.

Hopefull the above will clarify this. RObert Ramey

Robert Ramey said: (by the date of Mon, 9 Oct 2006 09:31:38 -0700)

Janek Kozicki wrote:

...

Think of it. With two separate libraries (one for serialization and one for plugins) the user will have to register each class twice. Serialization already registers classes with GUID (Global Unique IDentifier) so plugins/extensions library should take advantage of that.

Oh no. registering a data type for export would be sufficient for plug-ins AND serialization.

Yes you are right. Slight misunderstanding here. I was talking about "two separate, totally different" libraries. Like we already have boost::serialization and a new library from Jeremy that doesn't know or use the mechanisms from serialization but instead reinvents the whell. And I meant that this would be bad.

So I would like to see:

a) a plug-in library based on ehancement of extended_type_info b) serialization tweaked to use this library.

Then we get a "smaller" serialization library - and a separate self contained library which is useful for other things. Just a little code re-factoring.

yes, I agree 100% So if we start working on this, we simply continue the work started by introducing extended_type_info But since it's Jeremy who started this thread I await his (your ;) acknowledgement of this conclusion. I'll try to help in doing that (as usual - given enough time...) -- Janek Kozicki |

Jeremy Pack wrote:

Janek and Robert (and others),

First of all, the library that I'm working on does use the Serialization library, but not to the level that Robert is suggesting.

Two points:

1 - The exporting of classes from a dll or whatever is not a terribly difficult part to implement - if this were all that I needed, I would not have started my own library. However, I agree that this part SHOULD use boost::serialization (not reusing already proven code would be a mistake in this case). As long as the way that has been suggested will meet a few criteria, I think work should immediately begin on doing it using Robert's suggestion. I will check through the extended_type_info documentation to see if it will be able to provide each of the requirements that I feel necessary. It sounds like it does.

Hmm - hopefully it does. But it probably doesn't make them available in the most convenient way for general usage. Remember I just made it to solve a specific problem - serialization of derived pointers through a base class. So it will have to be looked at in a new light.

2 - The primary requirement that is important to me is tracking dependencies between loadable objects. I'm working on a relatively flexible way of handling that. I think it is a vital part of making this library more useful than just a classloader. It's also, I think, the hardest and most subjective part.

And this is something not addressed at all -

I will still attempt to get my code posted, keeping in mind that most of the object loading will probably be completely rewritten to use extended_type_info.

Robert, if you have the time, I believe your experience creating the boost::serialization library would make you the most qualified to head up the effort to create a plugin library for Boost. Even if you don't have that kind of time, your continuing input would be very appreciated.

All I know about the subject is included in the code and documentation for extended_type_info. I'm happy about offering that as a starting point - and answer any questions about why I used X to fullfill requierment Y (if I can remember) but that's the best I can do. Your on your own from now on. Of course, as a "veteran booster" I reserve the right to complain about your solution along with everyone else when its done. Also I'm not altogether sure that what I've done is useful to you. I just want to make sure you look at to see if we can factor out some commonality. If you take an honest look at it an conclude it doesn't work for you, I'm fine with that. I just want to make sure that an opportunity to make things smaller (and off load part of the library to someone elses library) isn't missed.

One important question: should we avoid requiring RTTI? Should boost::serialization make it unnecessary? I'm currently using a dynamic cast or two, mostly just for type safety. I know that certain older compilers have a problem with that (Borland at one time, perhaps still, cannot handle RTTI in DLLs correctly for instance), and it might annoy people to have to remember to enable it on those that do not.

Here is how the non-RTTI facility came to be and how it fits in. The first review of the serialization was subjected to whithering critique during the boost review process and failed to be accepted. Lots of issues were raised by different people. One reviewer (G Rosenthal) however, distinguished himself by making an exhaustive review of all its flaws and the problems he saw in the implementation. This list had about 12-15 items on it. At first I was pretty discouraged. Then I started picking at the library using this list. A couple of things weren't that hard to fix. Hmmm as time went on I went fixing each thing. And each thing was harder to fix than the previous thing. But after a while, the whole project had me by the throat again. (I'm a very determined person). Finally there was only one thing left on the list - make the usage of RTTI optional. I was planning to blow this off as only one person had even mentioned it in the review - and it wasn't obvious how to implement it. But I said to myself - I've come this far, and following the review critique actually has resulted better than I thought - maybe this won't be so bad - maybe not as hard as it looks. I don't remember how hard it was. I also had to make those extended_type_info based on RTTI to interoperate with those which use any other system. This is what the "test_no_rtti" tests. But in the course of all this it became clear that extended_type_info could be almost factored out as a separate library - thereby permiting consideration of these almost orthogonal libraries to time share my attention and permit the full power of the steadily shrinking surface area of my aging brain to be focused extended_type_info independent of serialization. So I had an almost independent library - extended_type_info. But although it kept the serialization simpler, it wasn't clear how much value - if any it would have outside the library. Then something really surprising happened. People started using defining classes and thier serialization in DLLS. And the whole system continued to function (after a couple of very minor tweaks). This worked for both DLLS imported at pre-runtime via the usage of and import library as well as DLLs loaded and unloaded dynamically at runtime. It turns out that loading a DLL constructs the extended_type_info object when the DLL is loaded. The extended_type_info constructor addes the objects to a global table. Unloading does the opposite. (I had to make the destructor more elaborate remove destroyed objects from the global table). So the whole managment of keeping track of which types can be dynamically created via GUID is totally automatic and dynamic !!! (Of courses can still hang themselves by unloading a DLL before a required destructor is called - but I can't help them there) This came for free - just be trying to fix things that occurred when some ambitious users went where no man had gone before. So that's how we came to where we are today. I don't know if all this is interesting - but there it is. I just wanted to get it off my chest. Robert Ramey

Janek Kozicki wrote:

...

One important question: should we avoid requiring RTTI?

IMHO one more option for the users is good. But better first have a working prototype, then add new features, like no_rtti support.

Note that the extended_type_info system in place works through a virtual base class. There are two different implementations one based on RTTI and one based on GUID's only. Code in the base class permits different systems to inter-operate. This was deemed necessary because I concieved the type registry system used to be a property of the type and I wanted to beable to import code that used a different type system. In principle any number of typeinfo system could be implemented in addition or instead of the two included. Another thing I forgot to mention. void_cast is to extended_type_info as dynamic_cast is to RTTI. Right now void_cast is documented separately, but that also would have to be broken out. Robert Ramey