On Fri, 2009-11-27 at 21:55 +1100, Christian Schladetsch wrote:

Hi Dave,

You may get more response if you posted or linked to some motivational examples.

On Fri, Nov 27, 2009 at 11:45 AM, Dave van Soest <dsoest@xs4all.nl> wrote:

...

Hi all,

I created a template-based header-only library for creating static plugin systems. I'm wondering whether there would be any interest to include such a library in Boost.

+1 on the examples. So far I'm merely intrigued to understand what the library is/does.

On Sun, Nov 29, 2009 at 9:34 AM, Dave van Soest <dsoest@xs4all.nl> wrote:

Hi Christian, Darryl, and anyone else interested,

...

...

You may get more response if you posted or linked to some motivational examples.

+1 on the examples. So far I'm merely intrigued to understand what the library is/does.

Suppose that your program provides this simplified plugin interface class 'ObjectBase' in the header 'ObjectBase.hpp':

---- Start of code sample 1 ---- class ObjectBase { public:    typedef std::string IdType;    ObjectBase() {}    virtual ~ObjectBase() {}    virtual void doSomething() = 0; }; ---- End of code sample 1 ----

Then the static plugin library will allow you to extend the functionality of this program merely by adding a separate source file containing the following code:

---- Start of code sample 2 ---- #include <static_plugin.hpp> #include "ObjectBase.hpp"

class SomeObject : ObjectBase {    void doSomething() { std::cout << "Doing something" << std::endl; }    static const IdType id; };

const SomeObject::IdType SomeObject::id = "SomeObject_ID"; static const static_plugin::Creator<SomeObject, ObjectBase> myPlugin; ---- End of code sample 2 ----

The above code causes a factory for the 'SomeObject' class to be created automatically and registered with the factory manager for classes implementing the 'ObjectBase' interface. This all happens when the program is just started (before it enters 'main').

The factory manager for 'ObjectBase' derivatives, which follows the singleton pattern, can be used from your program as follows (note: again this code is simplified):

---- Start of code sample 3 ---- #include <static_plugin.hpp> #include "ObjectBase.hpp"

typedef FactoryManager<ObjectBase> ObjectBaseFM;

int main(void) {    const ObjectBaseFM& fm(getFactoryManager<ObjectBase>());    assert(fm.factoryCount() > 0);    ObjectBase::IdType id(fm.factoryIdList().at(0));    std::auto_ptr<ObjectBase> object(fm.create(id));    object->doSomething();    return 0; } ---- End of code sample 3 ----

The only requirements are that ObjectBase defines the IdType type and SomeObject contains a static member of type IdType called 'id'. The declaration of the static_plugin::Creator<...> causes the SomeObject plugin to be registered.

Sidenote: I just tested this functionality in a dynamically loaded library and that works now too. This means that any plugin in your dynamic library is automatically registered without the need to call some 'initLibrary' function in the library.

Heh, I do the exact same thing for many of my things, especially in the console to register new commands. If you can make it more simple and/or efficient then I already have my code, I would use this, but so far your code is slightly more verbose.

Hi Dave, I appreciate your effort with this. However, it is uninteresting to me. Relying on static initialiser ordering is a huge mistake which IMO cannot be fixed. Factory models based on such approaches are doomed to failure. It is best to explicitly make a factory and then explicitly add runtime types dynamically (perhaps via a configuration file). This is not only more correct (the order or registration is repeatable), it preserves ideas of scope and removes requirements for globals. Best, Christian. On Mon, Nov 30, 2009 at 3:34 AM, Dave van Soest <dsoest@xs4all.nl> wrote:

Hi Christian, Darryl, and anyone else interested,

You may get more response if you posted or linked to some motivational

...

...

examples.

+1 on the examples. So far I'm merely intrigued to understand what the library is/does.

Suppose that your program provides this simplified plugin interface class 'ObjectBase' in the header 'ObjectBase.hpp':

---- Start of code sample 1 ---- class ObjectBase { public: typedef std::string IdType; ObjectBase() {} virtual ~ObjectBase() {} virtual void doSomething() = 0; }; ---- End of code sample 1 ----

Then the static plugin library will allow you to extend the functionality of this program merely by adding a separate source file containing the following code:

---- Start of code sample 2 ---- #include <static_plugin.hpp> #include "ObjectBase.hpp"

class SomeObject : ObjectBase { void doSomething() { std::cout << "Doing something" << std::endl; } static const IdType id; };

const SomeObject::IdType SomeObject::id = "SomeObject_ID"; static const static_plugin::Creator<SomeObject, ObjectBase> myPlugin; ---- End of code sample 2 ----

The above code causes a factory for the 'SomeObject' class to be created automatically and registered with the factory manager for classes implementing the 'ObjectBase' interface. This all happens when the program is just started (before it enters 'main').

The factory manager for 'ObjectBase' derivatives, which follows the singleton pattern, can be used from your program as follows (note: again this code is simplified):

---- Start of code sample 3 ---- #include <static_plugin.hpp> #include "ObjectBase.hpp"

typedef FactoryManager<ObjectBase> ObjectBaseFM;

int main(void) { const ObjectBaseFM& fm(getFactoryManager<ObjectBase>()); assert(fm.factoryCount() > 0); ObjectBase::IdType id(fm.factoryIdList().at(0)); std::auto_ptr<ObjectBase> object(fm.create(id)); object->doSomething(); return 0; } ---- End of code sample 3 ----

The only requirements are that ObjectBase defines the IdType type and SomeObject contains a static member of type IdType called 'id'. The declaration of the static_plugin::Creator<...> causes the SomeObject plugin to be registered.

Sidenote: I just tested this functionality in a dynamically loaded library and that works now too. This means that any plugin in your dynamic library is automatically registered without the need to call some 'initLibrary' function in the library.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Dave van Soest wrote :

...

...

...

I just tested this functionality in a dynamically loaded library and that works now too.

Does it work correctly with static member of templated classes for example? I have always had problem of conflicts for those. I mean conflicts like a creation of static members copy instead of a reuse of a shared static instance (when one dll use another one dll) ... -Please note that I am no dll expert -. Best regards, Pierre Morcello

Greetings, The key things to understand here are construction ordering and namespace pollution. When using static initialisers, you have absolutely no control over the practical ordering of those initialisations in the final image. One build may be A,B,C, the next may be C,A,B, even on the same platform with the same set of types. The ordering is effectively arbitrary from build to build. No, really, this is true. Given that, using such an approach to build a factory model is irrational. Many types need other types in their definitions. Without a deterministic order, using static initialisation to create a factory is impossible. Sure, you can come up with and practically use toy systems with a given linker in a given configuration that have no problem. But when you have complex and.or derived types, or use dynamic linking, it becomes an intractable problem. I thought that these days, such issues are generally well-known. If you want to do this sort of thing, *you must do it after main() has been called*. Ideally, use some code-gen as well to make it type-safe as well as build-time dynamic. The second issue is global scope pollution. Systems based on pre-main() initialisation are necessarily polluting the global namespace (you can hide the variables in a namespace, but they are still global). As such, they become impossible to test in isolation and further are problematic with threads. This is all just a longer-worded reply with the same answer as my previous post: such approaches as Dave suggests are relatively common, and all are doomed to failure in the general case. My best advice to avoid heartbreak is to move to a localised, scoped, dynamically driven model, perhaps with an automated code-gen back-end. Regards, Christian. On Mon, Nov 30, 2009 at 11:41 PM, OvermindDL1 <overminddl1@gmail.com> wrote:

...

Dave van Soest wrote :

...

...

...

...

I just tested this functionality in a dynamically loaded library and

On Mon, Nov 30, 2009 at 5:25 AM, Pierre Morcello <pmorcell-cppfrance@yahoo.fr> wrote: that works now too.

...

Does it work correctly with static member of templated classes for

example? I have always had problem of conflicts for those. I mean conflicts like a creation of static members copy instead of a reuse of a shared static instance (when one dll use another one dll) ... -Please note that I am no dll expert -.

For note, let me give my use case.

I have one class, say ConsoleCommands, and it is a singleton designed in such a way that it is constructed upon the first call to GetInstance (and it does not even attempt to construct it, just returns its pointer if you call GetInstancePtr).

I have things called ConsoleCommand that, as an example, can be called like this:

ConsoleCommand cmd_help("help", &cmd_helpHandler, "Displays the help menu");

Internally it basically calls ConsoleCommands::GetInstance.RegisterCommand("help", &cmd_helpHandler, "Displays the help menu");

That will construct a ConsoleCommands if necessary (which is nothing but an unordered_map essentially, nothing I need to worry about being run before main).

It is not necessarily pretty, but I have been using it for near ten years and never had a problem with it yet. I occasionally do other things that make such registrations as well, so if he made a more useful interface for it all so I do not need to make 3 helper structs (only one of which actually holds data, the others are empty), then I would be for it, but only if it really is easier. And yes, my method works across DLL boundaries (since my code is designed to allow the hot-plugging and removing of commands for the console for example). _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Christian Schladetsch

Greetings,

Please don't top post. See http://www.boost.org/community/policy.html#quoting.

When using static initialisers, you have absolutely no control over the practical ordering of those initialisations in the final image. One build may be A,B,C, the next may be C,A,B, even on the same platform with the same set of types. The ordering is effectively arbitrary from build to build. No, really, this is true.

Depending upon platform, the use of a dynamically linked library (DLL or shared object) may give desirable results, whereas the use of statically linked libraries will not. For example, in our environment, static objects in an object file in a static library *will not* be initialized unless a function in the same translation unit is called (at least indirectly) from main().

Given that, using such an approach to build a factory model is irrational.

Not so much irrational as misinformed or ignorant of portability considerations.

Many types need other types in their definitions. Without a deterministic order, using static initialisation to create a factory is impossible. Sure, you can come up with and practically use toy systems with a given linker in a given configuration that have no problem. But when you have complex and.or derived types, or use dynamic linking, it becomes an intractable problem.

That problem is worsened when it must be portable.

I thought that these days, such issues are generally well-known. If you want

There's a great deal to know and one has to go looking for it to find it. It is easy to develop something in isolation and think it will work more broadly.

Ideally, use some code-gen as well to make it type-safe as well as build-time dynamic.

Code generation is never an ideal solution, though it might be necessary to best satisfy a given set of criteria.

The second issue is global scope pollution. Systems based on pre-main() initialisation are necessarily polluting the global namespace (you can hide the variables in a namespace, but they are still global). As

That is nonsensical. How can a name in a namespace pollute the global namespace? Perhaps you're trying to say that such variables contribute to the global system state.

such, they become impossible to test in isolation and further are problematic with threads.

In the OP's environment, such statics may well be initialized before main() runs, thus avoiding threading problems. That isn't necessarily true, but it is accepted that threads shouldn't be created via static initialization, so it isn't unreasonable to make that a requirement for using such a library. You are certainly on point about testing being difficult in the OP's scheme. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

EDIT: I apologise, Robert said nothing about starting a thread as a result of the initialisation of a global. On Tue, Dec 1, 2009 at 12:50 AM, Christian Schladetsch < christian.schladetsch@gmail.com> wrote:

Dissenting points from my preceding post are commented on:

Rob Stewart wrote:

...

Depending upon platform, the use of a dynamically linked library (DLL or

...

shared object) may give desirable results, whereas the use of statically linked libraries will not.

Not only depending on platform, but also depending on the order of the linking of compilation units within a given DLL or other library; which is not deterministic.

[snip]

...

The second issue is global scope pollution. Systems based on

...

pre-main() initialisation are necessarily polluting the global namespace (you can hide the variables in a namespace, but they are still global). As

That is nonsensical. How can a name in a namespace pollute the global namespace? Perhaps you're trying to say that such variables contribute to the global system state.

Either or both my and the general terminology is lacking here. I don't know what the convention is for this, but to clarify: when I say 'global' I mean 'globally accessible, given matching declaration'. A variable in namespace *foo* can still be globally accessible, and remains traumatic when used in threaded applications.

...

In the OP's environment, such statics may well be initialized before main() runs, thus avoiding threading problems.

How could such a reprehensible act of creating a thread before main() runs avoid problems?

Regards, Christian.

My team works on code that uses a lot of static initialization for global variables. Based on this difficult experience, I would highly recommend initializing the factories inside of functions called after main() has started, rather than in static initializers. The amount of code to make this work correctly across platforms is nontrivial. In fact, we require that a certain initialization function be called as soon as main() has started, just to clean up all of the messes left by doing static initialization of so many things. I believe that this original design was a mistake - but we have too much code to change it overnight. I wouldn't call it "doomed to failure" - just difficult to get right, and probably not worth the effort to get it right. I wrote some code for the sandbox a while ago that does this: http://boost-extension.redshoelace.com/docs/boost/extension/boost_extension/... It works for both static and dynamic linking, and for shared libraries that are determined at run time (the shared libraries need to have been compiled with the same compiler options). It also allows arbitrary constructor signatures, inheritance across shared libraries, reflection and a few other features. I've been waiting to finalize the reflection functionality (it works, but I don't like parts of the API) until C++0x is more finalized in the various compilers. Jeremy Pack

Your example uses a centralized place (the type-map) to store the information about the derived classes/plugins. That is what I'm aiming to avoid with my static plugin library.

I did this because I figured it would be trivial for a user who wanted it global to make it global. It is just like your "Factory Manager", but it can be either global or local (with simple wrapper functions). You could make it safe with the static initialization by making sure that the Factory Manager is initialized before it is used. For your use case, I think it would be sufficient to instantiate the Factory Manager inside of the function that returns it (Get() below): template <class Base> class FactoryManager { public: static void RegisterTypeConstructor(Base* (*func)(), const string& id) { Get()->insert(make_pair(id, func)); } static map<string, Base*(*)()>* Get() { static map<string, Base*(*)()> constructors; return &constructors; } static Base* Create(const string& id) { return (*(*Get())[id])(); } }; template <class Derived, class Base> class Factory { public: Factory(const string& name) { FactoryManager<Base>::RegisterTypeConstructor(&Factory::Create, name); } static Base* Create() { return new Derived(); } }; class MyBaseClass { public: virtual void PrintString() { cout << "MyBaseClass" << endl; } }; class MyDerivedClass : public MyBaseClass { public: virtual void PrintString() { cout << "MyDerivedClass" << endl; } }; Factory<MyDerivedClass, MyBaseClass> i("derived"); class MyOtherDerivedClass : public MyBaseClass { public: virtual void PrintString() { cout << "MyOtherDerivedClass" << endl; } }; Factory<MyOtherDerivedClass, MyBaseClass> j("other_derived"); int main(int argc, char* argv[]) { cout << "Entering main." << endl; MyBaseClass* c = FactoryManager<MyBaseClass>::Create("derived"); MyBaseClass* d = FactoryManager<MyBaseClass>::Create("other_derived"); c->PrintString(); d->PrintString(); } /// End of code Jeremy

On 12/04/2009 03:11 AM, OvermindDL1 wrote:

...

You could make it safe with the static initialization by making sure that the Factory Manager is initialized before it is used. For your use case, I think it would be sufficient to instantiate the Factory Manager inside of the function that returns it (Get() below):

Which is what I do in the code I demonstrated, as I said... I am quite sure my emails are going to the list just fine...

Jeremy, your code example also has many similarities with the code in my library. So, now I'm interested why you aren't using this approach... What kind of troubles do you have with it?

Hi Christian, Robert, others interested, Thanks for sharing your thoughts in this discussion. I'll try to summarize your posts in this way: you both don't think it is possible to create this static plugin functionality using static initialization and get it working on every supported platform. Well, I've already created a early version of this library and successfully tested it on two different platforms with three different compilers (two on each platform). I also think it conforms to the C++ standard in the way it uses static initialization (see my post dated 2009-11-30 14:00:54). Additionally I ran a successful test using shared libraries and dynamic plugins. Unfortunately I currently don't have more platforms and compilers to test the library with, but hopefully I will soon find a way to do this. I'm very much interested to learn whether this library will work on other platforms, for example the platform Robert describes in his first post to this thread. My (now slightly adjusted) question still is: in the hypothetical situation that this library works across all supported platforms, would there be any interest to include it in Boost? Best regards, Dave van Soest

Hi Christian,

Sure, you can come up with and practically use toy systems with a given

...

linker in a given configuration that have no problem. But when you have complex and.or derived types, or use dynamic linking, it becomes an intractable problem.

Can you name any complex or derived types that wouldn't work? I've run some successful tests with dynamic linking. But I don't know yet whether it works on all platforms.

The problem here is that it is just a bad idea to use statics and type factories in this way. There will be problem after problem. What are these problems? The first problem is that of non-deterministic ordering of initialisers. The second problem is that you're doing work and calling functions before main(). The third problem are all the unknown problems that will arise by using this approach.

The order in which plugin factories are initialized shouldn't matter. The factory manager is initialized when the first plugin factory tries to register with it. So I don't see the problem here. What is wrong with calling functions before main(), if you do it carefully? The client coder won't get the opportunity to call any function before main() through this library. My response to the third problem is, just like the problem itself, yet unknown.

This approach has been tried many times before. AFAICT, it has never ended well. It requires bad practise (singletons, globals). It is untestable. It doesn't work well with DLL's. If you use it in a threaded application, there will be problems because of the globals. It doesn't scale to use types that have other type components, or derive from other types.

I will be doing more research to see whether my approach is different from the ones that didn't end well. I don't have any problems with using singletons, as long as they are used for the right purposes. Why is it untestable? I don't have a clue here. As I said, I ran a successful test using dynamically loaded libraries. I don't see any problems with threaded applications. The 'global' factories are not accessible outside there compilation unit. And the global factory manager(s) don't change any global information. I don't see why it doesn't scale to certain types. Can you give an example of a structure that wouldn't work?

Yes, it can work in a simple, limited case. But because of the scaling problems, it is inadvisable to do this. I think that perhaps everyone has to try to do this once, before they realise that it is a bad idea ;)

I haven't reached the point that I realised it's a bad idea. I'm seeing more use-cases each day. That's keeping me from giving up too early. Moreover, I haven't seen any concrete counter-example yet. And even if can only prove it to work on just a part of the platforms, it will still be valuable. But in that case not for inclusion in Boost of course.

I could well be wrong, and I don't think I will dissuade you from continuing your work on this. Best of luck.

Thanks. I really appreciate the time you spent on joining the discussion. Indeed you can't dissuade me from continuing this work, but still I really think it's valuable to have a sceptical counterweight to keep me from turning assumptions into beliefs and pointing me at potential problems I haven't really well thought out. Best regards, Dave van Soest

On Sat, Dec 5, 2009 at 9:20 AM, Dave van Soest <dsoest@xs4all.nl> wrote:

Hi Christian,

Sure, you can come up with and practically use toy systems with a given

...

...

linker in a given configuration that have no problem. But when you have complex and.or derived types, or use dynamic linking, it becomes an intractable problem.

Can you name any complex or derived types that wouldn't work?

// this requires that type A is registered before type B struct A { }; struct B { Pointer<A> a; static void Register(Factory &f) { a = f.Create<A>(); } }; ... // this also requires that A is registered before B struct A { }; struct B : DerivesFrom<A> { }; ...

What is wrong with calling functions before main(), if you do it carefully?

Because "doing it carefully" sometimes isn't enough. For example, if you're doing your own memory management (a common motivator for using factories at all), then any allocation done before main() will be using the clib malloc - and before any of your own custom memory allocation systems have been initialised. Then you have to work-around that. Then that becomes a problem because often this allocator has to be guarded against multiple threads, which requires semaphores or other locking structures to be setup, which then have to be done statically, and because of the ordering issue you are back to square one.

The client coder won't get the opportunity to call any function before main() through this library.

This is not true; initialisation code in the registration method will be called before main(), as will anything called from it. Over time you may want to read configuration files or do memory allocation in these pre-main systems. It grows. It gets gnarly. You work-around it. It grows more. Then you get it working, then change linkers and it all breaks again. Other example gotchas are initialisation of logging systems, string pools systems, data-structures with private static pools - basically, pre-main is a bad space in which to try to do anything. If you kept it extremely simple - no allocation, no threading, no file handling, no exceptions, no nested or derived types, and in a known, single development environment then yeah, sure, it can work. But its not portable, doesn't scale, is untestable, and its extremely limited. Oh, its not testable because the use of globals means that you cant create the system, test it and tear it down again in isolation of other systems. The lack of deterministic ordering means that testing is problematic even in the basic cases.

My response to the third problem is, just like the problem itself, yet unknown.

The C machine is not meant to be used this way. It can be used this way, but it invariably results in many subtle problems. This idea has been kicked around by various people and teams going back decades. It has always proven to be a neat idea that doesn't work in the real world. Perhaps it can work; but all I know about are counter-examples.

Thanks. I really appreciate the time you spent on joining the discussion. Indeed you can't dissuade me from continuing this work, but still I really think it's valuable to have a sceptical counterweight to keep me from turning assumptions into beliefs and pointing me at potential problems I haven't really well thought out.

Welcome, Christian.

On Wed, Dec 16, 2009 at 1:55 PM, Dave van Soest <dsoest@xs4all.nl> wrote:

Hi all,

The C++ standards document states that it is implementation dependent whether the constructor of a static object is called before entering main() or just before first use (ISO/IEC 14882:2003 section 3.6.2 part 3).

I can't help wondering why the standard leaves this choice open to implementation.

The idea is to allow for costly initializations to be postponed until they are actually needed. For example, if a static object allocates a gig of RAM, that would be wasteful if the static object isn't needed in a particular run of the program.

The compilers/linkers/platforms I tested my library with all enable calling the constructors before entering main() by default, but this behavior is not guaranteed by the language. I currently don't know whether this behavior can be guaranteed in any way. If so, the library may still be useful, but perhaps not for Boost.

No, the behavior can not be guaranteed. What's more, static objects, along with all relevant code, can be deadstripped unless you explicitly call a function from their translation unit (cpp file.) Emil Dotchevski Reverge Studios, Inc. http://www.revergestudios.com/reblog/index.php?n=ReCode

Hi Vicente,

Hi, I don't know if this is really related or not, there is a Functional/Factory library accepted in Boost.

In case this can help you, Vicente

Thanks for the suggestion. I'm currently not using Boost Functions or Factories, but I will look into it. Kind regards, Dave van Soest