Checking interest in DynamicLoad/DLL/DSO/Plugin library
Hi, There is a basic implementation of a header only Plugin library: https://github.com/apolukhin/Boost.Plugin Documentation with examples could be found here: http://apolukhin.github.io/Boost.Plugin/index.html Is there an interest in such library in Boost community? What features are missing in the library? Looking forward to hearing from you. Any feedback is more than welcomed. -- Best regards, Antony Polukhin
On Fri, Aug 15, 2014 at 1:36 PM, Antony Polukhin <antoshkka@gmail.com> wrote:
Hi,
There is a basic implementation of a header only Plugin library: https://github.com/apolukhin/Boost.Plugin
Documentation with examples could be found here: http://apolukhin.github.io/Boost.Plugin/index.html
Is there an interest in such library in Boost community? What features are missing in the library?
I'm highly interested and I suspect a lot of people are. For example, feedback and discussions related to (non-boost library) Boost.extension and CPPCMS's plugin loading tool (discussions happenned in this mailing list archive) points to a lot of people needing something like this. In any ways I would gladely try it in my projects and I believe such a solution would be helpful in my current dayjob too. I will take a look at the details of this specific library in the coming hours.
Looking forward to hearing from you. Any feedback is more than welcomed.
-- Best regards, Antony Polukhin
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 08/15/2014 07:36 AM, Antony Polukhin wrote:
Hi,
There is a basic implementation of a header only Plugin library: https://github.com/apolukhin/Boost.Plugin
Documentation with examples could be found here: http://apolukhin.github.io/Boost.Plugin/index.html
Is there an interest in such library in Boost community? What features are missing in the library?
Looking forward to hearing from you. Any feedback is more than welcomed. I would have interest in such a library. It looks pretty good to me, but I will let those more skilled than I comment on implementation details. Since I don't see it explicitly said anywhere, is there anything in Boost.Plugin that requires C++11?
Jason
2014-08-15 17:05 GMT+04:00 Jason Roehm <jasonr@3db-labs.com>: <...>
I would have interest in such a library. It looks pretty good to me, but I will let those more skilled than I comment on implementation details. Since I don't see it explicitly said anywhere, is there anything in Boost.Plugin that requires C++11?
Plugin library does not need C++11. It works well on Linux(GCC 4.6 in C++03 and C++11 modes, Clang 3.4) and Windows(MSVC11). Currently it requires Boost 1.56, but this requirement can be relaxed. -- Best regards, Antony Polukhin
I read the documentation and it looks good starting point, it's complete enough to start working with. Some typo: - Getting Started page: "just *cunstruct* that class with a path to the library as a parameter" should be "construct" - There is a missing '*' in this line in the tutorial page: "Linking plugin into the executable has the advantages of * reducing common size of distibution * simplification of installation of distribution * faster load times of plugin" Some suggestions for the doc: - provide some pages to help with declaring correctly plugin targets with some commonly used (meta-)build system, using this library macros. For example a CMake and a VS projects pages would be useful additions (I think) - clarify early what should happen when you use the library incorrectly (link errors most of the time): this in particular can help a lot understanding issues when you first try to setup a plugin system and don't have yet a good understanding of how it works; - some typical examples on when using "modes" on constructing shared_library can be useful (at least it's not clear to me when it is useful); - there is no reference page for the modes and no link generated when it appears in shared_library reference page. - a note clarifying that BOOST_SYMBOL_EXPORT is not defined by this library but by boost.config would be nice, also if you clarify that you could use your own symbol export/import macro if you already have one, it wouldn't change a thing for boost.plugin; - a page gathering the platform-specific warning per platform would be a useful reference, in particular if it describe the important/noticeable difference in behaviours of the different platforms; I took a very quick look at the implementation but it seems that some parts are temporary (FIXME comments) so I'll ignore that for now. Anyway it looks as simple as it should so far. I will play with it soon, see if I see any problem. Question: - would have it been possible/better to not expose the shared_ptr<> s in the interfaces? Maybe using concept-based type-erasing type instead?
2014-08-15 17:52 GMT+04:00 Klaim - Joël Lamotte <mjklaim@gmail.com>:
I read the documentation and it looks good starting point, it's complete enough to start working with.
Some typo: - Getting Started page: "just *cunstruct* that class with a path to the library as a parameter" should be "construct" - There is a missing '*' in this line in the tutorial page: "Linking plugin into the executable has the advantages of * reducing common size of distibution * simplification of installation of distribution * faster load times of plugin"
I'll fix that soon. Thanks!
Some suggestions for the doc: - provide some pages to help with declaring correctly plugin targets with some commonly used (meta-)build system, using this library macros. For example a CMake and a VS projects pages would be useful additions (I think)
- clarify early what should happen when you use the library incorrectly
(link errors most of the time): this in particular can help a lot understanding issues when you first try to setup a plugin system and don't have yet a good understanding of how it works;
Build plugin just like a usual shared library. Plugin library is actually a wrapper around native functions to DLL/SO. You can use it to import C functions from any DLL/SO. I'll try to clarify that somewhere in docs.
- some typical examples on when using "modes" on constructing shared_library can be useful (at least it's not clear to me when it is useful); - there is no reference page for the modes and no link generated when it appears in shared_library reference page.
Hm... Docs for modes are inplace, looks like doxygen failed to parse the markup. I'll fix that.
- a note clarifying that BOOST_SYMBOL_EXPORT is not defined by this library but by boost.config would be nice, also if you clarify that you could use your own symbol export/import macro if you already have one, it wouldn't change a thing for boost.plugin;
Good points!
- a page gathering the platform-specific warning per platform would be a useful reference, in particular if it describe the important/noticeable difference in behaviours of the different platforms;
I took a very quick look at the implementation but it seems that some parts are temporary (FIXME comments) so I'll ignore that for now.
Those "FIXME" are because the dll2.hpp header must be moved into the Boost.Winapi library with the name dll.hpp. I've postponed that, because dll2.hpp still may change a little bit and I do not wish to spam Boost.Winapi admins with pull requests.
Anyway it looks as simple as it should so far.
I will play with it soon, see if I see any problem.
Thanks!
Question: - would have it been possible/better to not expose the shared_ptr<> s in the interfaces? Maybe using concept-based type-erasing type instead?
If you are talking about shared_ptr in DLL ("shared_ptr create_plugin()" examples), then it is up to user what functions to export from DLL and what signatures those functions have. If you are talking about the boost::plugin::shared_variable() methods, then there is a "core" part of the library that does not use shared_ptr or boost::function. In "Refcountable" part of the library shared pointers are essential for internal needs. In theory a better solution would be to return unique_ptr from shared_variable() functions, however there is no unique_ptr at the moment in Boost and forcing user to use C++11 is not preferable. -- Best regards, Antony Polukhin
El 15/08/2014 16:28, Antony Polukhin wrote:
In theory a better solution would be to return unique_ptr from shared_variable() functions, however there is no unique_ptr at the moment in Boost and forcing user to use C++11 is not preferable.
We need to fix this ASAP. Not having boost::unique_ptr is an embarrassment.
On 15 Aug 2014 at 15:36, Antony Polukhin wrote:
There is a basic implementation of a header only Plugin library: https://github.com/apolukhin/Boost.Plugin
Documentation with examples could be found here: http://apolukhin.github.io/Boost.Plugin/index.html
Is there an interest in such library in Boost community? What features are missing in the library?
Looking forward to hearing from you. Any feedback is more than welcomed.
It definitely has the wrong name because it doesn't implement plugins, it basically is a wrapper for the POSIX and Windows dynamic shared library functions. Boost.SharedLibrary is a far better name. Now, if you did want to implement a Boost Plugins library, combining John Bandela's CppComponents (https://github.com/jbandela/cppcomponents) header only library which turns C++ objects into portable Microsoft COM compatible objects and Boost.ASIO as the event dispatch loop would probably be an optimal solution. Not only does that seamlessly plug into all Microsoft technologies, it also is an easy way of SWIG exposing your object into all the languages SWIG supports. If anyone is interested in implementing this solution to Boost quality, I believe I might be able to get you hourly funding to do it - contact me off list. That said, I have no objection to your library or its design which is solid, so long as it is named more appropriately to what it provides. I suppose my only real qualm is that I wish your library did more. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On Fri, Aug 15, 2014 at 6:44 PM, Niall Douglas <s_sourceforge@nedprod.com> wrote:
On 15 Aug 2014 at 15:36, Antony Polukhin wrote:
There is a basic implementation of a header only Plugin library: https://github.com/apolukhin/Boost.Plugin
Documentation with examples could be found here: http://apolukhin.github.io/Boost.Plugin/index.html
Is there an interest in such library in Boost community? What features are missing in the library?
Looking forward to hearing from you. Any feedback is more than welcomed.
It definitely has the wrong name because it doesn't implement plugins, it basically is a wrapper for the POSIX and Windows dynamic shared library functions. Boost.SharedLibrary is a far better name.
In my experience the "plugin" word is used as soon as a shared library could be loaded at an indeterminable time during execution (and be unloaded too). This is different from just "shared library" which is both a plugin implementation and a library linkage, including having the library automatically loaded at startup. To me plugin is far more precise than "shared library". You don't need such a library to make a shared library. You would need such a library to make a plugin (or extension, or module) library. Even CMake call them "module", not "shared" libraries. Even if implemented in unix-like systems as just "shared" libraries, it's still a special case of shared library, not the general case.
Now, if you did want to implement a Boost Plugins library, combining John Bandela's CppComponents (https://github.com/jbandela/cppcomponents) header only library which turns C++ objects into portable Microsoft COM compatible objects and Boost.ASIO as the event dispatch loop would probably be an optimal solution. Not only does that seamlessly plug into all Microsoft technologies, it also is an easy way of SWIG exposing your object into all the languages SWIG supports. If anyone is interested in implementing this solution to Boost quality, I believe I might be able to get you hourly funding to do it - contact me off list.
Could you clarify what you mean by plugin here? I don't see why ASIO would be a nice dependency for such a tool, but I don't understand what you mean either.
That said, I have no objection to your library or its design which is solid, so long as it is named more appropriately to what it provides. I suppose my only real qualm is that I wish your library did more.
Niall
-- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 15 Aug 2014 at 19:22, Klaim - Joël Lamotte wrote:
It definitely has the wrong name because it doesn't implement plugins, it basically is a wrapper for the POSIX and Windows dynamic shared library functions. Boost.SharedLibrary is a far better name.
In my experience the "plugin" word is used as soon as a shared library could be loaded at an indeterminable time during execution (and be unloaded too). This is different from just "shared library" which is both a plugin implementation and a library linkage, including having the library automatically loaded at startup. To me plugin is far more precise than "shared library". You don't need such a library to make a shared library. You would need such a library to make a plugin (or extension, or module) library.
A plugin/extension implies a well defined exported symbols and event processing ABI. Something adhering to Microsoft COM is a good example. Antony's library is far more low level, it really does just thinly wrap the two popular shared library APIs into a portable API. It says nothing about what the ABI should be other than C compatible, any form of inspection other than C format symbols, or anything about any event model. I wouldn't mind calling his library a plugin solution for C, but it isn't up to task for C++.
Even CMake call them "module", not "shared" libraries. Even if implemented in unix-like systems as just "shared" libraries, it's still a special case of shared library, not the general case.
Unfortunately Boost.Module might get confused with forthcoming C++ Modules. Otherwise I wouldn't object to that name.
Now, if you did want to implement a Boost Plugins library, combining John Bandela's CppComponents (https://github.com/jbandela/cppcomponents) header only library which turns C++ objects into portable Microsoft COM compatible objects and Boost.ASIO as the event dispatch loop would probably be an optimal solution. Not only does that seamlessly plug into all Microsoft technologies, it also is an easy way of SWIG exposing your object into all the languages SWIG supports. If anyone is interested in implementing this solution to Boost quality, I believe I might be able to get you hourly funding to do it - contact me off list.
Could you clarify what you mean by plugin here? I don't see why ASIO would be a nice dependency for such a tool, but I don't understand what you mean either.
ASIO is shortly to become the officially endorsed C++ event loop and dispatcher. As event processing is by definition part of any plugin or extension ABI - even if defined to be "no event processing", that means ASIO. So, just to be clear, if you load a Qt Plugin, the Qt event pump gets routed through ASIO's event pump. That way we can extend Vicente's monadic continuations framework easily to interoperate with all Qt threading and event objects. That also goes for all third party C++ libraries, and now you can mix and match loading Qt plugins with WinRT plugins with wxWidget plugins. You may find my C++ Now 2014 position paper useful. http://arxiv.org/pdf/1405.3323.pdf, starting from page 15 onwards. Just to be clear though, my earlier offer of potential paid work for a CppComponents based plugin system integrated with ASIO is a small subset of the C++ Now 2014 paper proposal, and is much, much easier. My current employer may decide on a Microsoft COM driven SWIG language bindings, and CppComponents is a quick way of metaprogramming the C++ compiler to spit out Microsoft COM compliant binaries. As always, there is a lack of available sufficiently experienced engineers on this. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On Fri, Aug 15, 2014 at 8:42 PM, Niall Douglas <s_sourceforge@nedprod.com> wrote:
On 15 Aug 2014 at 19:22, Klaim - Joël Lamotte wrote:
It definitely has the wrong name because it doesn't implement plugins, it basically is a wrapper for the POSIX and Windows dynamic shared library functions. Boost.SharedLibrary is a far better name.
In my experience the "plugin" word is used as soon as a shared library could be loaded at an indeterminable time during execution (and be unloaded too). This is different from just "shared library" which is both a plugin implementation and a library linkage, including having the library automatically loaded at startup. To me plugin is far more precise than "shared library". You don't need such a library to make a shared library. You would need such a library to make a plugin (or extension, or module) library.
A plugin/extension implies a well defined exported symbols and event processing ABI. Something adhering to Microsoft COM is a good example.
Antony's library is far more low level, it really does just thinly wrap the two popular shared library APIs into a portable API. It says nothing about what the ABI should be other than C compatible, any form of inspection other than C format symbols, or anything about any event model. I wouldn't mind calling his library a plugin solution for C, but it isn't up to task for C++.
I see. To me it's a good starting point and I fear that a more C++ than C library ends-up like boost.extension. But if it's not too hard to match your definition of plugin, it would be a win for me too as a user.
Even CMake call them "module", not "shared" libraries. Even if implemented in unix-like systems as just "shared" libraries, it's still a special case of shared library, not the general case.
Unfortunately Boost.Module might get confused with forthcoming C++ Modules. Otherwise I wouldn't object to that name.
Indeed. (I wasn't proposing such name, for the same reason).
Now, if you did want to implement a Boost Plugins library, combining John Bandela's CppComponents (https://github.com/jbandela/cppcomponents) header only library which turns C++ objects into portable Microsoft COM compatible objects and Boost.ASIO as the event dispatch loop would probably be an optimal solution. Not only does that seamlessly plug into all Microsoft technologies, it also is an easy way of SWIG exposing your object into all the languages SWIG supports. If anyone is interested in implementing this solution to Boost quality, I believe I might be able to get you hourly funding to do it - contact me off list.
Could you clarify what you mean by plugin here? I don't see why ASIO would be a nice dependency for such a tool, but I don't understand what you mean either.
ASIO is shortly to become the officially endorsed C++ event loop and dispatcher. As event processing is by definition part of any plugin or extension ABI - even if defined to be "no event processing", that means ASIO.
I would prefer to be able to use any event loop implementation instead of forcing asio, personally, because it's not a depencency of my projects right now and I already have another event loop which is specifically designed for my case (and is not general at all and require a very specific impl, for a very specific feature). Basically I prefer composing library instead of using a library that assume that everybody uses the same event loop. I might be wrong though, I don't see the whole picture with asio yet. But otherwise I see your general point now.
So, just to be clear, if you load a Qt Plugin, the Qt event pump gets routed through ASIO's event pump. That way we can extend Vicente's monadic continuations framework easily to interoperate with all Qt threading and event objects. That also goes for all third party C++ libraries, and now you can mix and match loading Qt plugins with WinRT plugins with wxWidget plugins.
You may find my C++ Now 2014 position paper useful. http://arxiv.org/pdf/1405.3323.pdf, starting from page 15 onwards.
Just to be clear though, my earlier offer of potential paid work for a CppComponents based plugin system integrated with ASIO is a small subset of the C++ Now 2014 paper proposal, and is much, much easier. My current employer may decide on a Microsoft COM driven SWIG language bindings, and CppComponents is a quick way of metaprogramming the C++ compiler to spit out Microsoft COM compliant binaries. As always, there is a lack of available sufficiently experienced engineers on this.
Niall
-- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 15 Aug 2014 at 20:56, Klaim - Joël Lamotte wrote:
A plugin/extension implies a well defined exported symbols and event processing ABI. Something adhering to Microsoft COM is a good example.
Antony's library is far more low level, it really does just thinly wrap the two popular shared library APIs into a portable API. It says nothing about what the ABI should be other than C compatible, any form of inspection other than C format symbols, or anything about any event model. I wouldn't mind calling his library a plugin solution for C, but it isn't up to task for C++.
I see. To me it's a good starting point and I fear that a more C++ than C library ends-up like boost.extension. But if it's not too hard to match your definition of plugin, it would be a win for me too as a user.
My objection is purely to the use of the name "Boost.Plugin" given what the library currently does. Even "Boost.BinaryLoader" is a better name.
I don't see why ASIO would be a nice dependency for such a tool, but I don't understand what you mean either.
ASIO is shortly to become the officially endorsed C++ event loop and dispatcher. As event processing is by definition part of any plugin or extension ABI - even if defined to be "no event processing", that means ASIO.
I would prefer to be able to use any event loop implementation instead of forcing asio, personally, because it's not a depencency of my projects right now and I already have another event loop which is specifically designed for my case (and is not general at all and require a very specific impl, for a very specific feature). Basically I prefer composing library instead of using a library that assume that everybody uses the same event loop. I might be wrong though, I don't see the whole picture with asio yet.
Sorry, I didn't explain myself clearly: ASIO is the *master* event loop. All other event loops remain themselves, it's just that they can adapt into the master event loop which is ASIO. You can compose as many custom event loop implementations as you like, that's up to you. This is possible because it turns out that ASIO's design is a superset of all other event loop designs I could find during when I researched this at BlackBerry. Therefore all other event loop designs can plug into ASIO with no loss of functionality, though with potential loss of low latency admittedly. I don't know if Chris intended this, but he achieved it nonetheless. And of course because ASIO will enter the C++ standard, it'll simply be there in the STL all the time. No dependency problems (in the future). Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On August 15, 2014 2:42:38 PM EDT, Niall Douglas <s_sourceforge@nedprod.com> wrote:
On 15 Aug 2014 at 19:22, Klaim - Joël Lamotte wrote:
It definitely has the wrong name because it doesn't implement plugins, it basically is a wrapper for the POSIX and Windows dynamic shared library functions. Boost.SharedLibrary is a far better name.
In my experience the "plugin" word is used as soon as a shared library could be loaded at an indeterminable time during execution (and be unloaded too). This is different from just "shared library" which is both a plugin implementation and a library linkage, including having the library automatically loaded at startup. To me plugin is far more precise than "shared library". You don't need such a library to make a shared library. You would need such a library to make a plugin (or extension, or module) library.
A plugin/extension implies a well defined exported symbols and event processing ABI. Something adhering to Microsoft COM is a good example.
What that term implies is based upon one's experience. "Plugin", to me is any dynamically loaded component. The ABI you mention depends entirely upon the ubiquity of the plugin. If it must be usable by many applications and customers, it must have a documented, even standardized, API. Just as easily, however, one can create plugins based solely upon dlopen() (or LoadLibrary()) and friends of the plugins are used only for one application.
Antony's library is far more low level, it really does just thinly wrap the two popular shared library APIs into a portable API. It says nothing about what the ABI should be other than C compatible, any form of inspection other than C format symbols, or anything about any event model. I wouldn't mind calling his library a plugin solution for C, but it isn't up to task for C++.
Even CMake call them "module", not "shared" libraries. Even if implemented in unix-like systems as just "shared" libraries, it's still a special case of shared library, not the general case.
Unfortunately Boost.Module might get confused with forthcoming C++ Modules. Otherwise I wouldn't object to that name.
I use "dynamic library" as the generic name in my library. ___ Rob (Sent from my portable computation engine)
On 15 Aug 2014 at 17:34, Rob Stewart wrote:
A plugin/extension implies a well defined exported symbols and event processing ABI. Something adhering to Microsoft COM is a good example.
What that term implies is based upon one's experience. "Plugin", to me is any dynamically loaded component.
I agree. But this library doesn't load components, it loads shared object binaries.
The ABI you mention depends entirely upon the ubiquity of the plugin. If it must be usable by many applications and customers, it must have a documented, even standardized, API. Just as easily, however, one can create plugins based solely upon dlopen() (or LoadLibrary()) and friends of the plugins are used only for one application.
API != ABI. ABI is standardised way of describing an API to external code. dlsym/GetProcAddress adhere loosely to the C ABI, and aren't particularly useful for C++. Even if you added a mangled symbol to partial AST decoder you still only have a toolkit with which one /could/ build a Plugin library, because one has yet to implement a dynamic invocation and lifetime management layer, plus an event processing standard. One also would find that some additional metadata is needed above what can be deduced from what the C ABI and RTTI makes available. John Bandela's CppComponents library has done a huge amount of all of this grunt work already, plus it's all header only and is a very easy way of making your C++ 14 code available to anything which can speak Microsoft COM. He's presented at both the 2013 and 2014 C++ Now's, his talks are well worth viewing. I agree that Microsoft COM reduces your C++ API to an early 1990s level of C++, but that unfortunately is still the current state of the art because of the wider C++ community's continuing failure to resolve Exported Templates - though, with C++ Modules we are most of the way there now, indeed Chandler was joking last C++ Now that clang might just implement exported templates for fun as they are trivial once Modules are fully implemented. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On August 16, 2014 12:27:20 PM EDT, Niall Douglas <s_sourceforge@nedprod.com> wrote:
On 15 Aug 2014 at 17:34, Rob Stewart wrote:
A plugin/extension implies a well defined exported symbols and event processing ABI. Something adhering to Microsoft COM is a good example.
What that term implies is based upon one's experience. "Plugin", to me is any dynamically loaded component.
I agree. But this library doesn't load components, it loads shared object binaries.
Your reading too much into "plugin". I don't consider that term to require what you think it implies.
The ABI you mention depends entirely upon the ubiquity of the plugin. If it must be usable by many applications and customers, it must have a documented, even standardized, API. Just as easily, however, one can create plugins based solely upon dlopen() (or LoadLibrary()) and friends of the plugins are used only for one application.
API != ABI.
ABI is standardised way of describing an API to external code. dlsym/GetProcAddress adhere loosely to the C ABI, and aren't particularly useful for C++. Even if you added a mangled symbol to partial AST decoder you still only have a toolkit with which one /could/ build a Plugin library, because one has yet to implement a dynamic invocation and lifetime management layer, plus an event processing standard. One also would find that some additional metadata is needed above what can be deduced from what the C ABI and RTTI makes available.
Did you notice that my paragraph began with ABI? The switch to API was accidental. I fully comprehend the two. As for all that you described, it is not the minimum required to make such a library useful.
John Bandela's CppComponents library has done a huge amount of all of this grunt work already, plus it's all header only and is a very easy way of making your C++ 14 code available to anything which can speak Microsoft COM. He's presented at both the 2013 and 2014 C++ Now's, his talks are well worth viewing. I agree that Microsoft COM reduces your C++ API to an early 1990s level of C++, but that unfortunately is still the current state of the art because of the wider C++ community's continuing failure to resolve Exported Templates - though, with C++ Modules we are most of the way there now, indeed Chandler was joking last C++ Now that clang might just implement exported templates for fun as they are trivial once Modules are fully implemented.
What you describe is fine, when you need all of it, but overkill when you don't. The OP has agreed to change the name. It doesn't need the features you describe to be useful and those features can be built atop this library. ___ Rob (Sent from my portable computation engine)
On 08/15/2014 11:34 PM, Rob Stewart wrote:
What that term implies is based upon one's experience. "Plugin", to me is any dynamically loaded component. The ABI you mention depends entirely upon the ubiquity of the plugin. If it must be usable by many applications and customers, it must have a documented, even standardized, API. Just as easily, however, one can create plugins based solely upon dlopen() (or LoadLibrary()) and friends of the plugins are used only for one application.
My interpretation of a "plugin" is a bit different. Normally a component decides its own interface and the application has to adjust to it. In the opposite case, when an application dictates the interface and the component has to adjust, then it is called a plugin. Dynamically loading is orthogonal to this.
On August 17, 2014 8:05:32 AM EDT, Bjorn Reese <breese@mail1.stofanet.dk> wrote:
On 08/15/2014 11:34 PM, Rob Stewart wrote:
What that term implies is based upon one's experience. "Plugin", to me is any dynamically loaded component. The ABI you mention depends entirely upon the ubiquity of the plugin. If it must be usable by many applications and customers, it must have a documented, even standardized, API. Just as easily, however, one can create plugins based solely upon dlopen() (or LoadLibrary()) and friends of the plugins are used only for one application.
My interpretation of a "plugin" is a bit different. Normally a component decides its own interface and the application has to adjust to it. In the opposite case, when an application dictates the interface and the component has to adjust, then it is called a plugin. Dynamically loading is orthogonal to this.
I was a little loose in my description. I didn't mean to suggest that just calling dlopen() means you have a plugin. Your plugin description is reasonable. ___ Rob (Sent from my portable computation engine)
Hi Antony, This is very much needed! I've tinkered with possible plugin designs and this reminds me to look back at it. Have you ever looked at ELFIO? http://sourceforge.net/projects/elfio/ It has no dependency library(not even libdl) to read/write elf files. My only stumbling block was I didn't get through a 64 bit test case yet and stepped away because the developers of ELFIO weren't responding. But I really am impressed with the elfio design and functionality I wanted to apply it to plugins as well as a new way for xml binding for c++ (but there are only so many hours in a day:-) On 08/15/2014 07:36 AM, Antony Polukhin wrote:
Hi,
There is a basic implementation of a header only Plugin library: https://github.com/apolukhin/Boost.Plugin
Documentation with examples could be found here: http://apolukhin.github.io/Boost.Plugin/index.html
Is there an interest in such library in Boost community? What features are missing in the library?
Looking forward to hearing from you. Any feedback is more than welcomed.
On Fri, Aug 15, 2014 at 8:36 AM, Antony Polukhin <antoshkka@gmail.com> wrote:
Hi,
There is a basic implementation of a header only Plugin library: https://github.com/apolukhin/Boost.Plugin
Documentation with examples could be found here: http://apolukhin.github.io/Boost.Plugin/index.html
Antony, I understand that this is the product of a split of Boost.Application. How does Boost.Application relate to Boost.Plugin after the split? Does Boost.Application depend on Boost.Plugin or vice versa? Also, this could change the review process for Boost.Application. If Boost.Plugin is considered, it's review would be separate. Can anyone confirm this? Finally, +1 for the rename to SharedLibrary. Regards, Madera
2014-08-15 22:00 GMT+04:00 Rodrigo Madera <rodrigo.madera@gmail.com>: <...>
How does Boost.Application relate to Boost.Plugin after the split? Does Boost.Application depend on Boost.Plugin or vice versa?
There's no dependencies between those two libraries.
Also, this could change the review process for Boost.Application. If Boost.Plugin is considered, it's review would be separate. Can anyone confirm this?
Yes, it would be two different reviews.
Finally, +1 for the rename to SharedLibrary.
How about Dynamic Library Load and namespace boost::dll:: ?
Regards, Madera
Thanks for the comments! -- Best regards, Antony Polukhin
There is a basic implementation of a header only Plugin library: https://github.com/apolukhin/Boost.Plugin
Documentation with examples could be found here: http://apolukhin.github.io/Boost.Plugin/index.html
Is there an interest in such library in Boost community? What features are missing in the library?
Looking forward to hearing from you. Any feedback is more than welcomed.
Caveat: I might have missed part or all of this in your docs and any/all of those things might be already available.
From our experience a plugin library should additionally to your design support:
- arbitrary constructor arguments to the plugin object created: typedef boost::shared_ptr<my_plugin_api> (pluginapi_create_t)(); boost::function<pluginapi_create_t> creator = boost::plugin::shared_function_alias<pluginapi_create_t>( shared_library_path, "create_plugin"); boost::shared_ptr<my_plugin_api> plugin = creator(42, "foo", 3.14); - the ability to extract the full file system path of a loaded shared library (this is not always the same as the file name you specified to load the library) - the ability to extract a list of plugin object names from a given shared library ( I assume that your library allows to have several plugin types in the same shared library) - support for binding/instantiating plugin objects even if the plugin is linked as a static library (some HPC systems do not support dynamic/shared libraries and this functionality allows to keep the code which using the plugins unchanged) - it is important that the pointer to the plugin object keeps the shared library loaded All of this is functionality we ended up implementing in HPX's plugin system (see here: https://github.com/STEllAR-GROUP/hpx/tree/master/hpx/util/plugin). Regards Hartmut --------------- http://boost-spirit.com http://stellar.cct.lsu.edu
I'll try to answer all common questions in a single letter: 2014-08-16 3:39 GMT+04:00 Hartmut Kaiser <hartmut.kaiser@gmail.com>: <...>
From our experience a plugin library should additionally to your design support:
- arbitrary constructor arguments to the plugin object created:
typedef boost::shared_ptr<my_plugin_api> (pluginapi_create_t)(); boost::function<pluginapi_create_t> creator = boost::plugin::shared_function_alias<pluginapi_create_t>( shared_library_path, "create_plugin");
boost::shared_ptr<my_plugin_api> plugin = creator(42, "foo", 3.14);
Yes, you may export/import *any* functions with any signatures in any namespace using the Plugin library. ABI problem and name mangling is solved by BOOST_PLUGIN_ALIAS macro, that produces a portable C name of an exported function or variable. That's all what can be done. Constructing a C structures from C++ classes for each C++ class in attempt to make std::string/std::vector portable across library boundaries looks very unreasonable to me. Compiler developers and library users must take care of it, not the library itself. I'll add this notes from above to the docs.
- the ability to extract the full file system path of a loaded shared library (this is not always the same as the file name you specified to load the library)
This would be a good addition. Thanks!
- the ability to extract a list of plugin object names from a given shared library ( I assume that your library allows to have several plugin types in the same shared library)
Could be queried using the bool res = lib.search_symbol("some_name_of_the_plugin")
- support for binding/instantiating plugin objects even if the plugin is linked as a static library (some HPC systems do not support dynamic/shared libraries and this functionality allows to keep the code which using the plugins unchanged)
Yes, it is supported via shared_library::load_self()
- it is important that the pointer to the plugin object keeps the shared library loaded
Supported via shared_* functions. 2014-08-15 20:44 GMT+04:00 Niall Douglas <s_sourceforge@nedprod.com>: <...>
Now, if you did want to implement a Boost Plugins library, combining John Bandela's CppComponents (https://github.com/jbandela/cppcomponents) header only library which turns C++ objects into portable Microsoft COM compatible objects and Boost.ASIO as the event dispatch loop would probably be an optimal solution.
Let's take a look at the situation from a bigger distance. There are 3-5 known open source plugin libraries based or proposed for Boost. There are hundredths plugin systems in commercial products. All those plugins have a common part - shared_library class. Other parts differ a lot: * factory methods signatures * plugin api * plugin names * event loops * OS specific techlologoies support (COM) Almost all of those differences can not be unified/solved in a common way. Any solution would not be generic enough. That's why Plugin library does not force any plugin API or does not specify event loops. It just stopped at the point where the differences begin. There are interesting things in each existing plugin system. Example from cppcomponents looks great: #include "library.h" int main() { Person p; std::cout << p.SayHello(); } I'll take a deeper look into it. Thanks for the good hint! HPC's plugins have some good functions that are missing in subject library. I'll take a deeper look into it. Thanks for the good hint! Answering to all the naming proposals: SharedLibray/DLL/DynamicLinkLibrary sound too platform specific. Plugin sounds more generic, thou it confuses users. How about Dynamic Library Load and namespace boost::dll:: ? -- Best regards, Antony Polukhin
On Sat, Aug 16, 2014 at 10:12 AM, Antony Polukhin <antoshkka@gmail.com> wrote:
Answering to all the naming proposals: SharedLibray/DLL/DynamicLinkLibrary sound too platform specific. Plugin sounds more generic, thou it confuses users. How about Dynamic Library Load and namespace boost::dll:: ?
Dynamic Library seems to be a nice compromise. By the way, would it be problematic to reuse the "extension" name?
I'll try to answer all common questions in a single letter:
2014-08-16 3:39 GMT+04:00 Hartmut Kaiser <hartmut.kaiser@gmail.com>: <...>
From our experience a plugin library should additionally to your design support:
- arbitrary constructor arguments to the plugin object created:
typedef boost::shared_ptr<my_plugin_api> (pluginapi_create_t)(); boost::function<pluginapi_create_t> creator = boost::plugin::shared_function_alias<pluginapi_create_t>( shared_library_path, "create_plugin");
boost::shared_ptr<my_plugin_api> plugin = creator(42, "foo", 3.14);
Yes, you may export/import *any* functions with any signatures in any namespace using the Plugin library.
ABI problem and name mangling is solved by BOOST_PLUGIN_ALIAS macro, that produces a portable C name of an exported function or variable. That's all what can be done. Constructing a C structures from C++ classes for each C++ class in attempt to make std::string/std::vector portable across library boundaries looks very unreasonable to me. Compiler developers and library users must take care of it, not the library itself.
I'll add this notes from above to the docs.
Good. Nice to hear that I can have 'virtual' constructors with your library.
- the ability to extract a list of plugin object names from a given shared library ( I assume that your library allows to have several plugin types in the same shared library)
Could be queried using the bool res = lib.search_symbol("some_name_of_the_plugin")
That's not what I meant. You're suggesting to find a known symbol. I was suggesting to find the list of names for the existing (unknown) plugins. Thanks! Regards Hartmut --------------- http://boost-spirit.com http://stellar.cct.lsu.edu
2014-08-16 16:29 GMT+04:00 Hartmut Kaiser <hartmut.kaiser@gmail.com>: <...>
- the ability to extract a list of plugin object names from a given shared library ( I assume that your library allows to have several plugin types in the same shared library)
Could be queried using the bool res = lib.search_symbol("some_name_of_the_plugin")
That's not what I meant. You're suggesting to find a known symbol. I was suggesting to find the list of names for the existing (unknown) plugins.
This can be currently achieved using the following: // in plugin/dll namespace foo { std::vector<std::string> list_plugins(); } BOOST_PLUGIN_ALIAS(foo::list_plugins, list_plugins) // in code that uses plugin shared_library lib(path_to_lib); auto plugins = lib.get<std::vector<std::string>()>("list_plugins"); for (auto& s: plugins) { ... lib.get</**/>(s) ... } Hmmm... This does not look user-friendly. I could do some experiments with putting the aliased names into a separate section of the binary and iterating over values in that section. Will iteration over BOOST_PLUGIN_ALIASed symbols suit you? Iterating over all the exported symbols does not look useful: there;ll be a lot of mangled names and sompiler-specific trash. -- Best regards, Antony Polukhin
On 16 Aug 2014 at 12:12, Antony Polukhin wrote:
Let's take a look at the situation from a bigger distance. There are 3-5 known open source plugin libraries based or proposed for Boost. There are hundredths plugin systems in commercial products. All those plugins have a common part - shared_library class. Other parts differ a lot: * factory methods signatures * plugin api * plugin names * event loops * OS specific techlologoies support (COM)
Almost all of those differences can not be unified/solved in a common way. Any solution would not be generic enough. That's why Plugin library does not force any plugin API or does not specify event loops. It just stopped at the point where the differences begin.
Oh I totally understand, absolutely. I think where you stopped is just about right. Your design is good too.
Answering to all the naming proposals: SharedLibray/DLL/DynamicLinkLibrary sound too platform specific. Plugin sounds more generic, thou it confuses users. How about Dynamic Library Load and namespace boost::dll:: ?
I wonder if Microsoft have trademarked "DLL"? :) If not, that works for me. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
2014-08-16 20:35 GMT+04:00 Niall Douglas <s_sourceforge@nedprod.com>:
On 16 Aug 2014 at 12:12, Antony Polukhin wrote:
Answering to all the naming proposals: SharedLibray/DLL/DynamicLinkLibrary sound too platform specific. Plugin sounds more generic, thou it confuses users. How about Dynamic Library Load and namespace boost::dll:: ?
I wonder if Microsoft have trademarked "DLL"? :)
Possible trademark threat is exactly why name "Plugin" was chosen over "DLL" :) -- Best regards, Antony Polukhin
On 15 Aug 2014 at 18:39, Hartmut Kaiser wrote:
- the ability to extract the full file system path of a loaded shared library (this is not always the same as the file name you specified to load the library)
Yes, a very good point. I can supply some reference code once working on all major platforms which could be derived from to implement this. The code can take any arbitrary pointer to memory and tell you which shared library "owns" it. The code is not fast, but it can be cached and it's very, very useful for implementing everything from per-plugin human language string translations to debugging tracebacks.
- the ability to extract a list of plugin object names from a given shared library ( I assume that your library allows to have several plugin types in the same shared library)
A mangled symbol to partial AST generator is the ultimate solution here and would be an enormously valuable addition to Boost. It's probably its own standalone library though. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
On 2014-08-15 07:36, Antony Polukhin wrote:
There is a basic implementation of a header only Plugin library: https://github.com/apolukhin/Boost.Plugin
Documentation with examples could be found here: http://apolukhin.github.io/Boost.Plugin/index.html
Is there an interest in such library in Boost community? What features are missing in the library?
Looking forward to hearing from you. Any feedback is more than welcomed.
Yes, tons of interest. Looking at the documentation, 2 features seem lacking; correct me if I am wrong. 1) Notification of shared library unloading and loading. If users and library/framework stick to the reference counting, which is recommended, how can a framework get an opportunity to do things when the end users/references finally disappear. While loading I may want to read a manifest, add services to a services registry. While unloading I may delete a dated version of the shared library as a more recent dated version of the same library has already been loaded. ... For web development or any development I would like to hot swap to speed development by shortening the redeploy test cycle like found in Java JSPs, Java JRebel or any dynamic language. To build limited interface based variations of this, notification seems essential. 2) How do we make our plugins object methods return references to other shared pointers which is also tied to the shared library counter? A better nested object example is in order.
2014-08-31 19:18 GMT+04:00 <jwaterloo@dynamicquest.com>:
On 2014-08-15 07:36, Antony Polukhin wrote:
There is a basic implementation of a header only Plugin library:
https://github.com/apolukhin/Boost.Plugin
Documentation with examples could be found here: http://apolukhin.github.io/Boost.Plugin/index.html
Is there an interest in such library in Boost community? What features are missing in the library?
Looking forward to hearing from you. Any feedback is more than welcomed.
Yes, tons of interest.
Looking at the documentation, 2 features seem lacking; correct me if I am wrong.
1) Notification of shared library unloading and loading. If users and library/framework stick to the reference counting, which is recommended, how can a framework get an opportunity to do things when the end users/references finally disappear. While loading I may want to read a manifest, add services to a services registry. While unloading I may delete a dated version of the shared library as a more recent dated version of the same library has already been loaded. ... For web development or any development I would like to hot swap to speed development by shortening the redeploy test cycle like found in Java JSPs, Java JRebel or any dynamic language. To build limited interface based variations of this, notification seems essential.
Well, in DLL/DSO you can track loads unloads using some global variables: struct do_at_load { do_at_load() { std::cout << "Loading\n"; } ~do_at_load() { std::cout << "Unloading\n"; } }; do_at_load instance; I'll update the examples to reflect this.
2) How do we make our plugins object methods return references to other shared pointers which is also tied to the shared library counter? A better nested object example is in order.
This is up to the plugin developer. However a few functions could make a developer's life easier. I was thinking of adding a function `shared_ptr<shared_library> this_library();` which would return an instance of a shared library in which `this_library()` function was called. Thanks to Niall, now I have some code that does things close to `this_library()`. -- Best regards, Antony Polukhin
On 2 Sep 2014 at 11:14, Antony Polukhin wrote:
1) Notification of shared library unloading and loading.
Well, in DLL/DSO you can track loads unloads using some global variables:
I think he meant that he wants the ability to hook into the load/unload of some arbitrary DLL/DSO, so like a DLL/DSO local atexit() as it were. This is trivial to implement, simply a static object defined per DLL/DSO which accepts a list of bound functions to call when it gets destructed. Something kinda cool would be combining that hook with the pointer-to-DLL/DSO code I sent you, so you could have code like: // Tell me when the DLL/DSO owning this string gets unloaded so I can // delete my use of it dll_from_ptr(str)->on_unload(std::bind(this, &remove_string, str)); Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
2014-09-02 13:49 GMT+04:00 Niall Douglas <s_sourceforge@nedprod.com>:
On 2 Sep 2014 at 11:14, Antony Polukhin wrote:
1) Notification of shared library unloading and loading.
Well, in DLL/DSO you can track loads unloads using some global variables:
I think he meant that he wants the ability to hook into the load/unload of some arbitrary DLL/DSO, so like a DLL/DSO local atexit() as it were.
This won't be supported out of the box because there's no nice way to do it without overhead for users that do not require such functionality.
This is trivial to implement, simply a static object defined per DLL/DSO which accepts a list of bound functions to call when it gets destructed.
Something kinda cool would be combining that hook with the pointer-to-DLL/DSO code I sent you, so you could have code like:
// Tell me when the DLL/DSO owning this string gets unloaded so I can // delete my use of it dll_from_ptr(str)->on_unload(std::bind(this, &remove_string, str));
You are right. Added an example how to do it using the existing functionality of the library. Example took about 30 lines of code (including #includes, namespaces, comments, class decorations). -- Best regards, Antony Polukhin
participants (11)
-
Antony Polukhin -
Bjorn Reese -
Hartmut Kaiser -
Ion Gaztañaga -
Jason Roehm -
jwaterloo＠dynamicquest.com -
Klaim - Joël Lamotte -
Niall Douglas -
Rob Stewart -
Rodrigo Madera -
Roger Martin