[OpenMethod] A pre-release feedback
Hi Everyone, The library review starts still in two weeks, but I wanted to give some early feedback. First, I wanted to thank Jean-Louis for exploring the subject, writing and sharing the library, and for submitting it for the Boost Review. My feedback is based on the reading of the documentation, I haven't played with the library just yet. [1]. Animals as examples There are already a number of things in the library that suggest it might not be suitable for production use (I describe them below), but using class Animal as explanatory examples amplifies the impression that this is going to be a toy library. Half of the books I have read on OO use the Animal (or Employee) example, but never in my C++ career did I have to use OO to model something close to "animal". I know they are just examples, but picking one so detached from programming reality is in my estimation harmful. The arithmetic expression example is way better. I, for one, use OO for representing dynamically-configurable parts of the applications, such as plugins or user-selectable sorting or selection predicates. [2]. The motivation In C++ we already have a dynamic-dispatch tool for "closed set of methods with open set of types": virtual functions. We also have a dynamic-dispatch tool for "open set of methods with closed set of types": the visitation of std::variant. You need to provide a motivating example (more convincing than the zoo) that needs exactly this: "open set of methods with the open set of types". The arithmetic expression example doesn't do the job: there is a fixed and well known list of concrete types. Also, the fact that Bjarne Stroustrup wanted to add open methods long ago is insufficient a motivation. [3]. Are open methods compatible with static type checking? It looks that when one of the two things -- number of types, number of methods -- is known statically, the compiler can verify if the programmer didn't forget to take some type or method into account. (Well, sometimes we get the "pure virtual function call" error, but these are very rare cases.) The same seems not to be the case for open methods. What this library advertises as flexibility would be the source of many programmer bugs detected only at runtime, possibly by end users. Imagine that programmer Alice is adding her new type A and plugs it into open methods known to her, and at the same time programmer Bob in another file adds a new open method B known only to him. Bob doesn't know about class A, and Alice doesn't know about open method B. The ISO C++ proposals quoted in the docs mentioned solving this by a mandated or encouraged linker error. Is it possible for this library to diagnose the incomplete specification of an open method in the `initialize()` function? The docs use an expression, "when an error is encountered [...]". But note that in C++ we often use the term "error" to express the situation where a _correct_ program encounters a hostile situation in the environment (lack of resources, bad user input). But openmethod docs actually mean something different: a programmer bug. It is very easy for a programmer to plant a bug in this flexible design. [4] In a similar vein, the decision to select an arbitrary specialization of an open method upon a tie, is another runtime surprise: not even a reported error. For multi methods it is not even clear how a static check for exhaustive unambiguous specialization should work. [5] It is concerning that so many things are defined in the global namespace. The docs say, "First we need to include the library’s main header. It defines a few macros, and injects a name - virtual_ptr - in the global namespace." But then the reference part says that virtual_ptr is in the namespace boost::openmethod. Who is right? The fact that as common a name as `next` is reserved inside open methods is also uncomfortable. Why not use a namespace-scoped name? Next, the docs use the using-directive: `using namespace boost::openmethod;`. This is a bad idea: both for real programs and for the documentation: it is not clear if the library was designed with namespaces in mind. Also, all the examples define open methods in the global namespace. I instead would like the docs to show how to declare open methods in my namespace, as this is a far more probable use case. [6] Friendship It puzzles me how the notion of friendship can be combined with this idea of flexibility in open methods. It is my understanding that open methods are added on top of the existing classes. That is: classes do not have to know that some open method will be added in the future. In other words, open methods are non-intrusive, right? So how would a class know who to grant the friendship to? Also, the examples use the declaration `friend struct`, even though we are talking about methods. I think docs should mention the existence of these structs more prominently. [7] Function signatures Some macros in the library use function types in declarations: `BOOST_OPENMETHOD_OVERRIDERS(poke)<void(std::ostream&, virtual_ptr<Cat>)>` Others specify the type separately: `BOOST_OPENMETHOD_OVERRIDE(value, (virtual_ptr<Plus> node), int)` Is there a reason for this? Can the clever use of macros not offer the same function signature syntax? [8] Passing virtual pointers to const objects If `virtual_ptr<Cat>` is an analogue of this pointer, could we see an example in the docs that shows a virtual pointer to const objects? It would be `virtual_ptr<const Cat>`, I guess? [9] Is this library intended for production use? The library comes with a lot of clever macros, a lot of pointer type templates, a lot of new syntax and runtime surprises (mentioned earlier). I am concerned about how the debugging would look if something goes wrong with the usage of this library. Would you recommend it for use in production systems? Or is it for experimentation? [10] Minor things Section Custom RTTI has sentence:
Now we can include the "core" header
and then header `boost/openmethod/core.hpp` is not included. Is it a bug? Regards, &rzej;
Hi Andrzej, Thank you for taking the time to take a look at my library. I agree with some of your points, and I will try to explain some of the choices I made. Before going over your remarks, a bit of context: OpenMethod is derived from YOMM2 (https://github.com/jll63/yomm2, started in 2017), which is a rewrite of YOMM11 (2013). I presented it at several conferences, the most prestigious being CppCon, in 2018. My most recent talk was at Core C++ in Israel, last year. The latest video of a talk is from using std::cpp in Madrid, here: https://youtu.be/cOYE6OiuuMo?si=wMsSADt3_sEC-qz1. I know that YOMM2 has been used in production in several places (like the Tokyo Institute of Technology), by people who cared enough to send me PRs, bug reports and suggestions. Also, vanity Google searches turned up conversations with positive remarks on YOMM2, between people I never heard of.
[1]. Animals as examples
There are already a number of things in the library that suggest it might not be suitable for production use (I describe them below), but using class Animal as explanatory examples amplifies the impression that this is going to be a toy library.
The arithmetic expression example is way better.
I agree that Animals doesn't look "serious". If you watch my talks, you will see that I never use it. My favorite would be Matrix: it uses concepts everybody knows already, AND it can be used to demonstrate both uni- and multi-methods. It deals with some interesting cases, like matrix transposition, where you probably want to return the same matrix if it is symmetrical. But it may give the impression that it's all about multiple dispatch. Nay, "multi" is just the cherry on the cake. My second most favorite is the AST. But it does not demonstrate multiple dispatch, nor `next`. Animals, on the other hand, has all three: uni, multi, next. The Dog/Bulldog example is the best way I've found to illustrate `next` in a snap. That is why I use Animals in the tutorial, but I am open to switching to another example.
[2]. The motivation
In C++ we already have a dynamic-dispatch tool for "closed set of methods with open set of types": virtual functions. We also have a dynamic-dispatch tool for "open set of methods with closed set of types": the visitation of std::variant.
Why can't we have both at the same time? The very fact that we have those two tools proves that extensibility is desirable along both axes: behaviors and types. Currently there is no easy way to write a library that is extensible in both directions. I believe that tying polymorphism to membership was the main cause of the failure of the Simula/Smalltalk/C++ way of OOP. Lisp was right.
You need to provide a motivating example (more convincing than the zoo) that needs exactly this: "open set of methods with the open set of types". The arithmetic expression example doesn't do the job: there is a fixed and well known list of concrete types.
The papers authored by Stroustrup and his PhD students Pirkelbauer and Solodkyy contain discussions of several examples. The most "recent" is N2216 (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2216.pdf). I disagree about AST. The narrative is: I have that AST library. One of my apps needs to represent the AST in JSON. If I own the library, I can add a virtual function, but then all the other apps get it, need it or not. And also the function's dependencies, e.g. the iostream library if `to_json` writes to a `std::ostream`. I should explain in the tutorial how open-methods can help with decoupling. It's not just about the Expression Problem. It's also about the banana-gorilla-jungle problem. If AST is a 3rd party library, it might provide a Visitor, if I'm "lucky". But Visitor is so clumsy. See here https://github.com/jll63/Boost.OpenMethod/blob/master/examples/slides.cpp for implementations of `to_json` using type switches, function pointers, Visitor and open-methods. This talk is worth watching too: https://www.youtube.com/watch?v=QhJguzpZOrk . When a program starts running, the set of operations and types is usually fixed (barring dynamic loading). Many such programs still use virtual functions, for many good reasons. They don't exist in isolation; they are part of a collection of programs that use internal and 3rd party libraries. It's about architecture. Ultimately, open-methods are a better version of virtual functions. More flexible, not toxic.
Also, the fact that Bjarne Stroustrup wanted to add open methods long ago is insufficient a motivation.
It is insufficient, but _not_ _insignificant_. I have a megaton of respect for the man and his intelligence. D&E is one of my favorite books.
[3]. Are open methods compatible with static type checking?
It looks that when one of the two things -- number of types, number of methods -- is known statically, the compiler can verify if the programmer didn't forget to take some type or method into account. [...] The same seems not to be the case for open methods.
Correct. Full static validation can be done only with a complete view of the entire program.
The ISO C++ proposals quoted in the docs mentioned solving this by a mandated or encouraged linker error. Is it possible for this library to diagnose the incomplete specification of an open method in the `initialize()` function?
Yes. `initialize()` returns an object of an unspecified type that contains various metrics and diagnostics, like the number of methods missing overrides for some combinations of virtual arguments. By the time the review starts, I will document some of this. Also, the trace contains that info. I also have plans for a global analysis tool, based on clang, that would create the dispatch tables as a generated source file; it would also generate hints for static offsets in the v-tables, thus beating the speed of Stroustrup's prototype. That tool would also report possible errors, without the need to run the program.
[4] In a similar vein, the decision to select an arbitrary specialization of an open method upon a tie, is another runtime surprise: not even a reported error. For multi methods it is not even clear how a static check for exhaustive unambiguous specialization should work.
I don't like it either. YOMM2 reports an "error" in that case. Stroustrup & co argue that all best overriders are "correct", so why crash the program? Also, I am lukewarm on using return types as tie-breakers, in some circumstances. As I understand it, their reasoning is guided by considerations on dynamic loading. The number of ambiguities is available in `initialize()`'s return value - to be documented. I am open to adding a customization point to make ambiguous calls runtime errors.
[5] It is concerning that so many things are defined in the global namespace.
Please bear in mind that OpenMethod (like YOMM2) attempts to emulate a _language_ _feature_. The names it publishes are similar to language keywords. `virtual_ptr` is like `virtual`, and `virtual` is global ;-)
The docs say, "First we need to include the library’s main header. It defines a few macros, and injects a name - virtual_ptr - in the global namespace." But then the reference part says that virtual_ptr is in the namespace boost::openmethod. Who is right?
Both. <boost/openmethod.hpp> contains `using boost::openmethod::virtual_ptr`. OK I can change "injects" to "aliases". You can use the library with zero macro (well, just include guards) and zero "injections" if you like - that's <boost/openmethod/core.hpp>. And if you want the macros but not the alias, you can include <boost/openmethod/core.hpp> and <boost/openmethod/macros.hpp>. I will improve the doc on this... I am also not completely opposed to removing the injection of `virtual_ptr`.
The fact that as common a name as `next` is reserved inside open methods is also uncomfortable. Why not use a namespace-scoped name?
There are two `next`s. One is scoped in boost::openmethod::method (core API). The other is inside the body of a BOOST_OPENMETHOD_OVERRIDE (keyword API). It doesn't exist outside of it. I can rename it to something else. In CLOS, it is called call-next-method. In CLOS parlance, a method is a "generic" (defgeneric), and an overrider is a "method" (defmethod). `next` can become call_next_overrider, no biggie...
Next, the docs use the using-directive: `using namespace boost::openmethod;`. This is a bad idea: both for real programs and for the documentation: it is not clear if the library was designed with namespaces in mind.
Oh it was. And I thought a lot about `using namespace` in the doc. Reading documentation is not the same thing as writing a real-life large program. Long scoped names can get in the way while learning. The goal is different. I looked at the documentation of many Boost libraries. Some assume that their namespace is in scope. Others use the `namespace bom = boost::openmethod; ... bom::virtual_ptr` approach - which is just as "bad", right? But I am flexible on this...
Also, all the examples define open methods in the global namespace. I instead would like the docs to show how to declare open methods in my namespace, as this is a far more probable use case.
I should add an entry on namespaces to the tutorial. [6] Friendship
It puzzles me how the notion of friendship can be combined with this idea of flexibility in open methods. It is my understanding that open methods are added on top of the existing classes. That is: classes do not have to know that some open method will be added in the future. In other words, open methods are non-intrusive, right? So how would a class know who to grant the friendship to?
I used to agree with this. Targeted friendship seems so much in contradiction with openness. On the other hand, all features of C++ should play ball with one another. Friendship is part of the language. Also, I am not a nanny. I was often asked about friendship during the Q&A of my talks. I replied: You are probably doing something wrong if you need it, but I already have a design for it, and I will implement it when someone comes up with a good example where it makes sense. And then someone did. Discussion is here: https://github.com/jll63/yomm2/issues/7 Ponder this: polymorphism and encapsulation are orthogonal concepts, right? But traditional virtual functions have access to the private and protected parts. You just wanted polymorphism, but look!, you get access to private parts as well, like it or not. Well, with open-methods, you can have _just_ polymorphism. Better virtual functions, not toxic :)
Also, the examples use the declaration `friend struct`, even though we are talking about methods. I think docs should mention the existence of these structs more prominently.
Maybe. It is mentioned in two places (https://jll63.github.io/Boost.OpenMethod/#tutorials_friendship, https://jll63.github.io/Boost.OpenMethod/#BOOST_OPENMETHOD_OVERRIDE). I can see two reasons to need this info. You just want to grant friendship, in which case you can just follow the recipe; or you are doing more evolved things, like mixing core and keyword API and perhaps templates. In that case, you don't shy away from reading and decanting the reference, and I think all the info is there. But let's see what others say...
[7] Function signatures
Some macros in the library use function types in declarations: `BOOST_OPENMETHOD_OVERRIDERS(poke)<void(std::ostream&, virtual_ptr<Cat>)>`
Others specify the type separately: `BOOST_OPENMETHOD_OVERRIDE(value, (virtual_ptr<Plus> node), int)`
Is there a reason for this? Can the clever use of macros not offer the same function signature syntax?
These macros came out the way they are after trying many alternatives. CPP macros are surprisingly versatile, but they're not Lisp macros. I would like the syntax to be: BOOST_OPENMETHOD(poke(virtual_ptr<Animal>, std::ostream&), void); BOOST_OPENMETHOD_OVERRIDE(poke(virtual_ptr<Dog> dog, std::ostream& os), void) BOOST_OPENMETHOD_OVERRIDERS(poke(std::ostream&, virtual_ptr<Cat>, void) But I don't think it's feasible. However, this is probably possible: BOOST_OPENMETHOD_OVERRIDERS(poke, (std::ostream&, virtual_ptr<Cat>), void) Which mirrors nicely: BOOST_OPENMETHOD_OVERRIDE(poke, (std::ostream&, virtual_ptr<Cat>, void) Let me try this, it looks like a good idea.
[8] Passing virtual pointers to const objects
If `virtual_ptr<Cat>` is an analogue of this pointer, could we see an example in the docs that shows a virtual pointer to const objects? It would be `virtual_ptr<const Cat>`, I guess?
Yep! There, it looks like it's quite predictable, innit? :) The tutorial is ready and willing to evolve. I wanted to start light because I feel that, over the years, YOMM2's doc has become bloated, disorganized...
[9] Is this library intended for production use?
Intended? Yes, definitely. Ready is a better question. YOMM2 has existed for eight years. It has preserved backward compatibility throughout. It has evolved a lot. I got bug reports. I fixed them. I know that it has been used in production in more than one place. Now OpenMethod is not only YOMM2, renamed. It also contains "new research", so I cannot honestly say it is as field-tested as YOMM2; yet the new ideas address real problems. For example, moving the method's return type to the end of the parameter lists of BOOST_OPENMETHOD and BOOST_OPENMETHOD_OVERRIDE. It is not (only) to look "more modern". It actually solves a real annoyance with YOMM2: it does not cope well with return types that contain commas, so you have to use workarounds.
The library comes with a lot of clever macros, a lot of pointer type templates, a lot of new syntax and runtime surprises (mentioned earlier). I am concerned about how the debugging would look if something goes wrong with the usage of this library.
The inherent reduction in static checking has always been on my mind. I say "inherent" because in presence of dynamic loading (which YOMM11, YOMM2 and OpenMethod all support), what looks like a bug when you look at the main program vanishes when you (dynamically) load a shared object or a DLL. You mentioned plugins, you see what I mean. In debug builds, the library performs all sorts of runtime checks. If you forget to register a class, you will get an error message. If you turn on trace, you will get a very detailed explanation of what classes, methods, and overriders the library knows about, and how they relate. It will explain how it selects the best and next best overriders for each virtual tuple. It will tell you which slots in the v-tables are assigned to which methods. If you trace through a method call with a debugger - something I have done a lot -, with a little bit of practice, you will understand what is going on. Even if you make use of customization points. The code is organized to be step-in friendly. Try it :)
Would you recommend it for use in production systems? Or is it for experimentation?
In 2013 I started my work on open-(multi-)methods for C++ because: - I used multi-methods in previous projects in other languages. - I worked on a project that would have really benefited from open-methods (open-uni-methods in this instance). - I fantasized that I could help Stroustrup promote the idea by giving people a way to experiment with it. And then I would retire YOMM. However, YOMM11, and its successors, were never "toys". For example, they build redundancy free dispatch tables for multi-methods. That is not necessary for proof-of-concept, but it is for a production capable library. Recent releases of YOMM2 added customization points to address real-life problems. For example, in the game industry, they disable RTTI, and use custom systems instead. Because games are an important use-case, I make it possible to interface with custom RTTI. I came to recognize that we will have a colony on Mars before open-methods make it into the standard. So I posit OpenMethod as the next best option.
[10] Minor things
Section Custom RTTI has sentence:
Now we can include the "core" header
and then header `boost/openmethod/core.hpp` is not included. Is it a bug?
`boost/openmethod.hpp` includes `boost/openmethod/core.hpp` but you are right, I'll fix the doc, thanks :)
pon., 14 kwi 2025 o 22:44 Jean-Louis Leroy via Boost <boost@lists.boost.org> napisał(a):
Before going over your remarks, a bit of context: OpenMethod is derived from YOMM2 (https://github.com/jll63/yomm2, started in 2017), which is a rewrite of YOMM11 (2013). I presented it at several conferences, the most prestigious being CppCon, in 2018. My most recent talk was at Core C++ in Israel, last year. The latest video of a talk is from using std::cpp in Madrid, here: https://youtu.be/cOYE6OiuuMo?si=wMsSADt3_sEC-qz1.
I know that YOMM2 has been used in production in several places (like the Tokyo Institute of Technology), by people who cared enough to send me PRs, bug reports and suggestions. Also, vanity Google searches turned up conversations with positive remarks on YOMM2, between people I never heard of.
Thank you. So I had a look at the YOMM2 documentation, and I must say I find the motivation in YOMM2 stronger. It mentions the "cross-cutting concern" and the to_json+Matrices example: they are concrete and practical. I think OpenMethod would benefit from taking the motivation section from YOMM2. I am not sure that I am convinced, even with YOMM2, but I am certainly more convinced.
[2]. The motivation
In C++ we already have a dynamic-dispatch tool for "closed set of methods with open set of types": virtual functions. We also have a dynamic-dispatch tool for "open set of methods with closed set of types": the visitation of std::variant.
Why can't we have both at the same time?
The very fact that we have those two tools proves that extensibility is desirable along both axes: behaviors and types. Currently there is no easy way to write a library that is extensible in both directions.
It is clear (to me) that we sometimes need extensibility along one axis, and sometimes along the other axis. It is not clear to me that we ever need extensibility along two axes at the same time. On the other hand, I can see how the extensibility along two axes is error-prone: there is no static checking anymore. There is no way for the author of one part of the app to convince themselves if they extended their part comprehensively, as they do not know what other parties are extending in other parts of the program.
I believe that tying polymorphism to membership was the main cause of the failure of the Simula/Smalltalk/C++ way of OOP. Lisp was right.
Even if this was true, this is not necessarily an argument in favour of open methods. You can also have a non-intrusive polymorphism on class hierarchies, as proposed in P2688R5 (pattern matching) and implemented in Mach7 (https://github.com/solodon4/Mach7).
You need to provide a motivating example (more convincing than the zoo) that needs exactly this: "open set of methods with the open set of types". The arithmetic expression example doesn't do the job: there is a fixed and well known list of concrete types.
The papers authored by Stroustrup and his PhD students Pirkelbauer and Solodkyy contain discussions of several examples. The most "recent" is N2216 (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2216.pdf).
Question: do users need to read the above paper to appreciate the library and understand why they would need it?
I disagree about AST. The narrative is: I have that AST library. One of my apps needs to represent the AST in JSON. If I own the library, I can add a virtual function, but then all the other apps get it, need it or not. And also the function's dependencies, e.g. the iostream library if `to_json` writes to a `std::ostream`.
I should explain in the tutorial how open-methods can help with decoupling. It's not just about the Expression Problem. It's also about the banana-gorilla-jungle problem.
The "expression problem" looks like an academic problem, or a logical challenge, but not a practical problem. The authors of N2216 (open methods) then switched to N3449 (pattern matching), as a more practical and safer solution to the same problem: you get the non-intrusive polymorphism and a guaranteed static checking.
If AST is a 3rd party library, it might provide a Visitor, if I'm "lucky". But Visitor is so clumsy. See here https://github.com/jll63/Boost.OpenMethod/blob/master/examples/slides.cpp for implementations of `to_json` using type switches, function pointers, Visitor and open-methods. This talk is worth watching too: https://www.youtube.com/watch?v=QhJguzpZOrk .
If you are comparing open methods to a visitor pattern, then I agree that the latter is clumsy. But if we compare open methods to a "type switch" (pattern matching), the latter is cleaner and safer (as in: "buggy usages are detected at compile-time or clearly visible").
When a program starts running, the set of operations and types is usually fixed (barring dynamic loading). Many such programs still use virtual functions, for many good reasons. They don't exist in isolation; they are part of a collection of programs that use internal and 3rd party libraries. It's about architecture.
Ultimately, open-methods are a better version of virtual functions. More flexible, not toxic.
Again, I think you will need to convince us that open methods are preferable to a type-switch (pattern matching). Based on my current understanding, I am inclined to think that the OpenMethod library is offering a suboptimal solution to what I recognize as real life problems. Regards, &rzej;
чт, 1 мая 2025 г. в 01:03, Andrzej Krzemienski via Boost <boost@lists.boost.org>:
Even if this was true, this is not necessarily an argument in favour of open methods. You can also have a non-intrusive polymorphism on class hierarchies, as proposed in P2688R5 (pattern matching) and implemented in Mach7 (https://github.com/solodon4/Mach7). ... The authors of N2216 (open methods) then switched to N3449 (pattern matching), as a more practical and safer solution to the same problem: you get the non-intrusive polymorphism and a guaranteed static checking.
Andrzej, if you don't find the time to write a review please still elaborate on this point in a regular post, I find it very interesting.
On Sat, May 3, 2025, 10:47 Дмитрий Архипов via Boost <boost@lists.boost.org> wrote:
чт, 1 мая 2025 г. в 01:03, Andrzej Krzemienski via Boost <boost@lists.boost.org>:
Even if this was true, this is not necessarily an argument in favour of open methods. You can also have a non-intrusive polymorphism on class hierarchies, as proposed in P2688R5 (pattern matching) and implemented in Mach7 (https://github.com/solodon4/Mach7). ... The authors of N2216 (open methods) then switched to N3449 (pattern matching), as a more practical and safer solution to the same problem:
you
get the non-intrusive polymorphism and a guaranteed static checking.
Andrzej, if you don't find the time to write a review please still elaborate on this point in a regular post, I find it very interesting.
I intend to submit a review within the review period. Just need a couple days to formulate it clearly. Regards, &rzej;
participants (3)
-
Andrzej Krzemienski -
Jean-Louis Leroy -
Дмитрий Архипов