Hi Andrzej, Thank you for taking the time to take a look at my library. I agree with some of your points, and I will try to explain some of the choices I made. Before going over your remarks, a bit of context: OpenMethod is derived from YOMM2 (https://github.com/jll63/yomm2, started in 2017), which is a rewrite of YOMM11 (2013). I presented it at several conferences, the most prestigious being CppCon, in 2018. My most recent talk was at Core C++ in Israel, last year. The latest video of a talk is from using std::cpp in Madrid, here: https://youtu.be/cOYE6OiuuMo?si=wMsSADt3_sEC-qz1. I know that YOMM2 has been used in production in several places (like the Tokyo Institute of Technology), by people who cared enough to send me PRs, bug reports and suggestions. Also, vanity Google searches turned up conversations with positive remarks on YOMM2, between people I never heard of.
[1]. Animals as examples
There are already a number of things in the library that suggest it might not be suitable for production use (I describe them below), but using class Animal as explanatory examples amplifies the impression that this is going to be a toy library.
The arithmetic expression example is way better.
I agree that Animals doesn't look "serious". If you watch my talks, you will see that I never use it. My favorite would be Matrix: it uses concepts everybody knows already, AND it can be used to demonstrate both uni- and multi-methods. It deals with some interesting cases, like matrix transposition, where you probably want to return the same matrix if it is symmetrical. But it may give the impression that it's all about multiple dispatch. Nay, "multi" is just the cherry on the cake. My second most favorite is the AST. But it does not demonstrate multiple dispatch, nor `next`. Animals, on the other hand, has all three: uni, multi, next. The Dog/Bulldog example is the best way I've found to illustrate `next` in a snap. That is why I use Animals in the tutorial, but I am open to switching to another example.
[2]. The motivation
In C++ we already have a dynamic-dispatch tool for "closed set of methods with open set of types": virtual functions. We also have a dynamic-dispatch tool for "open set of methods with closed set of types": the visitation of std::variant.
Why can't we have both at the same time? The very fact that we have those two tools proves that extensibility is desirable along both axes: behaviors and types. Currently there is no easy way to write a library that is extensible in both directions. I believe that tying polymorphism to membership was the main cause of the failure of the Simula/Smalltalk/C++ way of OOP. Lisp was right.
You need to provide a motivating example (more convincing than the zoo) that needs exactly this: "open set of methods with the open set of types". The arithmetic expression example doesn't do the job: there is a fixed and well known list of concrete types.
The papers authored by Stroustrup and his PhD students Pirkelbauer and Solodkyy contain discussions of several examples. The most "recent" is N2216 (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2216.pdf). I disagree about AST. The narrative is: I have that AST library. One of my apps needs to represent the AST in JSON. If I own the library, I can add a virtual function, but then all the other apps get it, need it or not. And also the function's dependencies, e.g. the iostream library if `to_json` writes to a `std::ostream`. I should explain in the tutorial how open-methods can help with decoupling. It's not just about the Expression Problem. It's also about the banana-gorilla-jungle problem. If AST is a 3rd party library, it might provide a Visitor, if I'm "lucky". But Visitor is so clumsy. See here https://github.com/jll63/Boost.OpenMethod/blob/master/examples/slides.cpp for implementations of `to_json` using type switches, function pointers, Visitor and open-methods. This talk is worth watching too: https://www.youtube.com/watch?v=QhJguzpZOrk . When a program starts running, the set of operations and types is usually fixed (barring dynamic loading). Many such programs still use virtual functions, for many good reasons. They don't exist in isolation; they are part of a collection of programs that use internal and 3rd party libraries. It's about architecture. Ultimately, open-methods are a better version of virtual functions. More flexible, not toxic.
Also, the fact that Bjarne Stroustrup wanted to add open methods long ago is insufficient a motivation.
It is insufficient, but _not_ _insignificant_. I have a megaton of respect for the man and his intelligence. D&E is one of my favorite books.
[3]. Are open methods compatible with static type checking?
It looks that when one of the two things -- number of types, number of methods -- is known statically, the compiler can verify if the programmer didn't forget to take some type or method into account. [...] The same seems not to be the case for open methods.
Correct. Full static validation can be done only with a complete view of the entire program.
The ISO C++ proposals quoted in the docs mentioned solving this by a mandated or encouraged linker error. Is it possible for this library to diagnose the incomplete specification of an open method in the `initialize()` function?
Yes. `initialize()` returns an object of an unspecified type that contains various metrics and diagnostics, like the number of methods missing overrides for some combinations of virtual arguments. By the time the review starts, I will document some of this. Also, the trace contains that info. I also have plans for a global analysis tool, based on clang, that would create the dispatch tables as a generated source file; it would also generate hints for static offsets in the v-tables, thus beating the speed of Stroustrup's prototype. That tool would also report possible errors, without the need to run the program.
[4] In a similar vein, the decision to select an arbitrary specialization of an open method upon a tie, is another runtime surprise: not even a reported error. For multi methods it is not even clear how a static check for exhaustive unambiguous specialization should work.
I don't like it either. YOMM2 reports an "error" in that case. Stroustrup & co argue that all best overriders are "correct", so why crash the program? Also, I am lukewarm on using return types as tie-breakers, in some circumstances. As I understand it, their reasoning is guided by considerations on dynamic loading. The number of ambiguities is available in `initialize()`'s return value - to be documented. I am open to adding a customization point to make ambiguous calls runtime errors.
[5] It is concerning that so many things are defined in the global namespace.
Please bear in mind that OpenMethod (like YOMM2) attempts to emulate a _language_ _feature_. The names it publishes are similar to language keywords. `virtual_ptr` is like `virtual`, and `virtual` is global ;-)
The docs say, "First we need to include the library’s main header. It defines a few macros, and injects a name - virtual_ptr - in the global namespace." But then the reference part says that virtual_ptr is in the namespace boost::openmethod. Who is right?
Both. <boost/openmethod.hpp> contains `using boost::openmethod::virtual_ptr`. OK I can change "injects" to "aliases". You can use the library with zero macro (well, just include guards) and zero "injections" if you like - that's <boost/openmethod/core.hpp>. And if you want the macros but not the alias, you can include <boost/openmethod/core.hpp> and <boost/openmethod/macros.hpp>. I will improve the doc on this... I am also not completely opposed to removing the injection of `virtual_ptr`.
The fact that as common a name as `next` is reserved inside open methods is also uncomfortable. Why not use a namespace-scoped name?
There are two `next`s. One is scoped in boost::openmethod::method (core API). The other is inside the body of a BOOST_OPENMETHOD_OVERRIDE (keyword API). It doesn't exist outside of it. I can rename it to something else. In CLOS, it is called call-next-method. In CLOS parlance, a method is a "generic" (defgeneric), and an overrider is a "method" (defmethod). `next` can become call_next_overrider, no biggie...
Next, the docs use the using-directive: `using namespace boost::openmethod;`. This is a bad idea: both for real programs and for the documentation: it is not clear if the library was designed with namespaces in mind.
Oh it was. And I thought a lot about `using namespace` in the doc. Reading documentation is not the same thing as writing a real-life large program. Long scoped names can get in the way while learning. The goal is different. I looked at the documentation of many Boost libraries. Some assume that their namespace is in scope. Others use the `namespace bom = boost::openmethod; ... bom::virtual_ptr` approach - which is just as "bad", right? But I am flexible on this...
Also, all the examples define open methods in the global namespace. I instead would like the docs to show how to declare open methods in my namespace, as this is a far more probable use case.
I should add an entry on namespaces to the tutorial. [6] Friendship
It puzzles me how the notion of friendship can be combined with this idea of flexibility in open methods. It is my understanding that open methods are added on top of the existing classes. That is: classes do not have to know that some open method will be added in the future. In other words, open methods are non-intrusive, right? So how would a class know who to grant the friendship to?
I used to agree with this. Targeted friendship seems so much in contradiction with openness. On the other hand, all features of C++ should play ball with one another. Friendship is part of the language. Also, I am not a nanny. I was often asked about friendship during the Q&A of my talks. I replied: You are probably doing something wrong if you need it, but I already have a design for it, and I will implement it when someone comes up with a good example where it makes sense. And then someone did. Discussion is here: https://github.com/jll63/yomm2/issues/7 Ponder this: polymorphism and encapsulation are orthogonal concepts, right? But traditional virtual functions have access to the private and protected parts. You just wanted polymorphism, but look!, you get access to private parts as well, like it or not. Well, with open-methods, you can have _just_ polymorphism. Better virtual functions, not toxic :)
Also, the examples use the declaration `friend struct`, even though we are talking about methods. I think docs should mention the existence of these structs more prominently.
Maybe. It is mentioned in two places (https://jll63.github.io/Boost.OpenMethod/#tutorials_friendship, https://jll63.github.io/Boost.OpenMethod/#BOOST_OPENMETHOD_OVERRIDE). I can see two reasons to need this info. You just want to grant friendship, in which case you can just follow the recipe; or you are doing more evolved things, like mixing core and keyword API and perhaps templates. In that case, you don't shy away from reading and decanting the reference, and I think all the info is there. But let's see what others say...
[7] Function signatures
Some macros in the library use function types in declarations: `BOOST_OPENMETHOD_OVERRIDERS(poke)<void(std::ostream&, virtual_ptr<Cat>)>`
Others specify the type separately: `BOOST_OPENMETHOD_OVERRIDE(value, (virtual_ptr<Plus> node), int)`
Is there a reason for this? Can the clever use of macros not offer the same function signature syntax?
These macros came out the way they are after trying many alternatives. CPP macros are surprisingly versatile, but they're not Lisp macros. I would like the syntax to be: BOOST_OPENMETHOD(poke(virtual_ptr<Animal>, std::ostream&), void); BOOST_OPENMETHOD_OVERRIDE(poke(virtual_ptr<Dog> dog, std::ostream& os), void) BOOST_OPENMETHOD_OVERRIDERS(poke(std::ostream&, virtual_ptr<Cat>, void) But I don't think it's feasible. However, this is probably possible: BOOST_OPENMETHOD_OVERRIDERS(poke, (std::ostream&, virtual_ptr<Cat>), void) Which mirrors nicely: BOOST_OPENMETHOD_OVERRIDE(poke, (std::ostream&, virtual_ptr<Cat>, void) Let me try this, it looks like a good idea.
[8] Passing virtual pointers to const objects
If `virtual_ptr<Cat>` is an analogue of this pointer, could we see an example in the docs that shows a virtual pointer to const objects? It would be `virtual_ptr<const Cat>`, I guess?
Yep! There, it looks like it's quite predictable, innit? :) The tutorial is ready and willing to evolve. I wanted to start light because I feel that, over the years, YOMM2's doc has become bloated, disorganized...
[9] Is this library intended for production use?
Intended? Yes, definitely. Ready is a better question. YOMM2 has existed for eight years. It has preserved backward compatibility throughout. It has evolved a lot. I got bug reports. I fixed them. I know that it has been used in production in more than one place. Now OpenMethod is not only YOMM2, renamed. It also contains "new research", so I cannot honestly say it is as field-tested as YOMM2; yet the new ideas address real problems. For example, moving the method's return type to the end of the parameter lists of BOOST_OPENMETHOD and BOOST_OPENMETHOD_OVERRIDE. It is not (only) to look "more modern". It actually solves a real annoyance with YOMM2: it does not cope well with return types that contain commas, so you have to use workarounds.
The library comes with a lot of clever macros, a lot of pointer type templates, a lot of new syntax and runtime surprises (mentioned earlier). I am concerned about how the debugging would look if something goes wrong with the usage of this library.
The inherent reduction in static checking has always been on my mind. I say "inherent" because in presence of dynamic loading (which YOMM11, YOMM2 and OpenMethod all support), what looks like a bug when you look at the main program vanishes when you (dynamically) load a shared object or a DLL. You mentioned plugins, you see what I mean. In debug builds, the library performs all sorts of runtime checks. If you forget to register a class, you will get an error message. If you turn on trace, you will get a very detailed explanation of what classes, methods, and overriders the library knows about, and how they relate. It will explain how it selects the best and next best overriders for each virtual tuple. It will tell you which slots in the v-tables are assigned to which methods. If you trace through a method call with a debugger - something I have done a lot -, with a little bit of practice, you will understand what is going on. Even if you make use of customization points. The code is organized to be step-in friendly. Try it :)
Would you recommend it for use in production systems? Or is it for experimentation?
In 2013 I started my work on open-(multi-)methods for C++ because: - I used multi-methods in previous projects in other languages. - I worked on a project that would have really benefited from open-methods (open-uni-methods in this instance). - I fantasized that I could help Stroustrup promote the idea by giving people a way to experiment with it. And then I would retire YOMM. However, YOMM11, and its successors, were never "toys". For example, they build redundancy free dispatch tables for multi-methods. That is not necessary for proof-of-concept, but it is for a production capable library. Recent releases of YOMM2 added customization points to address real-life problems. For example, in the game industry, they disable RTTI, and use custom systems instead. Because games are an important use-case, I make it possible to interface with custom RTTI. I came to recognize that we will have a colony on Mars before open-methods make it into the standard. So I posit OpenMethod as the next best option.
[10] Minor things
Section Custom RTTI has sentence:
Now we can include the "core" header
and then header `boost/openmethod/core.hpp` is not included. Is it a bug?
`boost/openmethod.hpp` includes `boost/openmethod/core.hpp` but you are right, I'll fix the doc, thanks :)