Reminder: Describe needs a review manager
This is a reminder that my library Describe https://github.com/pdimov/describe https://pdimov.github.io/describe/doc/html/describe.html needs a review manager. Describe is a library that allows programmers to annotate their classes (and enumerations) in order to enable simple reflection. It provides a very important fundament on which other libraries can build upon to provide automatic support for user-defined types. For example, given an annotated type, Boost.JSON can automatically provide conversions from/to json::value. Boost.ContainerHash can automatically provide a hash function hash_value for the type. Boost.Serialization can automatically provide serialization support. See the examples: https://pdimov.github.io/describe/doc/html/describe.html#example_to_json https://pdimov.github.io/describe/doc/html/describe.html#example_from_json https://pdimov.github.io/describe/doc/html/describe.html#example_serializati... Other JSON (and serialization) libraries already offer a similar mechanism, e.g. https://github.com/nlohmann/json#arbitrary-types-conversions. But since each library has its own private way to annotate types, a programmer needs to repeat the type annotations for every library he wants his type to be known to. This is unnecessary and undesirable, and a standard mechanism to describe types avoids it. Describe is intended to be this standard mechanism, and I believe that we need it in Boost sooner rather than later. Thanks in advance to whoever volunteers.
Em ter., 23 de fev. de 2021 às 15:56, Peter Dimov via Boost <boost@lists.boost.org> escreveu:
Other JSON (and serialization) libraries already offer a similar mechanism, e.g. https://github.com/nlohmann/json#arbitrary-types-conversions. But since each library has its own private way to annotate types, a programmer needs to repeat the type annotations for every library he wants his type to be known to. This is unnecessary and undesirable, and a standard mechanism to describe types avoids it.
I agree with the diagnosis, but I'm not so keen on the solution. I insist you broaden your view and take a look at other libraries that need manual annotation. If your solution only covers the JSON case, it could very well be just an abstraction inside the JSON library and isn't a vocabulary at all. The documentation lacks a section comparing your solution with what's available on the market today. I'll give you a starter pack on what's to look for, but you shouldn't limit yourself to these links: * https://doc.qt.io/qt-5/properties.html * https://doc.qt.io/qt-5/animation-overview.html * https://gi.readthedocs.io/en/latest/ * https://archive.fosdem.org/2015/schedule/event/reinventing_the_enlightenment... Except for EFL which is an unfortunately ignored project, all libraries listed are widely used. All of them have been in the market for quite some time and are battle-tested. Do notice that each project pursues manual annotation for different reasons and none of them are JSON related. In one project, the property system is the core of an animation system. In such a place, it's not enough to have fragmented symbols that have no connection with one another at all. You need to glue properties to the real getters and setters and that's what must be used as changing one property must trigger signals to other parts of the system. In fact, if you don't provide at least this, it's hard for me to see which value I'd get from Describe that I can't already get from PFR or Boost.Hana Structs. In the other two systems linked, properties are used to extract the "OO/class layout" and export them for use in language bindings without the need for a C/C++ parser in the other languages. JSON serialization is just a small part of the problem that this library tries to tackle. The research might be daunting, but it always pays off. Research is what separates a "cool hack" from "serious solutions". I'm actually looking forward to using this library myself. Furthermore, I won't have time to participate in Describe review. So I thought I could share this info with you now and trust you'll just do the right thing here. Have fun. -- Vinícius dos Santos Oliveira https://vinipsmaker.github.io/
On 24/02/2021 12:55, Vinícius dos Santos Oliveira via Boost wrote:
The research might be daunting, but it always pays off. Research is what separates a "cool hack" from "serious solutions". I'm actually looking forward to using this library myself.
I'm very sure Peter did his prior art research. For me personally, I'd like a C++ 23 Reflection emulation library which emulates as much of future Reflection as is possible today, and would if on C++ 23 switch to using proper Reflection. No idea if such a thing is possible however. Niall
Niall Douglas wrote:
For me personally, I'd like a C++ 23 Reflection emulation library which emulates as much of future Reflection as is possible today, and would if on C++ 23 switch to using proper Reflection. No idea if such a thing is possible however.
It's not. Or rather, it's possible, but of limited utility. Describe produces the reflection metadata in a form with which we can do something in today's C++. The C++23/26 reflection produces a constexpr vector<meta::info> which can be manipulated via C++20 consteval functions, but to actually do something useful with the result, you need injection, which is also part of the same 23/26 package.
Vinícius dos Santos Oliveira wrote:
Em ter., 23 de fev. de 2021 às 15:56, Peter Dimov via Boost <boost@lists.boost.org> escreveu:
Other JSON (and serialization) libraries already offer a similar mechanism, e.g. https://github.com/nlohmann/json#arbitrary-types-conversions. But since each library has its own private way to annotate types, a programmer needs to repeat the type annotations for every library he wants his type to be known to. This is unnecessary and undesirable, and a standard mechanism to describe types avoids it.
I agree with the diagnosis, but I'm not so keen on the solution. I insist you broaden your view and take a look at other libraries that need manual annotation. If your solution only covers the JSON case, it could very well be just an abstraction inside the JSON library and isn't a vocabulary at all.
Well, there's nothing JSON-specific in the library. It provides basic reflection metadata: base classes, data members, member functions. I use JSON, hashing, and serialization as examples of libraries already in Boost that can take advantage of this information.
The documentation lacks a section comparing your solution with what's available on the market today. I'll give you a starter pack on what's to look for, but you shouldn't limit yourself to these links:
* https://doc.qt.io/qt-5/properties.html * https://doc.qt.io/qt-5/animation-overview.html * https://gi.readthedocs.io/en/latest/ * https://archive.fosdem.org/2015/schedule/event/reinventing_the_enlighten ment_object_system/
Thanks, I'll look at these.
On Tue, 23 Feb 2021 at 19:56, Peter Dimov via Boost <boost@lists.boost.org> wrote:
This is a reminder that my library Describe
https://github.com/pdimov/describe https://pdimov.github.io/describe/doc/html/describe.html
needs a review manager.
Describe is a library that allows programmers to annotate their classes (and enumerations) in order to enable simple reflection. It provides a very important fundament on which other libraries can build upon to provide automatic support for user-defined types.
For example, given an annotated type, Boost.JSON can automatically provide conversions from/to json::value. Boost.ContainerHash can automatically provide a hash function hash_value for the type. Boost.Serialization can automatically provide serialization support. See the examples:
https://pdimov.github.io/describe/doc/html/describe.html#example_to_json https://pdimov.github.io/describe/doc/html/describe.html#example_from_json
https://pdimov.github.io/describe/doc/html/describe.html#example_serializati...
Other JSON (and serialization) libraries already offer a similar mechanism, e.g. https://github.com/nlohmann/json#arbitrary-types-conversions. But since each library has its own private way to annotate types, a programmer needs to repeat the type annotations for every library he wants his type to be known to. This is unnecessary and undesirable, and a standard mechanism to describe types avoids it.
Describe is intended to be this standard mechanism, and I believe that we need it in Boost sooner rather than later.
I can identify a number of cases where this library would be useful for me. The easy implementation of serialisation being a major one. I'd be happy to propose myself to manage the review.
Thanks in advance to whoever volunteers.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
-- Richard Hodges hodges.r@gmail.com office: +442032898513 home: +376841522 mobile: +376380212
On 2/24/21 8:10 AM, Richard Hodges via Boost wrote:
On Tue, 23 Feb 2021 at 19:56, Peter Dimov via Boost <boost@lists.boost.org>
I can identify a number of cases where this library would be useful for me.
The easy implementation of serialisation being a major one.
I understand the appeal of this idea. But, save trivial cases, I'm doubtful that reflection will make serialization easier than it already is.
Robert Ramey wrote:
On 2/24/21 8:10 AM, Richard Hodges via Boost wrote:
I can identify a number of cases where this library would be useful for me.
The easy implementation of serialisation being a major one.
I understand the appeal of this idea. But, save trivial cases, I'm doubtful that reflection will make serialization easier than it already is.
It will in some cases and won't in others. There are scenarios where f.ex. your applications need to talk to each other over the wire, but the actual format is transient and doesn't really matter, because you're deploying both endpoints (in sync.) In this case, you don't need to care about versioning. You may still need to override the default de/serializer by hand for some types, but every one of them for which you don't is still a win. There's also a scenario where the format is predetermined, and you create your C++ structs to conform to it, rather than vice versa. (This of course assumes that you have an archive supporting the format.)
On Thu, Feb 25, 2021 at 4:16 AM Peter Dimov via Boost <boost@lists.boost.org> wrote:
There's also a scenario where the format is predetermined, and you create your C++ structs to conform to it, rather than vice versa.
Hi Peter. That's exactly my case (just not with Boost.Serialization). The real question is then, since those C++ struct sources are code-generated, is it worth to invest in Boost.Describe once in the code generator, and then use the techniques of your examples (i.e. "compile-time" code-gen), or update the source generator instead (i.e. "ahead-of-time" source-gen). In my case, the (de)serialization code is already generated, but those struct lack op== for round-trip unit-testing for example. So is it worth to invest once in describe? And use it for both binary serialization and op==, and perhaps later for JSON serialization, mock generation (no example of that?), etc... Or simply update the source generator? Depends on complexity of use, compile-time speed, runtime overhead (if any) compared to "hand-written code" (well, source-generated), these kind of things. Boost.Describe caught my eye during the BoostJSON review, and I looked at it then. But I haven't taken the time to investigate further. My structs notably use optionals and variants, of the non-standard very ad-hoc kind, so I wasn't sure how those would fair. But overall, I'm very interested in what Boost.Describe promises. --DD
Dominique Devienne wrote:
On Thu, Feb 25, 2021 at 4:16 AM Peter Dimov via Boost <boost@lists.boost.org> wrote:
There's also a scenario where the format is predetermined, and you create your C++ structs to conform to it, rather than vice versa.
Hi Peter. That's exactly my case (just not with Boost.Serialization).
The real question is then, since those C++ struct sources are code-generated, is it worth to invest in Boost.Describe once in the code generator, and then use the techniques of your examples (i.e. "compile-time" code-gen), or update the source generator instead (i.e. "ahead-of-time" source-gen).
It's hard to say; one would have to experiment with both and see as it depends on non-technical factors. Adding Describe support to the generated structs is fairly cheap so I'd do that either way, as it enables more flexibility on the C++ side without having to touch the generator for every change.
On 2/24/21 7:16 PM, Peter Dimov via Boost wrote:
There's also a scenario where the format is predetermined, and you create your C++ structs to conform to it, rather than vice versa. (This of course assumes that you have an archive supporting the format.)
I've considered this from time to time. Basically this is the approach that google uses. boost serialization C++ data structures => serialization lay out features of this system no format specification required no extra code mapping the the C++ data structures to the externally defined format very easy maintenance. archives not easily read/written by other languages google protocol buffers format specified externally. This can be significant work and maintenance. code required to map to format. archives designed to be read/written by other languages Both offer pretty good performance to the point that I suspect that few will find either a performance bottle neck. Soooo - how about writing an archive format that either uses or generates an external specification and perhaps even code to make the achives read/write able by other languages (javascript anyone). It would be quite a pain and no one has asked for it. So I'm content to just suggest that those who need an external format just use google protocol buffers which is quite popular in any case. Lately, I've been interested in upgrading the boost serialization documentation to be more like the safe numerics one. It's already pretty good, but would benefit by a little more formality, lots more case studies and examples. It's amazing to me that the serialization library has lasted 17 years. I have no idea how many people (if any still use it). The codebase is more healthy than ever having incorporated a huge number of small improvements over the years. It doesn't take much of my time to keep it up to date. Most of that time is due to weird extensions to C++ which add confusion and break previously undefined stuff. Here is some information comparing serialization with google protocol buffers. https://stackoverflow.com/questions/1061169/boost-serialization-vs-google-pr... To the main point, I can see Describe working with serialization in a reasonable manner. I not yet convinced that the benefit will be worth the cost. But since it's entirely optional and doesn't require the serialization code to be altered, I don't see any problem with it. Robert Ramey
participants (7)
-
Dominique Devienne -
Niall Douglas -
pdimov＠gmail.com -
Peter Dimov
-
Richard Hodges -
Robert Ramey -
Vinícius dos Santos Oliveira