[metaparse] performance comparisons?
Abel, After looking at Metaparse, I inquired on the c++std-ext list as to whether user-defined operators can create char packs from integer or floating point literals, but not from string literals, as this would have made the MPLLIBS_STRING macro unnecessary. As it turns out, there has been such a proposal, N3599: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3599.html later referenced by two competing proposals for compile-time string literals, N4121 and N4236: https://isocpp.org/files/papers/n4121.pdf http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4236.html N3599 has been however rejected, as the committee apparently feels that representing compile-time strings as char packs is inefficient and impractical, and prefers constexpr char arrays. I cited Metaparse as an argument that char packs are quite obviously practical if they work in practice, and was asked whether measurement data has been presented as part of the formal review, perhaps compared to alternatives. I'm not aware of any alternatives to Metaparse though, so this question may be hard to answer. :-) Either way, the only performance data I see is http://abel.web.elte.hu/mpllibs/metaparse/performance.html which seems rather slim. Is there any other? (On an unrelated note, the cpp-next.com link is dead, but archive.org still has it at http://web.archive.org/web/20140217173026/http://cpp-next.com/archive/2012/1...).
Peter Dimov <lists <at> pdimov.com> writes:
[...]
As it turns out, there has been such a proposal, N3599:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3599.html
later referenced by two competing proposals for compile-time string literals, N4121 and N4236:
https://isocpp.org/files/papers/n4121.pdf http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4236.html
N3599 has been however rejected, as the committee apparently feels that representing compile-time strings as char packs is inefficient and impractical, and prefers constexpr char arrays.
`constexpr` char arrays and character packs are not equivalent. The problem is that constexpr-ness is stripped away by argument passing. For illustration, consider the following example: template <typename CompileTimeString> void f(CompileTimeString s) { static_assert(s == "abc", ""); } constexpr std::string_literal<n> n4121 = "abc"; constexpr auto n4236 = "abc"_s; f(n4121); // error: s is not constexpr within the body of f f(n4236); // ok: the value of s is contained in its type The only way to achieve this with N4121 is to unpack the `std::string_literal` into a template holding a character pack, passing this to `f` and then doing the comparison with a character pack. But then this is equivalent to N4236. Regards, Louis
Louis Dionne wrote:
`constexpr` char arrays and character packs are not equivalent. The problem is that constexpr-ness is stripped away by argument passing. For illustration, consider the following example:
template <typename CompileTimeString> void f(CompileTimeString s) { static_assert(s == "abc", ""); }
constexpr std::string_literal<n> n4121 = "abc"; constexpr auto n4236 = "abc"_s;
f(n4121); // error: s is not constexpr within the body of f f(n4236); // ok: the value of s is contained in its type
I assume that you mean for 'f' to be constexpr?
Peter Dimov <lists <at> pdimov.com> writes:
Louis Dionne wrote:
`constexpr` char arrays and character packs are not equivalent. The problem is that constexpr-ness is stripped away by argument passing. For illustration, consider the following example:
template <typename CompileTimeString> void f(CompileTimeString s) { static_assert(s == "abc", ""); }
constexpr std::string_literal<n> n4121 = "abc"; constexpr auto n4236 = "abc"_s;
f(n4121); // error: s is not constexpr within the body of f f(n4236); // ok: the value of s is contained in its type
I assume that you mean for 'f' to be constexpr?
No, it does not change a thing. It was on purpose that I did _not_ make it constexpr, in order not to mix concerns. What we're interested in is the constexpr-ness of `s` within the body of `f`, whether `f` is constexpr or not does not change anything. The above scenario is equivalent to something I presented in my C++Now talk [1]. The same concept is also explained in Hana's tutorial, in the "Advanced constexpr" section [2]. It seems that anyone trying to design something constexpr for the standard should have a solid grasp of those concepts, or we may very well end up missing the target of offering a useful compile-time string facility in C++17. Regards, Louis [1]: http://ldionne.com/hana-cppnow-2015/#/19/2 [2]: http://ldionne.com/hana/#tutorial-appendix-constexpr
Louis Dionne wrote:
I assume that you mean for 'f' to be constexpr?
No, it does not change a thing. It was on purpose that I did _not_ make it constexpr, in order not to mix concerns. What we're interested in is the constexpr-ness of `s` within the body of `f`, whether `f` is constexpr or not does not change anything.
I know that it doesn't, but we're demonstrating that 's' isn't constexpr even when 'f' is, which is a stronger claim than that 's' isn't constexpr when 'f' isn't. :-)
Peter Dimov <lists <at> pdimov.com> writes:
Louis Dionne wrote:
I assume that you mean for 'f' to be constexpr?
No, it does not change a thing. It was on purpose that I did _not_ make it constexpr, in order not to mix concerns. What we're interested in is the constexpr-ness of `s` within the body of `f`, whether `f` is constexpr or not does not change anything.
I know that it doesn't, but we're demonstrating that 's' isn't constexpr even when 'f' is, which is a stronger claim than that 's' isn't constexpr when 'f' isn't.
Sure, I agree with your point. However, just to clarify my view of things, I see f's constexpr-ness as being orthogonal (i.e. unrelated) to the constexpr-ness of its argument. Hence, I do not really perceive it as a stronger claim, just a different one that might make things trickier for someone not understanding this orthogonality. But I guess this boils down to pedagogy, and I think we have no real technical disagreement here. Regards, Louis
Hi Peter, On 2015-06-01 18:06, Peter Dimov wrote:
I cited Metaparse as an argument that char packs are quite obviously practical if they work in practice, and was asked whether measurement data has been presented as part of the formal review, perhaps compared to alternatives.
I'm not aware of any alternatives to Metaparse though, so this question may be hard to answer. :-)
Either way, the only performance data I see is
http://abel.web.elte.hu/mpllibs/metaparse/performance.html
which seems rather slim. Is there any other?
Measurements: - http://abel.sinkovics.hu/download.php?fn=dsltemp.pdf section 5.3 - https://github.com/boostcon/cppnow_presentations_2012/blob/master/tue/metapa... starting at slide 157: memory footprint + compilation time - http://abel.sinkovics.hu/download.php?fn=2014_dsl.pdf slide 268 ("Compile-time parsing" is about Metaparse)
(On an unrelated note, the cpp-next.com link is dead, but archive.org still has it at http://web.archive.org/web/20140217173026/http://cpp-next.com/archive/2012/1...).
Thank you for pointing this out. I've fixed the URL. Regards, Ábel
On 2015-06-01 18:06, Peter Dimov wrote:
Abel,
After looking at Metaparse, I inquired on the c++std-ext list as to whether user-defined operators can create char packs from integer or floating point literals, but not from string literals, as this would have made the MPLLIBS_STRING macro unnecessary.
As it turns out, there has been such a proposal, N3599:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3599.html
later referenced by two competing proposals for compile-time string literals, N4121 and N4236:
https://isocpp.org/files/papers/n4121.pdf http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4236.html
N3599 has been however rejected, as the committee apparently feels that representing compile-time strings as char packs is inefficient and impractical, and prefers constexpr char arrays. FWIW, sqlpp11 also uses char packs to represent and being able to compare strings at runtime. It is used like this (full example is attached, requires c++14, won't work with current MSVC-2015RC):
struct A { static constexpr const char _literal[] = "delta"; using name_t = sqlpp::make_char_sequence<sizeof(_literal), _literal>; }; static_assert( std::is_same<A::name_t, sqlpp::char_sequence<'d', 'e', 'l', 't', 'a', '\000'>>::value, ""); However, this is limited in the way that the type cannot be based on a literal outside a struct/class. There also is a macro that can be employed to create such types, e.g: SQLPP_ALIAS_PROVIDER(hello); static_assert( std::is_same<hello_t::_alias_t, sqlpp::char_sequence<'h', 'e', 'l', 'l', 'o', '\000'>>::value, ""); The MACRO cannot be called in a function, though, since local classes must not have static members. Thus, while I assume that it is faster than what happens inside the MPLLIBS_STRING (haven't measured it), its use is more limited, too. Best, Roland
07.06.2015 12:07, Roland Bock пишет:
However, this is limited in the way that the type cannot be based on a literal outside a struct/class. There also is a macro that can be employed to create such types, e.g:
SQLPP_ALIAS_PROVIDER(hello);
static_assert( std::is_same<hello_t::_alias_t, sqlpp::char_sequence<'h', 'e', 'l', 'l', 'o', '\000'>>::value, "");
The MACRO cannot be called in a function, though, since local classes must not have static members.
Following code works OK: int main() { struct specific_compiletime_string { static constexpr const char *value() { return "abc"; } }; static_assert(specific_compiletime_string::value()[0] == 'a', ""); // passes! } And you can get char sequence in following way: https://github.com/panaseleus/ctte/blob/master/proof_of_concept/proof_of_con...
Thus, while I assume that it is faster than what happens inside the MPLLIBS_STRING (haven't measured it), its use is more limited, too.
As I understand, complexity of MPLLIBS_STRING comes from fact that it can be passed immediately as template argument. -- Evgeny Panasyuk
On 2015-06-07 11:41, Evgeny Panasyuk wrote:
07.06.2015 12:07, Roland Bock пишет:
However, this is limited in the way that the type cannot be based on a literal outside a struct/class. There also is a macro that can be employed to create such types, e.g:
SQLPP_ALIAS_PROVIDER(hello);
static_assert( std::is_same<hello_t::_alias_t, sqlpp::char_sequence<'h', 'e', 'l', 'l', 'o', '\000'>>::value, "");
The MACRO cannot be called in a function, though, since local classes must not have static members.
Following code works OK:
int main() { struct specific_compiletime_string { static constexpr const char *value() { return "abc"; } }; static_assert(specific_compiletime_string::value()[0] == 'a', ""); // passes! }
And you can get char sequence in following way: https://github.com/panaseleus/ctte/blob/master/proof_of_concept/proof_of_con...
Very nice, indeed! Using a lambda, I can then do: #define MAKE_CHAR_SEQUENCE(name) \ []() \ { \ struct _intern \ { \ static constexpr const char* value() \ { \ return #name; \ } \ }; \ \ return make_string<_intern, sizeof(#name)>{}; \ }() int main() { auto x = MAKE_CHAR_SEQUENCE(delta); static_assert( std::is_same< decltype(x), sqlpp::char_sequence<'d', 'e', 'l', 't', 'a', '\000'>>::value, ""); } (code attached) Cool, I'll use that. It will allow users to define/use aliases in-place. But: Lamda expressions cannot live in unevaluated code. Thus using X = decltype(MAKE_CHAR_SEQUENCE(delta)); is illegal :-(
Thus, while I assume that it is faster than what happens inside the MPLLIBS_STRING (haven't measured it), its use is more limited, too.
As I understand, complexity of MPLLIBS_STRING comes from fact that it can be passed immediately as template argument.
I guess so, too. MAKE_CHAR_SEQUENCE can be used as a function argument at least :-)
07.06.2015 21:40, Roland Bock:
Cool, I'll use that. It will allow users to define/use aliases in-place.
But: Lamda expressions cannot live in unevaluated code. Thus
using X = decltype(MAKE_CHAR_SEQUENCE(delta));
is illegal :-(
Yes, this limitation was discussed a bit earlier - http://boost.2283326.n4.nabble.com/Boost-announce-metaparse-Review-period-st... (and below) I think that we should have several versions of CT string macros in Boost - because unfortunately there are different trade-offs for different approaches.
Thus, while I assume that it is faster than what happens inside the MPLLIBS_STRING (haven't measured it), its use is more limited, too.
As I understand, complexity of MPLLIBS_STRING comes from fact that it can be passed immediately as template argument.
I guess so, too.
MAKE_CHAR_SEQUENCE can be used as a function argument at least :-)
I used it following proof-of-concept - https://github.com/panaseleus/ctte And it was perfectly optimized by compiler (identical ASM code with handwritten version) Best Regards, Evgeny Panasyuk
Hi Evgeny, On 2015-06-07 11:41, Evgeny Panasyuk wrote:
Thus, while I assume that it is faster than what happens inside the MPLLIBS_STRING (haven't measured it), its use is more limited, too.
As I understand, complexity of MPLLIBS_STRING comes from fact that it can be passed immediately as template argument.
Exactly. The point is to be able to write: some_template<MPLLIBS_STRING(".....")> without having to mention the string (define it in some way) anywhere earlier in the code. Regards, Ábel
07.06.2015 12:07, Roland Bock:
However, this is limited in the way that the type cannot be based on a literal outside a struct/class. There also is a macro that can be employed to create such types, e.g:
SQLPP_ALIAS_PROVIDER(hello);
By the way, I saw sqlpp11 earlier, and I think that alias/field provider is a useful meta-programming tool on it's own. https://github.com/rbock/sqlpp11/blob/master/include/sqlpp11/alias_provider.... For example it can be used in transformation of vector of structs into structs of vectors - such macro would provide names for fields in synthesized struct of references. Perhaps it could be embedded by default into BOOST_FUSION_DEFINE_STRUCT and others. Also I think following template intrinsic can be useful addition to C++ ISO: template<char...> struct names_provider; Which would be automatically specialized by compiler on usage, for example something like: template<> struct names_provider<'f', 'o', 'o'> { template<typename T> struct field { T foo; }; template<typename F> struct method { template<typename ...Ts> auto foo(Ts... args) { return F{}(args...); } }; // ... }; With help of such intrinsic it is possible to parse string with some EDSL at compile time, and based only on this generate all required structures defined by language embedded in string. Best Regards, Evgeny Panasyuk
On 2015-06-07 12:40, Evgeny Panasyuk wrote:
07.06.2015 12:07, Roland Bock:
However, this is limited in the way that the type cannot be based on a literal outside a struct/class. There also is a macro that can be employed to create such types, e.g:
SQLPP_ALIAS_PROVIDER(hello);
By the way, I saw sqlpp11 earlier, and I think that alias/field provider is a useful meta-programming tool on it's own. https://github.com/rbock/sqlpp11/blob/master/include/sqlpp11/alias_provider....
For example it can be used in transformation of vector of structs into structs of vectors - such macro would provide names for fields in synthesized struct of references.
Perhaps it could be embedded by default into BOOST_FUSION_DEFINE_STRUCT and others.
Also I think following template intrinsic can be useful addition to C++ ISO:
template<char...> struct names_provider;
Which would be automatically specialized by compiler on usage, for example something like:
template<> struct names_provider<'f', 'o', 'o'> { template<typename T> struct field { T foo; }; template<typename F> struct method { template<typename ...Ts> auto foo(Ts... args) { return F{}(args...); } }; // ... };
Right, I use this in sqlpp11 and sqlpp11-connector-stl (an experimental SQL interface to vectors or other containers).
With help of such intrinsic it is possible to parse string with some EDSL at compile time, and based only on this generate all required structures defined by language embedded in string.
It is still a PITA with the inheritance required to turn such things into members of a struct. I presented an idea of how to do this with much less pain here: https://groups.google.com/a/isocpp.org/forum/#!msg/std-proposals/hYh3hWB0mwg... Best, Roland
07.06.2015 18:51, Roland Bock:
On 2015-06-07 12:40, Evgeny Panasyuk wrote:
07.06.2015 12:07, Roland Bock:
However, this is limited in the way that the type cannot be based on a literal outside a struct/class. There also is a macro that can be employed to create such types, e.g:
SQLPP_ALIAS_PROVIDER(hello);
By the way, I saw sqlpp11 earlier, and I think that alias/field provider is a useful meta-programming tool on it's own. https://github.com/rbock/sqlpp11/blob/master/include/sqlpp11/alias_provider....
For example it can be used in transformation of vector of structs into structs of vectors - such macro would provide names for fields in synthesized struct of references.
Perhaps it could be embedded by default into BOOST_FUSION_DEFINE_STRUCT and others.
Also I think following template intrinsic can be useful addition to C++ ISO:
template<char...> struct names_provider;
Which would be automatically specialized by compiler on usage, for example something like:
template<> struct names_provider<'f', 'o', 'o'> { template<typename T> struct field { T foo; }; template<typename F> struct method { template<typename ...Ts> auto foo(Ts... args) { return F{}(args...); } }; // ... };
Right, I use this in sqlpp11 and sqlpp11-connector-stl (an experimental SQL interface to vectors or other containers).
sqlpp11 uses a bit different thing - it generates special name_t struct which has inside "field" provider and name stored as compile-time string. Here I am talking about mapping from compile-time string to "field" provider.
With help of such intrinsic it is possible to parse string with some EDSL at compile time, and based only on this generate all required structures defined by language embedded in string. It is still a PITA with the inheritance required to turn such things into members of a struct.
I presented an idea of how to do this with much less pain here:
https://groups.google.com/a/isocpp.org/forum/#!msg/std-proposals/hYh3hWB0mwg...
As I understand you proposed here name/identifier literal, which of course is useful thing, but here specifically I am talking about mapping from compile-time string to field provider. That would enable following use-case: generate_struct_t<CT_STRING(R"( struct { 2: i32 id, 1: string name } )")> value; value.id = 11; value.name = "abc"; Metafunction generate_struct_t accepts EDSL compile-time string, parses it, extracts compile-time substrings "id" and "name", uses names_provider<'i', 'd'> and names_provider<'n','a','m','e'> to get field providers, gathers them to one struct and returns it's type. Best Regards, Evgeny Panasyuk
On 2015-06-07 11:07, Roland Bock wrote:
On 2015-06-01 18:06, Peter Dimov wrote:
Abel,
After looking at Metaparse, I inquired on the c++std-ext list as to whether user-defined operators can create char packs from integer or floating point literals, but not from string literals, as this would have made the MPLLIBS_STRING macro unnecessary.
As it turns out, there has been such a proposal, N3599:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3599.html
later referenced by two competing proposals for compile-time string literals, N4121 and N4236:
https://isocpp.org/files/papers/n4121.pdf http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4236.html
N3599 has been however rejected, as the committee apparently feels that representing compile-time strings as char packs is inefficient and impractical, and prefers constexpr char arrays. FWIW, sqlpp11 also uses char packs to represent and being able to compare strings at runtime. It is used like this (full example is attached, requires c++14, won't work with current MSVC-2015RC):
struct A { static constexpr const char _literal[] = "delta"; using name_t = sqlpp::make_char_sequence<sizeof(_literal), _literal>; };
static_assert( std::is_same<A::name_t, sqlpp::char_sequence<'d', 'e', 'l', 't', 'a', '\000'>>::value, "");
However, this is limited in the way that the type cannot be based on a literal outside a struct/class. There also is a macro that can be employed to create such types, e.g:
SQLPP_ALIAS_PROVIDER(hello);
static_assert( std::is_same<hello_t::_alias_t, sqlpp::char_sequence<'h', 'e', 'l', 'l', 'o', '\000'>>::value, "");
The MACRO cannot be called in a function, though, since local classes must not have static members.
Thus, while I assume that it is faster than what happens inside the MPLLIBS_STRING (haven't measured it), its use is more limited, too.
Best,
Roland Sorry, I experimented with the file before sending it and corrupted it. The attached version actually compiles...
Hi Roland, On 2015-06-07 11:07, Roland Bock wrote:
FWIW, sqlpp11 also uses char packs to represent and being able to compare strings at runtime. It is used like this (full example is attached, requires c++14, won't work with current MSVC-2015RC):
struct A { static constexpr const char _literal[] = "delta"; using name_t = sqlpp::make_char_sequence<sizeof(_literal), _literal>; };
It looks like you managed to use a char array as a template argument. I don't know how sqlpp uses these strings, however, if the same could be achieved in Metaparse, I'd definitely try carrying the char array reference/pointer around and write "smart" getter functions (front, pop_front, equal_to etc) around it and check if it really makes things faster and lower the memory consumption - I'd expect it to do so.
Thus, while I assume that it is faster than what happens inside the MPLLIBS_STRING (haven't measured it), its use is more limited, too. The main limit here (from Metaparse's perspective) is that you need to define the string (separately) before you can use it.
Regards, Ábel
Hi Abel, On 2015-06-07 20:41, Abel Sinkovics wrote:
Hi Roland,
On 2015-06-07 11:07, Roland Bock wrote:
FWIW, sqlpp11 also uses char packs to represent and being able to compare strings at runtime. It is used like this (full example is attached, requires c++14, won't work with current MSVC-2015RC):
struct A { static constexpr const char _literal[] = "delta"; using name_t = sqlpp::make_char_sequence<sizeof(_literal), _literal>; };
It looks like you managed to use a char array as a template argument. I don't know how sqlpp uses these strings, however, if the same could be achieved in Metaparse, I'd definitely try carrying the char array reference/pointer around and write "smart" getter functions (front, pop_front, equal_to etc) around it and check if it really makes things faster and lower the memory consumption - I'd expect it to do so. In sqlpp11 the char_sequence contains a static function that returns a const char*. Storing a reference/pointer to the "original" array does not seem to work in all cases (after being handed through some template aliases, they seem to break).
The char_sequence is used to compare names of objects representing columns, tables and the like. Instead of comparing actual strings at compile time, the library can compare types. Also, the char_sequence is used in serializing expressions (this is where that static function comes into play).
Thus, while I assume that it is faster than what happens inside the MPLLIBS_STRING (haven't measured it), its use is more limited, too. The main limit here (from Metaparse's perspective) is that you need to define the string (separately) before you can use it.
Right. With the input from Evgeny I was able to change that to the extent, that I can now use them in-place as function arguments (but not as template parameters or type declarations like yours). Hmm. This might lead to an alternative syntax, btw. Instead of using X = typename exp_parser3::apply<MPLLIBS_STRING("11 ")>::type; we could call auto x = exp_parser3::apply(MPLLIBS_STRING("11 ")); Not sure if and how this could be useful in the context of your library. Best, Roland
Hi Roland, On 2015-06-07 21:30, Roland Bock wrote:
In sqlpp11 the char_sequence contains a static function that returns a const char*. Storing a reference/pointer to the "original" array does not seem to work in all cases (after being handed through some template aliases, they seem to break).
The char_sequence is used to compare names of objects representing columns, tables and the like. Instead of comparing actual strings at compile time, the library can compare types.
Also, the char_sequence is used in serializing expressions (this is where that static function comes into play).
So sqlpp's use case is different from Metaparse's. (comparisons instead of interpreting the content).
Right. With the input from Evgeny I was able to change that to the extent, that I can now use them in-place as function arguments (but not as template parameters or type declarations like yours).
Hmm. This might lead to an alternative syntax, btw. Instead of
using X = typename exp_parser3::apply<MPLLIBS_STRING("11 ")>::type;
we could call
auto x = exp_parser3::apply(MPLLIBS_STRING("11 "));
Not sure if and how this could be useful in the context of your library.
I expect to end up with a similar interface to this alternate one when trying to build a Metaparse-like library around Hana. Using parsers built with Metaparse the result of parsing can be used as a template argument. For example: some_template<exp_parser3::apply<MPLLIBS_STRING("11 ")>::type> in which case we have the unevaluated context problem again, as it would be some_template<decltype(exp_parser3::apply(MPLLIBS_STRING("11 "))> with the alternate syntax. Regards, Ábel
On 2015-06-07 21:43, Abel Sinkovics wrote:
using X = typename exp_parser3::apply<MPLLIBS_STRING("11 ")>::type;
we could call
auto x = exp_parser3::apply(MPLLIBS_STRING("11 "));
Not sure if and how this could be useful in the context of your library. I expect to end up with a similar interface to this alternate one when trying to build a Metaparse-like library around Hana. Cool, looking forward to it :-)
participants (5)
-
Abel Sinkovics -
Evgeny Panasyuk -
Louis Dionne -
Peter Dimov -
Roland Bock