On 11/21/2019 4:51 PM, Robert Ramey via Boost-users wrote:

On 11/21/19 10:49 AM, Edward Diener via Boost-users wrote:

...

On 11/21/2019 10:23 AM, Krzysztof Jusiak via Boost-users wrote:

...

Hi,

I was wondering whether there is any interest in exploring a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

Github: https://github.com/boost-experimental/ut

Additional links: - Try it online: https://godbolt.org/z/uVDxkW - Benchmarks: https://github.com/boost-experimental/ut#benchmarks - How it works?: https://github.com/boost-experimental/ut#how-it-works

Actual documentation, rather than a huge bunch of examples, is always welcome for this programmer. Otherwise I have no idea what your code is about or how to use it.

Hmmm - what constitutes actual documentation in your mind as opposed to a "huge bunch of examples".  Sounds like you consider the examples unnecessary.

I did not find a single line of explanation in the docs of ut. Examples are fine but if the entire documentation consists of examples I do not think that is nearly enough to understand a library and what it is about. I was hoping my comment would get the developer of ut to consider docs that explain what his library is about, how to use it, and of what the individual constructs of the library consists. But that is solely up to the developer and if others find the docs adequate I wish them well in using it.

On Fri, Nov 22, 2019 at 9:18 AM Richard Hodges via Boost-users < boost-users@lists.boost.org> wrote:

It looks very interesting. The idea of not using macros is very appealing to me.

Indeed. What surprised me, given the announcement on the Boost ML, was the lack of comparison to Boost's own Boost.Test library (and its other single-header lightweight one, which AFAIK is not an official library, please correct me if I'm wrong). --DD

Thanks for your valuable feedback. I added initial versions of tutorial and user-guide to README. It's not ideal yet but it's a start. * https://github.com/boost-experimental/ut#tutorial * https://github.com/boost-experimental/ut#user-guide I also updated benchmarks with Boost.Test-1.71.0 (static library). I didn't add Boost.LightweightTest though as it's just assertions (no tests/suites/etc...) so it makes it difficult to compare due to missing features. * https://github.com/boost-experimental/ut#benchmarks I hope that helps a bit. Thanks again, Kris On Fri, Nov 22, 2019 at 1:17 AM Richard Hodges via Boost-users < boost-users@lists.boost.org> wrote:

It looks very interesting. The idea of not using macros is very appealing to me.

From looking at the examples, I get the feeling of "I could probably use this".

However, as the other commenters have mentioned, without a tutorial and full reference it's difficult to judge whether it's ready for use in production.

Are there plans to develop this?

On Thu, 21 Nov 2019 at 16:24, Krzysztof Jusiak via Boost-users < boost-users@lists.boost.org> wrote:

...

Hi,

I was wondering whether there is any interest in exploring a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

Github: https://github.com/boost-experimental/ut

Additional links: - Try it online: https://godbolt.org/z/uVDxkW - Benchmarks: https://github.com/boost-experimental/ut#benchmarks - How it works?: https://github.com/boost-experimental/ut#how-it-works

Thanks, Kris _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users

-- Richard Hodges hodges.r@gmail.com office: +442032898513 home: +376841522 mobile: +376380212

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users

I see your point and I'm not going to argue it. I think it's a valid one, however, I see it a bit differently because test/suites registration is not automatic in case of Boost.LT and or C++, which IMHO, is one of the main functionalities of a testing framework (it allows not to write the boilerplate code, filter, print run tests, report them, etc. which is not provided by Boost.LT out of the box). In principle, the baseline for benchmarks could be raw functions and asserts or to compare apples to apples one could take expect from ut, BOOST_TEST from Boost.Test, CHECK from Catch2, etc. and do the manual registration as you presented, which would be fine, but not ideal, IMHO. Hence, I took the other approach and only compare frameworks that provide at least assertions/tests and suites in an automated way. All in all, I just didn't see much value in comparing Boost.LT (not an official boost library) because, in my view, it doesn't provide the same functionality as other tested frameworks but I also see your point, is just not what I wanted to focus on in the provided benchmarks. On Mon, Nov 25, 2019 at 12:12 PM Mateusz Loskot via Boost-users < boost-users@lists.boost.org> wrote:

(FYI, https://www.boost.org/community/policy.html says: Don't Overquote, Don't Top-Post,...)

On Mon, 25 Nov 2019 at 19:10, Krzysztof Jusiak via Boost-users wrote:

...

I also updated benchmarks with Boost.Test-1.71.0 (static library). I didn't add Boost.LightweightTest though as it's just assertions (no

tests/suites/etc...) so it makes it difficult to compare due to missing features.

...

* https://github.com/boost-experimental/ut#benchmarks

Boost.LT does not offer formal vocabulary for cases and suites, but saying it's "just assertions" is not quite accurate, I think.

According to some of TDD advocates, there shall be single assertion per unit test [1] (test case is an individual unit of testing, a unit test):

int sqr(int x) { return x * x; } int main() { BOOST_TEST( sqr(2) == 4 ); // test case 1 BOOST_TEST_EQ( sqr(-3), 9 ); // test case 2 return boost::report_errors(); }

For those who multiple assertions:

void test_sqr(int input, int expect) { BOOST_TEST( sqr(input) == expect); BOOST_TEST( sqr(-input) == expect); } int main() { test_sqrt(2, 4); // test case 1 test_sqrt(-3, 9); // test case 1 return boost::report_errors(); }

Finally, the single runnable test program is a test suite.

So, I will argue that Boost.LT does support test cases and test suites.

[1] https://osherove.com/blog/2006/10/3/avoid-multiple-asserts-in-a-single-unit-...

Best regards, -- Mateusz Loskot, http://mateusz.loskot.net _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users

Ok – with more docs it now looks convincing. I’ll try it on my next standalone project. It won’t replace Boost.Test – we have much too much invested in it (and we support much older compilers), but that’s no reason why it couldn’t be a Boost library. To get through a Boost review, you need some enthusiastic users. Paul Paul A. Bristow Prizet Farmhouse Kendal, Cumbria LA8 8AB UK From: Boost-users On Behalf Of Krzysztof Jusiak via Boost-users Sent: 25 November 2019 18:10 To: boost-users@lists.boost.org; boost@lists.boost.org List Cc: Krzysztof Jusiak Subject: Re: [Boost-users] Interest in a C++20 Unit Testing Framework? Thanks for your valuable feedback. I added initial versions of tutorial and user-guide to README. It's not ideal yet but it's a start. * https://github.com/boost-experimental/ut#tutorial * https://github.com/boost-experimental/ut#user-guide I also updated benchmarks with Boost.Test-1.71.0 (static library). I didn't add Boost.LightweightTest though as it's just assertions (no tests/suites/etc...) so it makes it difficult to compare due to missing features. * https://github.com/boost-experimental/ut#benchmarks I hope that helps a bit. Thanks again, Kris On Fri, Nov 22, 2019 at 1:17 AM Richard Hodges via Boost-users mailto:boost-users@lists.boost.org > wrote: It looks very interesting. The idea of not using macros is very appealing to me.

From looking at the examples, I get the feeling of "I could probably use this".

However, as the other commenters have mentioned, without a tutorial and full reference it's difficult to judge whether it's ready for use in production. Are there plans to develop this? On Thu, 21 Nov 2019 at 16:24, Krzysztof Jusiak via Boost-users mailto:boost-users@lists.boost.org > wrote: Hi, I was wondering whether there is any interest in exploring a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies? Github: https://github.com/boost-experimental/ut Additional links: - Try it online: https://godbolt.org/z/uVDxkW - Benchmarks: https://github.com/boost-experimental/ut#benchmarks - How it works?: https://github.com/boost-experimental/ut#how-it-works Thanks, Kris _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org mailto:Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users -- Richard Hodges hodges.r@gmail.com mailto:hodges.r@gmail.com office: +442032898513 home: +376841522 mobile: +376380212 _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org mailto:Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users

Thank you for your feedback. I appreciate it. Well, IMHO, there are pros and cons to any solution; I'm not going to start a flame war here about whether macros are evil or not. However, with [Boost].UT there is nothing stopping anyone from using simple macros (one-lines) to achieve other frameworks syntax The good bit about it is that it's an opt-in 'feature' as opposed to being the only available option (see example below [1]). This way more users can be satisfied and other benefits such as faster execution, quicker compilation times and other features are still available. ``` #define EXPECT(...) ::boost::ut::expect(::boost::ut::that % __VA_ARGS__) #define SUITE ::boost::ut::suite _ = [] #define TEST(name) ::boost::ut::detail::test{"test", name} = [=]() mutable SUITE { TEST("suite") { EXPECT(42 == 42); }; }; int main() { TEST("macro") { EXPECT(1 != 2); }; TEST("vector") { std::vector<int> v(5); EXPECT(5u == std::size(v)) << "fatal"; TEST("resize bigger") { v.resize(10); EXPECT(10u == std::size(v)); }; }; } ``` [1]: https://godbolt.org/z/tvy-nP -Kris On Fri, Nov 22, 2019 at 2:27 PM Mateusz Loskot via Boost-users < boost-users@lists.boost.org> wrote:

(CC'ed to boost-users)

On Fri, 22 Nov 2019 at 21:53, Vinnie Falco via Boost wrote:

...

On Thu, Nov 21, 2019 at 7:24 AM Krzysztof Jusiak via Boost-users < boost-users@lists.boost.org>>

...

I was wondering whether there is any interest in exploring a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

I have no interest in a library that requires C++20, especially considering that C++20 is not even official yet but also because once C++20 is released, there will be hardly any users for many years. This project seems very much like it was written "just because", to use the latest language features, rather than for pragmatic reasons. I don't see anything compelling to use it over Boost.LightweightTest for example.

I had a look at the current documentation and examples, and I'm glad to see I'm not the only sceptical about it. My opinion is very similar to what Hans, Raffi and Vinnie explained.

Macros in testing and benchmarking frameworks are kosher, in fact, they can be very helpul to organize and structure large amount of tests that in turn makes it easier to search and browse through the code. I can't imagine how any of the modern IDE's will support the string-driven syntax.

As a long time maintainer of SOCI library, I'm no stranger to the syntax-first library development [1], still, I find the UT's string-based test cases approach as an interesting curiosity with very little practical value. (It falls into similar drawer as the (over)use of emoji in commit messages on GitHub :)).

Boost is known as a C++ guinea pigs playground, so any library can make it in. If it does, I just hope Boost will not allow to adopt the UT as a test framework of choice of any/too many of its libraries.

[1] https://www.drdobbs.com/database/a-simple-oracle-call-interface/184405930?pg...

Best regards, -- Mateusz Loskot, http://mateusz.loskot.net _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users

Le 22/11/2019 à 22:28, Mateusz Loskot via Boost-users a écrit :

Macros in testing and benchmarking frameworks are kosher, in fact, they can be very helpul to organize and structure large amount of tests that in turn makes it easier to search and browse through the code. I can't imagine how any of the modern IDE's will support the string-driven syntax.

Not sure to understand how macros help organizing code but well.

As a long time maintainer of SOCI library, I'm no stranger to the syntax-first library development [1], still, I find the UT's string-based test cases approach as an interesting curiosity with very little practical value. (It falls into similar drawer as the (over)use of emoji in commit messages on GitHub :)).

Catch2 and some Javascript frameworks use strings based tests too. I don't see any issues with that. Regarding emojis, this is clearly a mainstream hype unfortunately [0] [1]

I just hope Boost will not allow to adopt the UT as a test framework of choice of any/too many of its libraries.

I'd like to know what are the benefits of this kind of comments that are very subjective. But it's not the first time I see this on this mailing list. Refrain yourself next time, not liking a library is opinion based but trying to demotivate people is irrelevant and not productive. [0]: https://github.com/nlohmann/json/commits/develop [1]: https://rocket.rs/v0.4/guide/getting-started -- David

FYI, [Boost].UT (since v1.1.2) has relaxed the standard requirement to C++17 (although some limitations apply depending on the standard/compiler combination). In general, source_location is available since C++20, however, the builtin functionality powering it (__builtin_FILE(), __builtin_LINE()) is available since GCC-9/Clang-9. Therefore, now: * [Boost].UT with Clang/GCC >= 9 and with either C++17 or C++20 works as expected (although with C++17 it uses compiler builtin designed for C++20 source_location) * [Boost].UT with Clang/GCC < 9 and with either C++17 or C++20 still compiles/`works`, however, the file/line in assertion won't be propagated Also, MSVC-2019 doesn't support source_location yet neither with /std:C++20 nor with /std:C++latest, hence, the file/line is also not propagated yet. Example here -> https://godbolt.org/z/XNgMdN -Kris On Fri, Nov 22, 2019 at 1:53 PM Vinnie Falco wrote:

On Thu, Nov 21, 2019 at 7:24 AM Krzysztof Jusiak via Boost-users > I was wondering whether there is any interest in exploring

...

a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

I have no interest in a library that requires C++20, especially considering that C++20 is not even official yet but also because once C++20 is released, there will be hardly any users for many years. This project seems very much like it was written "just because", to use the latest language features, rather than for pragmatic reasons. I don't see anything compelling to use it over Boost.LightweightTest for example.

Thanks