On Thu, Oct 30, 2014 at 10:23 AM, Adam Wulkiewicz wrote: Paul A. Bristow wrote: So I developed a simple program downloading summaries, test logs and
saving
the enhanced result.
You may check it out here: https://github.com/awulkiew/summary-enhancer Neat :-) +1 I'd certainly find this very helpful when checking the Boost.Math test
monster
matrix. Yes, it's big and various kind of errors can be found there. FYI, it has some new features.
It can keep a log about the tests previously detected as failing and
generate a report about new failures.
It's also possible to send an email containing this report (to many
recipients). It uses SMTP without authentication and e.g. Google throws
such messages into SPAM, but a filter can be set for them. Alternatively
SPF record could be configured in the machine's domain but I haven't played
with it. Some local SMTP server could also probably work. Currently the
mailing configuration must be placed in the file mail.cfg containing
per-line settings (see the example in the repo).
So if run periodically the program should notify about new regressions. That's also very helpful!
The big question in my mind is whether we should enhance the current
regression testing system or start fresh using one of the continuous
integration frameworks?
--Beman