Aleksey Gurtovoy wrote:
Peter Dimov writes:
Question: what should I do if a test failure does not render the library unusable, but I don't want to mark the failure as "expected" because I _never_ expect failures. ;-)
Peter, I'd really like to finally make your dissatisfaction with the current markup rules go away, but in order to come up with a resolution that we all can agree on I need to understand your use case, so please bear with me while I'm trying to achieve that :).
I guess my question is: do you want to keep the failures yellow "for yourself" or for users of the library? If it's the former, wouldn't keeping the already known, "cannot-do-anything-about-it" failures highlighted in the report make it much harder to notice possible new failures, thus basically rendering the detailed view useless for the purpose of examining, well, a detailed regressions/failures picture?
I don't want to hide the failures in the detailed view, but the users should see a green box in the summary. This kind of failure is not common (in the libraries I maintain). There are two such failures on the smart_ptr page, and a bit more on the bind page, but I don't think that they can mask new failures. A release should never go out with a "real" failure, only with "non-critical" failures. Once released, any new failures would be regressions and impossible to miss. I want to keep the non-critical failures visible because they are failures. :-) Any non-green color is fine with me.