Reporting accurate code coverage
General Code Coverage Tip (for everyone): Measure your coverage using branches, not lines. Branch coverage tells you how well error cases are tested and therefore a far better measure of quality than line-based. I have seen over 90% line coverage but the branch coverage is more like 60% to 70% on average over the boost CMT-managed repositories. Here's the shell script I use on travis to generate the data and feed it to codecov.io: https://github.com/jeking3/boost-ci/blob/master/ci/travis/codecov.sh The trick is to feed --rc lcov_branch_coverage=1 to the correct commands to generate and preserve the branch information. Thanks, - Jim
On Wed, Jul 11, 2018 at 7:34 AM James E. King III <jking@apache.org> wrote:
General Code Coverage Tip (for everyone):
Measure your coverage using branches, not lines. Branch coverage tells you how well error cases are tested and therefore a far better measure of quality than line-based. I have seen over 90% line coverage but the branch coverage is more like 60% to 70% on average over the boost CMT-managed repositories.
Below is some proof of what I mentioned about line coverage being (quite) deceptive - Boost.Rational has 98.8% "line" coverage but only 59.7% branch coverage. Line coverage reported by lcov in this case means the line was hit in some way. It doesn't mean every branch on that line was exercised for true and false cases to ensure proper behavior. Good unit tests will exercise all the branch cases and this typically proves that error handling code works as intended. In short, don't rely on line coverage for your quality measurements. Always rely on branch coverage. Extracting /home/travis/build/boostorg/boost-root/boost/rational.hpp Extracted 1 files Writing data to coverage.info Summary coverage rate: lines......: 98.9% (281 of 284 lines) functions..: 97.5% (424 of 435 functions) branches...: 59.7% (964 of 1615 branches) +lcov --gcov-tool=gcov-7 --rc lcov_branch_coverage=1 --list coverage.info Reading tracefile coverage.info |Lines |Functions |Branches Filename |Rate Num|Rate Num|Rate Num ======================================================== [/home/travis/build/boostorg/boost-root/boost/] rational.hpp |98.9% 284|97.5% 435|59.7% 1615 ======================================================== Total:|98.9% 284|97.5% 435|59.7% 1615 - Jim
Dear James,
On 13. Jul 2018, at 16:45, James E. King III via Boost <boost@lists.boost.org> wrote:
In short, don't rely on line coverage for your quality measurements. Always rely on branch coverage.
I generally see the wisdom in your argument. I use coveralls.io to display and track coverage. On Travis, I execute: pip install --user cpp-coveralls coveralls -r .. -b . --verbose --exclude ${TRAVIS_BUILD_DIR}/deps --gcov=`which ${GCOV}` --gcov-options '\-lp'; Seems like coveralls may have added tracking of branch coverage in 2017 for some languages, https://github.com/lemurheavy/coveralls-public/issues/31 <https://github.com/lemurheavy/coveralls-public/issues/31> but I can't see it for my project. The situation is not clear. Do I have to migrate to codecov.io <http://codecov.io/> to follow your advice? Has anyone been able to get branch coverage working with coveralls? Best regards, Hans
Yes: See the PR linked in the issue: https://github.com/openssl/openssl/pull/4447/files Simply replace '\-lp'; by '\-lpbc'; Side note on branch coverage: I think it was coveralls who first supported "partial" coverage. That is a line, where not all branches are covered. You can configure, which types of such partials are counted as "hit" or "miss". Doing that I quickly found the branch coverage in C++ is incredibly hard. Even for non-optimized debug builds the compiler will add branches where you won't expect them and you have no clue in how to get them covered. This is much easier for scripting languages. @James What is the current state on that? Did you test this extensively? Did you also encounter such spurious branch misses? If not and the state is the same as like 2 years ago, then branch coverage as a metric is not really useful in C++, as you have many false negatives. Note: I'm not talking about branch coverage in general, just that it is (was?) to unreliable in C++. You can't achieve 100% (not that you really should) due to the compiler. Alex Grund Am 13.07.2018 um 17:12 schrieb Hans Dembinski via Boost:
Dear James,
On 13. Jul 2018, at 16:45, James E. King III via Boost <boost@lists.boost.org> wrote:
In short, don't rely on line coverage for your quality measurements. Always rely on branch coverage. I generally see the wisdom in your argument.
I use coveralls.io to display and track coverage. On Travis, I execute:
pip install --user cpp-coveralls coveralls -r .. -b . --verbose --exclude ${TRAVIS_BUILD_DIR}/deps --gcov=`which ${GCOV}` --gcov-options '\-lp';
Seems like coveralls may have added tracking of branch coverage in 2017 for some languages, https://github.com/lemurheavy/coveralls-public/issues/31 <https://github.com/lemurheavy/coveralls-public/issues/31> but I can't see it for my project. The situation is not clear.
Do I have to migrate to codecov.io <http://codecov.io/> to follow your advice? Has anyone been able to get branch coverage working with coveralls?
Best regards, Hans
participants (3)
-
Alexander Grund -
Hans Dembinski -
James E. King III