Should I include test files in code coverage calculations?

The background

Code tests (unit tests, integration tests, etc) are here to ensure your code is functioning as expected.

Many modern software development organizations use a Continuous Integration provider like CircleCI, TravisCI or Azure Pipeline to continually run these tests and make sure those tests are passing.

Code coverage is a metric that measures what % of your total lines of code are tested:

a.) Lines of tested code 

b.) all lines of code 

c.) % code coverage

The issue

Test files and source code (“src”), though different in intention, still all look like code.

So the key question emerges → Should test files themselves be counted in your code coverage? 

The argument AGAINST including test files in your code coverage reports

When you think about measuring code coverage and, over time, improving that metric, you want to focus on source code that will move the needle for your stakeholders.

Including test files in code coverage will increase the a.) numerator and b.) denominator of code coverage, making it harder to see key changes to your source / application code.

Simply put, test file coverage is not the code coverage you are looking for.

The argument FOR including test files in your code coverage

The separation of source code files and test files is an engineering best practice, but not inherent to software development.

Out of context, *all files look like code files* and therefore should contribute to your code coverage.  

Said simply, what will test the test files? 

The good news

Here at Codecov, we’ve put the power to make this decision back in your hands. Using our ignoring paths feature, you can choose to ignore all test files (or not) in your code coverage calculation.

For example, adding into your YAML:

ignore:
 "path/to/folder"  # ignore folders and all its contents
 "test_*.rb"       # wildcards accepted
 "[a-z]+/test_.*"  # regexp accepted 


Would ignore different types of tests and folders. 

Happy testing! 

Jerrod

%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close