New Product: Test only what you change with Carryforward Flags

We are extremely excited to announce our new product, Carryforward Flags.

What are Carryforward Flags and how can I use them?

Carryforward Flags is designed for engineering organizations who do not run their full test suite in their continuous integration (“CI”) build each time they open a pull request or merge code

Codecov is now the first universal code coverage solution to ingest partial test suite runs and still display accurate code coverage.


Daniel Strauss

As a result of Carryforward Flags, we can now get coverage for all our apps and packages in our monorepo (over 70 apps and hundreds of packages) in a way that is easy to use and easy for teams to digest. It also has allowed us to add blocking Github checks for code coverage reductions per app/package. This really helps ensure more thoroughly tested code is getting out the door.

Daniel Straus, Engineering Manager

Mike McCurry Group – Top Rated Realtor

Why are some teams not running their full test suite on every commit? 

CI tools fundamentally changed development and testing, but CI build processes can get very long.

Many CI/CD providers allow you to build in parallel to decrease the CI build time (Codecov already supports merging reports from parallel builds).

Even with trends in CI run parallelization, organizations are still seeking shorter test run cycles for 1) minimizing developer downtime during the CI build and 2) decreasing compute costs for running excess tests. 

Two specific use cases have emerged:

1). Partial test suite runs by category:
In this example, a user may have several logical groups of tests in their test suite. Consider the case of two types of test that can be run on every commit: unit, and end-to-end. Unit tests may be quick to run on every commit; however, end-to-end tests may be time consuming and users may only choose to run these tests as components change that may demonstrably impact the application. 

2). Delta coverage on an entire repo: 
Some organizations run tests only for code that a developer changes, and that code’s dependencies. Such setups are more complicated than traditional testing setups, but the primary benefit is that you only run the tests you need to run on a particular commit.  For large test suites, this testing approach can dramatically decrease CI run time. 

Give it a try

See how Codecov gives you coverage metrics right in your workflow

How does Codecov’s Carryforward Flags feature work? 

You can read more about Carryforward Flags in our Docs and see them in use in an example repository

Codecov’s Flags feature already permits users to tag different test coverage outputs (e.g, unit tests, integration tests, etc.) from CI processes such that those outputs can be meaningfully reviewed in Pull Requests and within the Codecov web application. 

Carryforward Flags are an enhancement to Flags that allows users to indicate to Codecov that it needs to look at coverage data from earlier commits and apply that coverage information to the commit currently being processed. Using Carryforward Flags, teams that leverage delta coverage and partial test suite runs to lower their CI run time can now have a complete picture of their project’s coverage on any commit. The coverage that is absent on the current commit, is simply “carried forward” from a previous commit where that coverage was uploaded to Codecov.

The key need addressed in Codecov’s Carryforward Flags feature is an understanding that, if parts of the test suite are not run, Codecov needs to go backwards in time and find coverage data from the most recent time that those parts of test suite were run.

Codecov already permits our users to tag different test coverage outputs from CI process via a product called Flags

Carryforward Flags are specified in the codecov.yml, just like Codecov’s standard Flags. This is best demonstrated with an example codecov.yml, shown below:

This example creates three flags: ui, unit, and enterprise. In each case, two files are specified in the paths block for each flag; but folders, regex’s, and glob patterns are also accepted. 

  • The ui and unit flags are given the carryforward: true property, 
  • While enterprise is set to carryforward: false, the default value.

Consider two commits, Commit 0 and Commit 1 that contain the above flag specification. In this example, it is assumed that coverage has been uploaded for all three flags in Commit 0 (providing a coverage baseline). In Commit 1 a coverage report (ui_cov.xml) will be uploaded for the ui flag. No coverage report will be uploaded for the unit and enterprise flags for Commit 1. Given this scenario, the following diagram represents the state of Coverage at Commit 0 and Commit 1:

Updated coverage with Carryforward Flags

In this example, the coverage for ui_1.py and ui_2.py was taken directly from the coverage report uploaded for Commit 1. The ui_cov.xml coverage report was uploaded using the following syntax:

bash <(curl https://codecov.io/bash) -f ui_cov.xml -F ui

This upload command specifies both the path to the coverage file for files under the ui flag and the appropriate flag name (ui in this case). Once uploaded, coverage was calculated to the ui_1.py and ui_2.py files in the same manner as any report upload. 

Coverage remains the same with Carryforward Flags and no coverage reports

Since no report is uploaded for the unit flag, which has a setting of carryforward: true, Codecov reaches back to Commit 0 to carry forward the coverage for all files covered by the unit flag. As a result, those files have coverage information for Commit 1 that is equivalent to Commit 0. 

Coverage drops with no coverage reports uploaded and Carryforward set to false

Finally, since the enterprise flag has a setting of carryforward: false no coverage is carried forward for any file covered by the enterprise flag. Since no coverage report was uploaded for the enterprise flag for Commit 1, the ent_1.py and ent_2.py files have 0% coverage for Commit 1. 


The above example demonstrates the power of Carryforward Flags for those build processes which do not run their full test suites on every commit. Previously, projects that relied on partial test suite runs or delta builds would be unable to have a consistent picture of their project’s total coverage since total coverage would only represent whatever coverage information was uploaded for a particular commit. Now, with Carryforward Flags, Codecov can fully support these more complex testing approaches.

Give it a try

See how Codecov gives you coverage metrics right in your workflow

%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close