Generating combined coverage reports for Cypress end to end tests and Jest unit tests

This post shows how different code coverage reports for Cypress end to end and Jest unit tests can be combined into one unified report.

This was motivated by a talk I’ve seen at ReactiveConf 2019 in Prague from Gleb Bahmutov, one of the core maintainers of Cypress. Among other interesting tidbits about testing with Cypress, he introduced a code coverage plugin for Cypress that could be used to generate coverage reports for end to end tests.

But first, let’s have a look at why it’s a good idea to collect code coverage in general.

Why we collect coverage

During development, we strive to keep the Jest unit test coverage at the current level of ~80%. But looking at the coverage number alone is not really helpful (and should not be a goal on its own). More importantly, we are looking for:

  • Sudden changes in coverage: If a pull request decreases coverage signficantly, chances are that we either forgot to write tests, or we deleted/disabled existing tests by accident. It might also be that we forgot to adapt tests after some refactoring.
  • Uncovered parts: Are there certain edge cases that are not covered but should be?

Codecov greatly helps us with that by providing diff views on every pull request that highlight uncovered lines.

All in all, this gives us greater confidence in our unit tests, but what about our end to end tests with Cypress?

How we combine unit and end to end test coverage

Following this excellent tutorial, it was straight forward to add the @cypress/code-coverage plugin to our Cypress setup. Summarizing the main steps that we had to do:

Add babel-plugin-istanbul

We installed the plugin and added this piece to our .babelrc file:

    "env": {
        "development": {
            "plugins": [
                "istanbul"
            ]
        }
    }

This instruments our client JavaScript code, i.e. it adds a global window.__coverage__ object. We made sure to only add it in development mode, as it would add quite some overhead to the production code otherwise.

Install @cypress/code-coverage plugin

To actually generate a coverage report for our Cypress tests, we need to install the plugin and its peer dependencies. Note: These instructions assume v1.10.4 of the plugin. With the most recent release, all necessary deps are installed automatically.

yarn add -D  @cypress/code-coverage nyc istanbul-lib-coverage

Then we register the plugin with Cypress as shown here. Now, whenever we run the Cypress end to end tests, we will have a coverage report generated.

Add CI pipeline steps

We would like to have both coverage reports merged and reported to any open pull request. So in our CI pipeline (using CircleCI), we have a step collect-test-coverage that runs after both, the end to end tests and the unit tests have finished. It performs the following steps:

  1. Copy both coverage reports into a single folder
  2. Run nyc merge on that folder
  3. Upload the resulting coverage.json using codecov

The steps are basically a simplified version of this example repo.

Results

We observed a ~10% increase in code coverage when looking at the combined code coverage (right hand chart), compared to only unit test coverage (left hand chart).

Visualised code coverage after merging end to end ad unit test coverage reports Visualised code coverage after merging end to end ad unit test coverage reports

What does this mean for us?

In general, end to end tests cover a lot more lines of code than unit tests, because they are spinning up the whole application and testing lots of different interacting components usually. This explains why we gained an additional 10% on top of our already high code coverage. But this does not mean that we’re not writing unit tests any more. Writing unit tests is still mandatory because of the following reasons:

  • They are faster to write.
  • They run faster (locally and on CI).
  • They are faster to debug.
  • There are certain error cases that cannot be tested via end to end tests, because they are not easily reachable from the UI.

So our usual flow still consists of writing lots of unit tests, making sure coverage does not drop significantly, and adding end to end tests afterwards to test user flows.

But in the end, it’s still nice to have one central overview of how much of our code is covered by any kind of test.

unsplash-logoYann Allegre