How we get coverage on parallelized test builds

We lacked a window into our code coverage, specifically to answer these questions:

  • Which files are covered and which are vulnerable?
  • How is our coverage changing over time?
  • Are there areas of the codebase that need extra testing?

It’s possible to manually generate coverage with RubyMine, the IDE that many of us use, but to get the full picture requires running each spec that might be related to your target file or method—in a codebase with as many files and tests as Indiegogo’s, it’s a time-consuming proposition.

SimpleCov

The SimpleCov gem watches when RSpec runs tests to see which lines of code it hits, and how many times. The gem is smart enough to understand which lines are code that was executed (represented in the resulting JSON as a positive integer), missed (a 0), or not relevant (null). The resulting JSON from a SimpleCov report might look like this:

(From these results, SimpleCov generates pretty HTML reports with percentages of lines covered by file and directory.)

The gem worked wonderfully for running a few local tests, but we wanted to see app-wide coverage, so we added SimpleCov to our CircleCI test build. Circle uses multiple VMs to parallelize large test suites, allowing ours to complete in a reasonable amount of time. The downside of this parallelization was that because each VM only runs a subset of all our tests, each VM’s coverage report is similarly incomplete. Specifically we saw results claiming there were uncovered lines of code when those lines had in fact been covered by tests that ran on a different VM. Unfortunately these incomplete coverage reports that were too inaccurate to be of much use.

Coveralls

That’s where Coveralls came in. Coveralls is a cloud-based service that hoovers up incomplete coverage reports you send it, merges them, and displays the results with stylish formatting. It’s a paid service for private repos, but pretty inexpensive and easy to set up.

We hoped to display coverage grouped by scrum team, but this wasn’t possible given the customization options in Coveralls. In addition, we also encountered performance issues due to the scope and breadth of our code base, so it became clear that we should opt for a more specifically-tailored solution.

CircleCI

Given the desired features and scope of the task, a better option was to work directly with the service that runs our full test suite, CircleCI. We’d wanted to do this from the start, but after seeing the benefits of visible, accurate, and complete coverage reports with Coveralls, we decided it was worth the effort to make SimpleCov work within CircleCI. We reached out to CircleCI for advice and they told us about new features (since announced) around SSH access and scripting the VMs. To that end, we added the SimpleCov gem to our test build and created a rake task for code coverage to run after our tests had completed.

Our proposed solution followed these steps:

  1. determine which container is the last to finish
  2. that container aggregates and merges the partial reports
  3. then it generates & saves the complete report as a Circle artifact

The first step was easy—copy the reports from each VM.

When you have at least one report from each VM, you know you’re on the last VM to finish (because reports only appear after that box has finished its tests).

Merging the SimpleCov reports was the trickiest part to get working–existing documentation and the gem’s structure didn’t help much, but eventually we were able to verify the merged report was accurate by checking it against what we were seeing in Coveralls.

Finally, we added coverage groups, so our scrum teams can look at aggregate coverage for sections of the code that interest them. A little friendly competition to increase coverage might be in order!

The new system is extensible, free, and reliable—a vast improvement over the “run it in your IDE” days. Now that this infrastructure is in place, we are thinking about other improvements like a commit-specific coverage group, so it’s obvious whether the files that were touched in a given commit are well-tested.

We’ve considered packaging this code into a gem so other CircleCI users can benefit from targeted, merged coverage. Comment below if you’d be interested!

3 thoughts on “How we get coverage on parallelized test builds”

Comments are closed.