Tag Archives: testing

How we get coverage on parallelized test builds

We lacked a window into our code coverage, specifically to answer these questions:

  • Which files are covered and which are vulnerable?
  • How is our coverage changing over time?
  • Are there areas of the codebase that need extra testing?

It’s possible to manually generate coverage with RubyMine, the IDE that many of us use, but to get the full picture requires running each spec that might be related to your target file or method—in a codebase with as many files and tests as Indiegogo’s, it’s a time-consuming proposition.

SimpleCov

The SimpleCov gem watches when RSpec runs tests to see which lines of code it hits, and how many times. The gem is smart enough to understand which lines are code that was executed (represented in the resulting JSON as a positive integer), missed (a 0), or not relevant (null). The resulting JSON from a SimpleCov report might look like this:

(From these results, SimpleCov generates pretty HTML reports with percentages of lines covered by file and directory.)

The gem worked wonderfully for running a few local tests, but we wanted to see app-wide coverage, so we added SimpleCov to our CircleCI test build. Circle uses multiple VMs to parallelize large test suites, allowing ours to complete in a reasonable amount of time. The downside of this parallelization was that because each VM only runs a subset of all our tests, each VM’s coverage report is similarly incomplete. Specifically we saw results claiming there were uncovered lines of code when those lines had in fact been covered by tests that ran on a different VM. Unfortunately these incomplete coverage reports that were too inaccurate to be of much use.

Coveralls

That’s where Coveralls came in. Coveralls is a cloud-based service that hoovers up incomplete coverage reports you send it, merges them, and displays the results with stylish formatting. It’s a paid service for private repos, but pretty inexpensive and easy to set up.

We hoped to display coverage grouped by scrum team, but this wasn’t possible given the customization options in Coveralls. In addition, we also encountered performance issues due to the scope and breadth of our code base, so it became clear that we should opt for a more specifically-tailored solution.

CircleCI

Given the desired features and scope of the task, a better option was to work directly with the service that runs our full test suite, CircleCI. We’d wanted to do this from the start, but after seeing the benefits of visible, accurate, and complete coverage reports with Coveralls, we decided it was worth the effort to make SimpleCov work within CircleCI. We reached out to CircleCI for advice and they told us about new features (since announced) around SSH access and scripting the VMs. To that end, we added the SimpleCov gem to our test build and created a rake task for code coverage to run after our tests had completed.

Our proposed solution followed these steps:

  1. determine which container is the last to finish
  2. that container aggregates and merges the partial reports
  3. then it generates & saves the complete report as a Circle artifact

The first step was easy—copy the reports from each VM.

When you have at least one report from each VM, you know you’re on the last VM to finish (because reports only appear after that box has finished its tests).

Merging the SimpleCov reports was the trickiest part to get working–existing documentation and the gem’s structure didn’t help much, but eventually we were able to verify the merged report was accurate by checking it against what we were seeing in Coveralls.

Finally, we added coverage groups, so our scrum teams can look at aggregate coverage for sections of the code that interest them. A little friendly competition to increase coverage might be in order!

The new system is extensible, free, and reliable—a vast improvement over the “run it in your IDE” days. Now that this infrastructure is in place, we are thinking about other improvements like a commit-specific coverage group, so it’s obvious whether the files that were touched in a given commit are well-tested.

We’ve considered packaging this code into a gem so other CircleCI users can benefit from targeted, merged coverage. Comment below if you’d be interested!

ASCII Tables for Clearer Testing

I often find it difficult to understand a test. This can happen when the test was written by someone else, or even when I wrote it myself mere months ago. Usually the heart of the test is simple and each line of code is easily understood, but the context and setup can be extensive and located far away.

I’ve started using ASCII tables to define the initial test data. These tables are both concise and easy to understand, and that leads to simpler tests that are easier to maintain.

For context, at Indiegogo we use rspec, jasmine, and other testing frameworks in the Behavior-Driven Development category. Our tests will often specify nested contexts that enumerate possible states before adding our expectations. A typical test looks something like this:

With this structure you know every possible state is tested.

Unfortunately, these structures often explode into huge files with thousands of lines of code. It’s pretty difficult to know the expected state for a line of code when the test data is initialized in multiple places hundreds of lines away.

Now, consider an alternative organization for the example above using ASCII tables:

The table provides a concise and easily-understood presentation of the various states and the expected result.

This is not a new idea. Ward Cunningham’s Fit: Framework for Integrated Test and Bob Martin’s FitNesse both show state and expectations in tables, but making tables with ASCII characters means that you can embed these in existing test files, yielding many of the benefits of Fit and FitNesse without adopting the whole framework.

Introducing The ATV Gem

We initially added tables as here documents to some spec files and parsed them with locally-defined methods. Now we have extracted these table-parsing methods and created an open source project on GitHub, as well as the ATV (ASCII Table Values) ruby gem.

Building Tests With ATV

ATV returns your data as strings. Your code can use those strings however you want:

You can insert dynamic values using string interpolation:

With a little meta-programming you can include methods that are rspec assertions:

One challenge with expressing rspec assertions in tables is keeping the rspec failure message meaningful. Normally that message is generated from the describe, contexts and example descriptions.

A possible remedy uses the table to dynamically create the complete rspec example, including the description. Another approach is to add a failure message to your rspec assertion, as we did above.

Here a method is used to load the data (via ATV) and assign the resulting instances to instance variables so that specific, individual object instances can be accessed in the example:

Tables That Are Not ASCII Tables

Some of my coworkers were inspired by the tables concept but wanted something different. Their approach uses white space to organize a hash for easy reading:

Their solution is concise, easy to understand and easy to create. It’s also actual ruby code making it easier customize. Obviously ASCII tables are not the only way to organize your data.

How do you create these ASCII tables?

The benefit of these tables is easy to see but they may be difficult to create. I suspect most modern editors have modes that ease the creation and maintenance of these tables. I prefer to use the built-in table editor that comes with emacs Org Mode, or you can use the terminal-table gem to create your tables.

RubyMine’s column selection mode helps but I’m hoping this offering will inspire you to create a RubyMine plugin that is column aware like the table editor in org mode. If you do, please share.

Are ASCII tables right for your test?

Scale is important when deciding if you should use an ASCII table or some other approach for establishing test data. An ASCII table is probably overkill if you are initializing just a few variables. Likewise, if your test is initializing 2 or more attributes with two or more states then consider what the initialization would be like if summarized in a table. Also consider organizing your data using arrays and hashes with white space to show structure.