All posts by Mike Luby

Rethinking Payments

Your friend sends you a link. It’s a heart-warming story of a community pulling together, and they need your help. Or maybe it’s this new device straight out of science fiction, and you can make it a reality by backing it. Or it’s a local artist with a bold new idea who needs support to bring their dreams to life. Point is, it’s on Indiegogo, and you want to give them money.

So you head over to the payment flow, and see this. Whoa.


There’s a lot going on over many pages, and the code is even more confusing!

Payments is the beating heart of crowdfunding, and the controller that supports these pages has been around since Indiegogo was founded in 2007. Needless to say, features were added, experiments were run, and bugs were fixed until the code had become so clogged with technical debt that even minor changes could send the whole system into cardiac arrest.

Indiegogo’s fearless leaders, recognizing this development bottleneck, tasked the Iron Bank, our payments team, with performing bypass surgery. Our mandate was to create a clean, responsive, modular foundation for future features and experiments, then stitch it into place to take over for the old code.

One fundamental idea behind the pay flow, as we call it, was a series of UI components that were mostly pure and stateless, allowing them to be reused, reasoned about, and tested with relative ease. It also made things easier to discuss with design. One of the low-level components we created was the input field, with some nice animation to keep users informed as they type:


We used a few base-level components to create larger components for each of the key sections that Design had distilled from discussions with users and internal teams. The payment module demonstrates the flexibility of this approach.


To make users’ lives easier, we added some polish, like detecting which card icon to show. We also added spaces or slashes to fields to reduce ambiguity.

Building up components in this way also made it easier to think about responsive design. The old code had dramatically different code paths for desktop and mobile payments, and though the user interfaces hid some of the differences, it meant double the work for any features in this area. Since a key priority for the the new version was to have one code path that supported multiple view sizes, we ensured that the components were flexible when it came to container widths so they look good from iPhone 4 to iMax.

With the UI looking pretty solid, we began linking it up with the backend. The old system had lots of implicit dependencies and business logic buried throughout it, so a large portion of our effort was tracking these down and writing tests to make sure no functionality was lost.

An interesting feature of the new system is that it unifies the process for two payment services, PayPal and Stripe. When you click submit, Indiegogo asks Stripe to process the payment, and Stripe responds right away, allowing us to redirect the user to the thank you page (they also send us a webhook later confirming a payment’s success). However, Paypal redirects the user to PayPal to complete the payment, then redirects back to Indiegogo’s thank you page (and sends a webhook confirming success). To have one code path that supports both processes, we have the server return a URL in both cases for the client to redirect. Then we consider the webhooks the true response from each payment processor. Abstracting away the differences between processors makes it much easier to add features that affect both, and keeps the code and tests leaner and more DRY.

Having learned from past releases that bugs and missing requirements always seem to crop up at the last minute, we held two bug testing parties, one midway through when the UI was mostly done, and one a week or two before launch, with the whole company. Having everyone try to break the new system had the added benefit of preparing everyone for the upcoming launch, so they could speak knowledgeably to users about the changes. Many bugs were caught early on, but thanks to the two parties the launch itself was rock-solid.

We’d hoped that users would find the new experience easier to complete, allowing more of them to make it all the way through. At worst, we were even willing to accept a slightly worse conversion rate given how much easier the underlying code would be to build upon. As it turned out, users loved the new way to pay, with desktop contributions rising 14% and mobile contributions doubling!

Given the excellent user response, a solid code foundation for new feature development, and an almost bug-free launch, we declare the patient again in good health. See for yourself, and look for more new crowdfunding features from Indiegogo!

How we get coverage on parallelized test builds

We lacked a window into our code coverage, specifically to answer these questions:

  • Which files are covered and which are vulnerable?
  • How is our coverage changing over time?
  • Are there areas of the codebase that need extra testing?

It’s possible to manually generate coverage with RubyMine, the IDE that many of us use, but to get the full picture requires running each spec that might be related to your target file or method—in a codebase with as many files and tests as Indiegogo’s, it’s a time-consuming proposition.


The SimpleCov gem watches when RSpec runs tests to see which lines of code it hits, and how many times. The gem is smart enough to understand which lines are code that was executed (represented in the resulting JSON as a positive integer), missed (a 0), or not relevant (null). The resulting JSON from a SimpleCov report might look like this:

(From these results, SimpleCov generates pretty HTML reports with percentages of lines covered by file and directory.)

The gem worked wonderfully for running a few local tests, but we wanted to see app-wide coverage, so we added SimpleCov to our CircleCI test build. Circle uses multiple VMs to parallelize large test suites, allowing ours to complete in a reasonable amount of time. The downside of this parallelization was that because each VM only runs a subset of all our tests, each VM’s coverage report is similarly incomplete. Specifically we saw results claiming there were uncovered lines of code when those lines had in fact been covered by tests that ran on a different VM. Unfortunately these incomplete coverage reports that were too inaccurate to be of much use.


That’s where Coveralls came in. Coveralls is a cloud-based service that hoovers up incomplete coverage reports you send it, merges them, and displays the results with stylish formatting. It’s a paid service for private repos, but pretty inexpensive and easy to set up.

We hoped to display coverage grouped by scrum team, but this wasn’t possible given the customization options in Coveralls. In addition, we also encountered performance issues due to the scope and breadth of our code base, so it became clear that we should opt for a more specifically-tailored solution.


Given the desired features and scope of the task, a better option was to work directly with the service that runs our full test suite, CircleCI. We’d wanted to do this from the start, but after seeing the benefits of visible, accurate, and complete coverage reports with Coveralls, we decided it was worth the effort to make SimpleCov work within CircleCI. We reached out to CircleCI for advice and they told us about new features (since announced) around SSH access and scripting the VMs. To that end, we added the SimpleCov gem to our test build and created a rake task for code coverage to run after our tests had completed.

Our proposed solution followed these steps:

  1. determine which container is the last to finish
  2. that container aggregates and merges the partial reports
  3. then it generates & saves the complete report as a Circle artifact

The first step was easy—copy the reports from each VM.

When you have at least one report from each VM, you know you’re on the last VM to finish (because reports only appear after that box has finished its tests).

Merging the SimpleCov reports was the trickiest part to get working–existing documentation and the gem’s structure didn’t help much, but eventually we were able to verify the merged report was accurate by checking it against what we were seeing in Coveralls.

Finally, we added coverage groups, so our scrum teams can look at aggregate coverage for sections of the code that interest them. A little friendly competition to increase coverage might be in order!

The new system is extensible, free, and reliable—a vast improvement over the “run it in your IDE” days. Now that this infrastructure is in place, we are thinking about other improvements like a commit-specific coverage group, so it’s obvious whether the files that were touched in a given commit are well-tested.

We’ve considered packaging this code into a gem so other CircleCI users can benefit from targeted, merged coverage. Comment below if you’d be interested!