Accessory gems, part I: Private Gemfiles

A Ruby application’s Gemfile does a lot for you: not only does it ensure that your application runs with exactly and only the gems that it needs, but it is a single place where you can look to find out what those gems are, and it makes it easy to install those gems with a single command. That easy installation, though, is a bit of an attractive nuisance. It’s tempting to add a gem to your Gemfile not because your application actually needs it, but only so the gem is installed when you

and will be there when you need it. Gemfiles can become cluttered with gems that are needed when working on a project, but not in the running application.

These accessory gems aren’t just distracting. A gem’s functionality might not be appropriate in production (if it logs additional, perhaps sensitive, information, for example). If an accessory gem monkey-patches something that your application depends on, it can cause bugs or other problems. Even if an accessory gem doesn’t actually change your application’s functionality, loading the gem’s code increases the time your application takes to start up, which slows development, testing and deployment. And most gems depend on other gems, each of which can cause any of the same problems as gems which are actually in your Gemfile. What’s more, if an accessory gem depends on a gem that your application needs anyway, your Gemfile then has an extra requirement on the depended-on gem’s version, which can prevent you from upgrading the depended-on gem when you need to.

So, what to do about those extra gems? If a gem’s code is actually used within the running application, the best you can do is to include it only in the appropriate Bundler groups, and if the gem or its dependencies still cause problems you’ll just have to stop using it or deal with the problems some other way. But if all the gem does is provide executables, as deployment tools like Chef or static analysis tools like rubocop do, you have a couple of other options.

You could move the accessory gem to a new project altogether. That solution makes the most sense when there is code that can move to that new project too. For example, some applications want their Chef cookbooks in a separate project, since cookbooks can be large and might be developed by a different team or released on a different schedule than the application itself. A separate project is cumbersome, however, and when the application and its deployment have to change at the same time it takes extra effort to ensure that the right versions of each get to production in the right order.

Or you can have it both ways: just move accessory gems to their own Gemfiles within the same project. The other files that those gems need can be changed in sync with your application, but the gems themselves and their dependencies won’t affect your application at runtime. It does mean a little extra effort to install and invoke the accessory gems, but that’s easy to hide with a little scripting.

Let’s look at an example. The POODLE attack on SSLv3 motivated many of Indiegogo’s third-party service providers to sunset their support of that protocol. Several HTTP client gems that our main web application uses defaulted to SSLv3, so we needed to upgrade them to newer versions that defaulted to TLS. But each of those gems was used by several other gems, each with its own version requirement. Eventually we’ll have to upgrade or replace all of those HTTP-client-gem-using gems, but that’s a big job, and it will be easier for some gems if we wait for their maintainers do the job for us. Fortunately, some of those gems didn’t need to be in our application’s Gemfile, and moving them into their own Gemfiles simplified the main application’s gem dependency tree enough that we could just update the HTTP-client gems that it uses to SSLv3-free versions.

The biggest drag on our application’s SSL/TLS usage was our Capistrano deploy, which uses Chef to find the instances to be deployed to. The application itself doesn’t use Capistrano and Chef, of course, so off they go to their own private Gemfile, in

:

Now, how to make it easy for everyone to use the gems even though they’re in their own Gemfiles? We just provide a little wrapper script (‘binstub’) for each gem’s executable. Bundler will generate binstubs for you, but we want our binstubs to do a little more than Bundler’s, so we write them ourselves. To point Bundler to the private Gemfile, we use the

environment variable. To save users the trouble of first installing and then bundle-execing, we just run

in the binstub before we

. In

:

Running the script the first time creates

(

always creates a lock file whose name matches the name of the

), and we put it in version control just like a regular

.

To deploy, we now just need to

deploy instead of

. And we can’t forget, because Capistrano is no longer in the main Gemfile and if we mistakenly

Bundler will tell us that.

There is one situation in which you need to be a little extra careful if you have multiple Gemfiles in a project. If you only work on one project for a given Ruby installation, or if you use RVM gemsets, you might be in the habit of using

to remove gems that you no longer use. If some gems are in their own Gemfiles, Bundler won’t know about them, and

will remove them. On the other hand, if all of your accessory gems’ binstubs

for you, the accessory gems will be reinstalled when they’re needed.

Private Gemfiles are, then, a nice way to knock a few chips off of your monolithic Rails application. But there is another scenario that we haven’t handled. What if we want to use the same Gemfile in different environments, but we don’t want one of the gems in it, or we want to add a gem without having to copy and modify it? For the answer to that one, see our next episode, on local Gemfiles.