All posts by Dave Schweisguth

Accessory gems, part II: Local Gemfiles

The Bundler Ruby gem does its job well: it ensures that your application sees only the gems listed in its Gemfile. Sometimes, though, you might feel like it’s doing its job too well. You might want to add a gem to your app temporarily, perhaps for monitoring or debugging, or your app might have a gem in its Gemfile that isn’t needed in every environment and that you don’t want in yours. Bundler just says no. If a gem is in your app’s Gemfile your app will see it; if not, it won’t. If you want to add a gem to or remove it from your app, you have to add it to or remove it from your Gemfile and

.

Just editing your app’s Gemfile when you need to add or remove a gem has a couple of drawbacks:

  • It’s cumbersome. If you need to make the same change to your Gemfile regularly, that will get old fast. Besides, doing things manually and repeatedly is not what programming is all about.
  • It dirties your version control. You should be able to commit quickly and effortlessly, without having to pick through modified files to exclude changes not meant for production. And the fewer chances you have to mistakenly commit a change that you meant to be local-only, the better off we all are.

How can we give our Gemfiles some flexibility without those hassles? Bundler gives us a couple of openings: we can use a different name for our customized Gemfile to keep it out of the way of version control, and, since Gemfiles are just Ruby scripts, we can write code in our customized Gemfile to reuse our base Gemfile without copying and modifying it.

First, let’s give our customized Gemfile a new name. In part I, when we solved a related problem involving Gemfiles, we learned that we can tell Bundler to look for an app’s Gemfile under a different name by setting the environment variable

. If we name our local Gemfile

and set

to that, Bundler will use

for the lock file name. We can tell our version control to ignore those files (e.g. for git, add them to our

) and never have to think about them again when committing. So

is a good way to have a customized Gemfile without version control hassles.

If you usually work on a single project, you can set

in your shell environment and forget about it. You might also want to configure your editor to set

before running your app or its tests, for example in RubyMine’s run/debug configuration defaults. If you work on some projects that do have a Gemfile.local and some that don’t, you can set

on the command line, as in the following example:

You could also set and unset

with direnv, or do it in scripts or shell aliases.

Second, let’s see how we can reuse one Gemfile in another. Here’s a

that includes a base Gemfile and then adds and removes gems.

is a method in Bundler’s DSL. It reads the file with the given name,

s its contents and handles errors the way that Bundler does elsewhere. It’s an internal method, so it might go away; if it does,

will work pretty much the same way.

Adding gems is simple: just use the

method as you would in a standalone Gemfile.

Removing gems is a little hackier. Peeking at Bundler’s DSL again, we see that the

method accumulates dependencies in the

array. To forget a gem that was in the base Gemfile, just delete it from that array. (This example deletes the debugger gem, which is in my app’s Gemfile for the convenience of our CLI diehards but breaks RubyMine’s debugger.)

That covers everything I’ve needed to do in my Gemfile.local.

Finally, let’s solve a problem that we just introduced: how to preserve all of the dependencies from the base Gemfile. If we just

, Bundler will build a new

with a potentially different set of dependencies than those in

. Fortunately, if we copy

to

before we

, Bundler does exactly what we need: it reads the versions from

, adds and removes the gems that we added and removed in

, and writes a new

!  To keep up with updates to the base

and

, each time those files change we need to

the base Gemfile, copy

to

, and then

our

. That’s a bit of a pain, but it’s not so bad with a shell alias:

Whenever you check out a new version of the base Gemfile and Bundler complains that a gem is missing, just run

instead of

like you would ordinarily do. (This example assumes you’ve set

in your shell and need to set it back to the default to

.) Alternatively, you could bundle, copy and bundle in a git hook.

Thanks to Ruby and Bundler’s open design, your app’s Gemfile isn’t the straitjacket that it first appears to be. With a little configuration you can keep your application’s carefully chosen and tested dependencies and still customize it to meet the needs of particular environments.

Accessory gems, part I: Private Gemfiles

A Ruby application’s Gemfile does a lot for you: not only does it ensure that your application runs with exactly and only the gems that it needs, but it is a single place where you can look to find out what those gems are, and it makes it easy to install those gems with a single command. That easy installation, though, is a bit of an attractive nuisance. It’s tempting to add a gem to your Gemfile not because your application actually needs it, but only so the gem is installed when you

and will be there when you need it. Gemfiles can become cluttered with gems that are needed when working on a project, but not in the running application.

These accessory gems aren’t just distracting. A gem’s functionality might not be appropriate in production (if it logs additional, perhaps sensitive, information, for example). If an accessory gem monkey-patches something that your application depends on, it can cause bugs or other problems. Even if an accessory gem doesn’t actually change your application’s functionality, loading the gem’s code increases the time your application takes to start up, which slows development, testing and deployment. And most gems depend on other gems, each of which can cause any of the same problems as gems which are actually in your Gemfile. What’s more, if an accessory gem depends on a gem that your application needs anyway, your Gemfile then has an extra requirement on the depended-on gem’s version, which can prevent you from upgrading the depended-on gem when you need to.

So, what to do about those extra gems? If a gem’s code is actually used within the running application, the best you can do is to include it only in the appropriate Bundler groups, and if the gem or its dependencies still cause problems you’ll just have to stop using it or deal with the problems some other way. But if all the gem does is provide executables, as deployment tools like Chef or static analysis tools like rubocop do, you have a couple of other options.

You could move the accessory gem to a new project altogether. That solution makes the most sense when there is code that can move to that new project too. For example, some applications want their Chef cookbooks in a separate project, since cookbooks can be large and might be developed by a different team or released on a different schedule than the application itself. A separate project is cumbersome, however, and when the application and its deployment have to change at the same time it takes extra effort to ensure that the right versions of each get to production in the right order.

Or you can have it both ways: just move accessory gems to their own Gemfiles within the same project. The other files that those gems need can be changed in sync with your application, but the gems themselves and their dependencies won’t affect your application at runtime. It does mean a little extra effort to install and invoke the accessory gems, but that’s easy to hide with a little scripting.

Let’s look at an example. The POODLE attack on SSLv3 motivated many of Indiegogo’s third-party service providers to sunset their support of that protocol. Several HTTP client gems that our main web application uses defaulted to SSLv3, so we needed to upgrade them to newer versions that defaulted to TLS. But each of those gems was used by several other gems, each with its own version requirement. Eventually we’ll have to upgrade or replace all of those HTTP-client-gem-using gems, but that’s a big job, and it will be easier for some gems if we wait for their maintainers do the job for us. Fortunately, some of those gems didn’t need to be in our application’s Gemfile, and moving them into their own Gemfiles simplified the main application’s gem dependency tree enough that we could just update the HTTP-client gems that it uses to SSLv3-free versions.

The biggest drag on our application’s SSL/TLS usage was our Capistrano deploy, which uses Chef to find the instances to be deployed to. The application itself doesn’t use Capistrano and Chef, of course, so off they go to their own private Gemfile, in

:

Now, how to make it easy for everyone to use the gems even though they’re in their own Gemfiles? We just provide a little wrapper script (‘binstub’) for each gem’s executable. Bundler will generate binstubs for you, but we want our binstubs to do a little more than Bundler’s, so we write them ourselves. To point Bundler to the private Gemfile, we use the

environment variable. To save users the trouble of first installing and then bundle-execing, we just run

in the binstub before we

. In

:

Running the script the first time creates

(

always creates a lock file whose name matches the name of the

), and we put it in version control just like a regular

.

To deploy, we now just need to

deploy instead of

. And we can’t forget, because Capistrano is no longer in the main Gemfile and if we mistakenly

Bundler will tell us that.

There is one situation in which you need to be a little extra careful if you have multiple Gemfiles in a project. If you only work on one project for a given Ruby installation, or if you use RVM gemsets, you might be in the habit of using

to remove gems that you no longer use. If some gems are in their own Gemfiles, Bundler won’t know about them, and

will remove them. On the other hand, if all of your accessory gems’ binstubs

for you, the accessory gems will be reinstalled when they’re needed.

Private Gemfiles are, then, a nice way to knock a few chips off of your monolithic Rails application. But there is another scenario that we haven’t handled. What if we want to use the same Gemfile in different environments, but we don’t want one of the gems in it, or we want to add a gem without having to copy and modify it? For the answer to that one, see our next episode, on local Gemfiles.