Maintaining a consistent Gemfile.lock for a gem with native extensions

Question

I work in a team of engineers that develops a gem with native executables. For contractual reasons, it's important that when we deploy to a new environment, all dependencies use the same exact versions that we tested with. The Gemfile can only maintain first-order dependency versions, not recursive ones. For this reason, we've historically kept our lockfile checked into github, but this has prevented us from upgrading bundler ever since the 1.14 release. The problem is that we have development machines that are both OSX and Linux, and starting in 1.14, the beginning of the lockfile in the gem repository changed from:

PATH
  remote: .
  specs:
    engine (21.2.13)
      <gemspec dependencies here>

to

PATH
  remote: .
  specs:
    engine (21.2.13-x86_64-linux)
      <gemspec dependencies here>

This is a problem because when a developer pulls the repo on OSX and runs bundle install, it will change the contents of the lockfile. Then, when a linux developer does the same, it'll change all over again, creating a bunch of spurious changes in the git history of the lockfile!

I tried running bundle lock --add-platform x86_64-linux and bundle lock --add-platform x86_64-darwin, hoping that this would convince bundler to maintain two entries for the different platforms, instead of flipflopping between them. It did produce duplicate entries for some of the gems in the GEM section of the lockfile, but not for the one in the PATH\n specs: section.

Currently, our Gemfile contains the line:

gemspec name: "engine"

which loads engine.gemspec. This file contains:

Gem::Specification.new do |spec|
  ...
  spec.platform = Gem::Platform::CURRENT
  ...
end

which I suspect is the problem. I tried including the gemspec twice in the Gemfile, and using a global variable to specify the platform to use, but bundler only loads it the first time, and skips over the second attempt.

Does anyone know of a solution that will allow us to keep both platform-specific gem versions in the same lockfile?

Or otherwise, is there a way to turn off bundler's new-ish behavior that appends the platform name to the gem version? In the past, when the lockfile simply specified "21.2.13" and our gemserver contained two copies of each version (with binaries built for the two platforms), bundler never had any issues resolving the correct version for the current machine, so this seems like the lockfile storing superfluous information. Can I somehow tell it to stop?

score 0 · Accepted Answer · answered Sep 13 '17 at 14:01

I had heard variously that it was both recommended and not-recommended to have your lockfile in version control, but never fully understood the reasons. This seems to have been the source of the problem!

The recommendation I see most often (although this may not always be possible), is that the lockfile should never be under version control for a gem, but always for a standalone application. The idea is that a gem is intended to be portable, and so should have some flexibility in its dependencies. Importantly, this flexibility doesn't prevent our development environment from being consistent, provided that all testing happens not in the gem repository but in the repository of the standalone application which uses the gem.

That standalone application's lockfile already tracks the dependencies that we thought we were using the gem's lockfile to track, and when the application gets deployed, all of the gem's dependencies are fixed inside the application, not inside the gem.

The "spec" section of the Gemfile.lock should only be present in a gem repo, and the fact this section names gems in a platform-dependent way is not an issue, since the gem is referred to by a platform-independent name in the lockfile of the application using it.

Maintaining a consistent Gemfile.lock for a gem with native extensions

1 Answers1