How to handle Gemfile.lock with local gem repositories?

Question

Introduction

We have a Rails5 application, which is splittend in about 10+ engines and a core application, which mounts these engines.

A engine in our case is a plain old rails engine defined as a gem and located in a dedicated git repository. The Gemfile in the core application refers all engines (see below).

Required Behavior

• For deployments a specific version of the gem/engine should be used (by the core).
• For local development the HEAD of a local cloned repository should be used (by the core).

Current Setup

We achieved that by doing following steps for each engine in the core app:

• Adding gem 'nice_engine1', '~> 0.0.1', branch: :develop, git: '[...]', tag: 'v0.0.1'
• Setting a bundler config entry: bundle config local.nice_engine1 ../nice_engine1

That seems to work, however we didn't try to run a deployment with that setup yet.

Issues with that Setup

Everytime one of the repositories is updated locally and we run a bundle install in the core, bundler updates the Gemfile.lock to the new HEAD ref of the local engine repository. We used to commit that change of the Gemfile.lock.

Unfortunately does that cause some issues:

1. If someone updates the core app, without updating the engines, it may happen, that the core Gemfile.lock refers to a git commit of an engine, which doesn't exist locally. That leads to errors if one tries to use the rails app.
2. At deployment time (I think) the Gemfile.lock may referring to a commit id, which is newer then the commit of the tag/version that I want to deploy. I'm not sure what will happen in that case, but I fear, that this will just lead us to troubles.
3. We have a lot of commits in the core changing the Gemfile.lock (potentially for each change in one of the engines).
4. Using another engine branch locally then master forces the developer to change the branch name in the main apps Gemfile

Question

What would be the correct/best way to manage the Gemfile and the Gemfile.lock in the given situation to avoid these issues?

For some hints about best practices, improvement suggestions and so on, how to use bundler and git in order to fulfill our requirements, I'd be thankful.

Answer 1

If someone updates the core app, without updating the engines, it may happen, that the core Gemfile.lock refers to a git commit of an engine, which doesn't exist locally. That leads to errors if one tries to use the rails app.

There are a few options here:

1) Use a gem like git-bundle so that you can run a single command such as gitb pull to update the main application and the engines.

2) Open the main application and the engines in an IDE like RubyMine as it updates all repositories if you click VCS -> Update project.

3) Create a script similar to git-bundle that updates all repositories at the same time.

At deployment time (I think) the Gemfile.lock may referring to a commit id, which is newer then the commit of the tag/version that I want to deploy. I'm not sure what will happen in that case, but I fear, that this will just lead us to troubles.

Lets say a specific revision of the main application is tagged with 1.0. The Gemfile.lock of that revision specifies which revisions of the engines are used for 1.0. It's not possible to deploy tag 1.0 of the main application where it runs engine revisions other than what is specified in Gemfile.lock.

We have a lot of commits in the core changing the Gemfile.lock (potentially for each change in one of the engines).

Yes that is correct. This is the file that effectively binds different git repositories and revision together into a single application. The fact that it's in a version controlled file makes it nice and traceable.

What would be the correct/best way to manage the Gemfile and the Gemfile.lock in the given situation to avoid these issues?

1) There is a gem for that. It's called git-bundle. I wrote it and we have been using it in different applications for about 5 years now with great success. Pull requests or feedback is welcome.

2) Don't use tags when referencing engines in the main application Gemfile. If new functionality is being created in an engine do that in a feature branch by branching all repositories on local override:

gitb checkout -b feature/branch_name

If needed, release management can be done by having multiple branches such as master, release and stable on all git repositories. Move code from one environment to the next with:

gitb checkout release
gitb pull origin master
gitb push

3) If you need to extend (decorate or monkey patch) classes in engines or vice versa: we also made activesupport-decorators after looking at the various options that are available.

Answer 2

After the good answer from Pierre Pretorius, some research and new experiences, I managed to solve all 4 problems. I'll explain the setup and the solution per problem:

1. If someone updates the core app, without updating the engines, it may happen, that the core Gemfile.lock refers to a git commit of an engine, which doesn't exist locally. That leads to errors if one tries to use the rails app.

To solve that issue (and some other issues which are not relevant for this thread), we wrote a script, which cares about updating the engines and does some other necessary setup tasks. For each engine the following steps will be executed:

• Set the bundler config for the main app to use the local repo
• git pull -r --autostash or clones if the repo is not cloned yet
• bundle install
• yarn install
• bower install

After that, the scripts pulls and sets up the main app.

For simpler setups, you can use the git-bundle gem, which let you execute a command on all engines.

That finally solves the issue, that something of the stack is not up-to-date.

---

2. At deployment time (I think) the Gemfile.lock may referring to a commit id, which is newer then the commit of the tag/version that I want to deploy. I'm not sure what will happen in that case, but I fear, that this will just lead us to troubles.

To solve that issue, we told the team NOT to commit the Gemfile.lock. Instead we update and commit the Gemfile.lock while releasing a new version of the main app. This way, the main app always references a concrete (the latest usually) version of each engine. Bundler however ensures by the local overwrite, that the HEAD of the local engine repository will be used independent of the locked version.

This way the deployments are consistent and locked against concrete versions, downgrades are easily possible and developers don't have to mess around with versions, commitIds etc.

---

3. We have a lot of commits in the core changing the Gemfile.lock (potentially for each change in one of the engines).

See solution for issue #2: We just don't commit those changes. That's not exactly brilliant, but it actually works.

---

4. Using another engine branch locally then master forces the developer to change the branch name in the main apps Gemfile.

In order to fix that, we set the bundler config disable_local_branch_check to true:

bundle config disable_local_branch_check true

This way, we can remove the branch name from the Gemfile for the engines and the developers can use whatever branch they may want to use locally (for example feature branches while developing/reviewing).

---

After that a Gemfile entry for one of our engines looks like this:

gem 'nice_engine1', '0.0.1', git: '[...]', tag: 'v0.0.1'