Zeitwerk: Add engine/gem directories to autoload path of parent Rails app

Question

I'm trying to switch to Zeitwerk in an existing, older Gem (Rails::Engine). Until now all files have been manually required and autoloaded. Plus the engine's lib-folder was added to autoload_paths via config.autoload_paths += paths["lib"].to_a in class MyEngine < Rails::Engine .

The switch to use Zeitwerk worked fine via the described way on the Readme:

require "zeitwerk"
loader = Zeitwerk::Loader.for_gem
.
. --> more project specific stuff here
.
loader.setup # ready!

So far so good! Now I want to use the Gem in a Rails app and add the engine's directories to the autoload_path of the Rails application. This worked fine previously via the above mentioned config.autoload_paths. If I do this now, it fails with the following error message:

Zeitwerk::Error:
  loader

#<Zeitwerk::Loader:0x00000001094d4bd0
...

wants to manage directory /gems/<NameOfGem>/lib, which is already managed by

#<Zeitwerk::Loader:0x0000000106b2d728
...

What is the correct way to add the engine's lib-directories to the autoload path of the Rails application?

Thank you!

Alex · Accepted Answer · 2022-07-11T17:42:25.813

Rails sets up two loaders main and once:

Rails.autoloaders.main
Rails.autoloaders.once

These are just instances of Zeitwerk::Loader. Rails also gives you a config to add root directories to these loaders:

config.autoload_paths         # main
config.autoload_once_paths    # once

When gem's lib directory is added to autoload through one of these configs, lib becomes a root directory:

# config.autoload_paths += paths["lib"].to_a

>> Rails.autoloaders.main.root_dirs
=> 
...                    
 "/home/alex/code/stackoverflow/my_engine/lib"=>Object,
...

When a class from the gem is called, zeitwerk uses registered loaders to look up and to load the file corresponding to this class.

If the gem then sets up its own loader:

require "zeitwerk"
loader = Zeitwerk::Loader.for_gem
loader.setup

another instance of Zeitwerk::Loader is created with its own root directories:

>> Zeitwerk::Registry.loaders.detect { |z| z.tag == "my_engine" }
=> 
#<Zeitwerk::GemLoader:0x00007fe5e53e0f80
...
 @root_dirs={"/home/alex/code/stackoverflow/my_engine/lib"=>Object},
...

# NOTE: these are the two loaders registered by rails
>> Zeitwerk::Registry.loaders.select { |z| z.tag =~ /rails/ }.count
=> 2

Zeitwerk doesn't allow two loaders to have a shared directory and raises an error showing two conflicting loaders.

Because the gem is a Rails::Engine the best option is to let rails manage zeitwerk loaders and remove Zeitwerk::Loader.for_gem setup.

# only use rails config
config.autoload_paths += paths["lib"].to_a

On the other hand, gem loader is already set up and config.autoload_paths is not needed.

# NOTE: without any loaders
>> MyEngine::Test
# (irb):1:in `<main>': uninitialized constant MyEngine::Test (NameError)                                            
# MyEngine::Test                                
#         ^^^^^^

# NOTE: with gem loader
#
#   require "zeitwerk"
#   loader = Zeitwerk::Loader.for_gem
#   loader.setup                           
#
>> MyEngine::Test
=> MyEngine::Test

# NOTE: with rails `main` loader
#
#   config.autoload_paths += paths["lib"].to_a
#
>> MyEngine::Test
=> MyEngine::Test

# NOTE: with gem loader and rails loader
$ bin/rails c
# /home/alex/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader.rb:480:in
# `block (3 levels) in raise_if_conflicting_directory':
# loader (Zeitwerk::Error)

Update

# Use rails loaders
# config.autoload_path          .->  Zeitwerk::Loader(@tag=rails.main)
# config.autoload_once_path     |->  Zeitwerk::Loader(@tag=rails.once)
#                               |
# Or create a new loader        |
# Zeitwerk::Loader.for_gem      |->  Zeitwerk::GemLoader(@tag=my_engine)
#                               |
# my_engine/lib can only be in one of these

Zeitwerk does the loading and reloading. Rails is just another gem here.

If you don't use rails config, Zeitwerk will find files through Zeitwerk::GemLoader(@tag=my_engine), that the gem has created.

If you use rails config, Zeitwerk will find files through Zeitwerk::Loader(@tag=rails.main), that rails has created (making GemLoader unnecessary).

If lib is a root directory in any of the existing loaders there is no need to have any requires or autoloads for files in lib directory. Except for things that are needed before Zeitwerk kicks in, like MyEngine::Engine from lib/my_engine/engine.rb.

Thanks for the detailed answer! So if I wanna get rid of all the requires (and what not) in the Engine and use Zeitwerk as a file loader, I cannot use `config.autoload_paths += paths["lib"].to_a` any longer. But it's not necessary, as the Zeitwerk::Loader from the main Rails app will find the files from the Engine anyway. Correct? Or I just keep everything as it is (with requires, autoloads, ...) and use `config.autoload_paths`. — Tococorocko, Jul 11 '22 at 13:25

Zeitwerk: Add engine/gem directories to autoload path of parent Rails app

1 Answers1