rails 7 zietwerk constants not found-CodePudding

I'm upgrading our app from rails 6 to 7, I was using zeitwerk instead of the classic autoloader with rails 6 without any issues, now without any changes to the folder/file naming convention or structure it fails to find any classes/modules inside lib folder. I've been debugging the zeitwerk gem and the autoloads hash has file paths as keys and [Namespace, Constant Name] as the value, with rails 6 it had correct name space, e.g. for a file in \user\project\rails6_test\lib\folder\config.rb it'd have Folder as the namespace and Config defined inside config.rb would be found correctly in Folder namespace but after the migration with rails 7 it always has Object as the namespace, not just for top level folders but for all nested files and obviously it can't find those constants in the Object.

I don't see anything that I missed from the upgrade guide, what could be missing here ?

Update

I tried running bin/rails r 'pp ActiveSupport::Dependencies.autoload_paths' and I get the error below,

/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/loader/callbacks.rb:25:in `on_file_autoloaded': expected file /home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/restful_resource.rb to define constant RestfulResource, but didn't (Zeitwerk::NameError)

  raise Zeitwerk::NameError.new("expected file #{file} to define constant #{cpath}, but didn't", cref.last)
  ^^^^^
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/kernel.rb:28:in `require'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/loader/helpers.rb:95:in `const_get'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/loader/helpers.rb:95:in `cget'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/loader.rb:238:in `block (2 levels) in eager_load'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/loader/helpers.rb:26:in `block in ls'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/loader/helpers.rb:18:in `each_child'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/loader/helpers.rb:18:in `ls'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/loader.rb:233:in `block in eager_load'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/loader.rb:218:in `synchronize'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/loader.rb:218:in `eager_load'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/loader.rb:318:in `each'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/loader.rb:318:in `eager_load_all'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/application/finisher.rb:78:in `block in <module:Finisher>'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/initializable.rb:32:in `instance_exec'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/initializable.rb:32:in `run'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/initializable.rb:61:in `block in run_initializers'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/3.1.0/tsort.rb:228:in `block in tsort_each'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/3.1.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/3.1.0/tsort.rb:431:in `each_strongly_connected_component_from'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/3.1.0/tsort.rb:349:in `block in each_strongly_connected_component'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/3.1.0/tsort.rb:347:in `each'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/3.1.0/tsort.rb:347:in `call'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/3.1.0/tsort.rb:347:in `each_strongly_connected_component'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/3.1.0/tsort.rb:226:in `tsort_each'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/3.1.0/tsort.rb:205:in `tsort_each'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/initializable.rb:60:in `run_initializers'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/application.rb:372:in `initialize!'
    from /home/leonidas/projects/ruby_on_rails/rainman/config/environment.rb:5:in `<top (required)>'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/kernel.rb:35:in `require'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/zeitwerk-2.5.4/lib/zeitwerk/kernel.rb:35:in `require'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/application.rb:348:in `require_environment!'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/command/actions.rb:28:in `require_environment!'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/command/actions.rb:15:in `require_application_and_environment!'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/commands/runner/runner_command.rb:33:in `perform'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/command/base.rb:87:in `perform'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/command.rb:48:in `invoke'
    from /home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/railties-7.0.2.2/lib/rails/commands.rb:18:in `<top (required)>'
    from bin/rails:4:in `require'
    from bin/rails:4:in `<main>'

I have config.autoload_paths << "#{config.root}/lib" in my application.rb and to test eager loading with zeitwerk I add config.eager_load_paths = Dir["#{config.root}/lib/**/"], let me know if you need any more info.

Update 2

After getting rid of the wildcard with just /lib running bin/rails r 'pp ActiveSupport::Dependencies.eager_load_paths' gives me the following output

["/home/leonidas/projects/ruby_on_rails/project_name/lib",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/microservices/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/microservices/dir/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/dir/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/dir/dir/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/dir/dir/dir/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/dir/dir/dir/dir/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/dir/dir/dir/dir/dir/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/dir/dir/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/dir/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/dir/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/project_name/dir/",
 "/home/leonidas/projects/ruby_on_rails/project_name/lib/tasks/",
 "/home/leonidas/projects/ruby_on_rails/project_name/app/controllers",
 "/home/leonidas/projects/ruby_on_rails/project_name/app/mailers",
 "/home/leonidas/projects/ruby_on_rails/project_name/app/models",
 "/home/leonidas/projects/ruby_on_rails/project_name/app/serializers",
 "/home/leonidas/projects/ruby_on_rails/project_name/app/workers",
 "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/actionmailbox-7.0.2.2/app/controllers",
 "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/actionmailbox-7.0.2.2/app/jobs",
 "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/actionmailbox-7.0.2.2/app/models",
 "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/activestorage-7.0.2.2/app/controllers",
 "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/activestorage-7.0.2.2/app/controllers/concerns",
 "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/activestorage-7.0.2.2/app/jobs",
 "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/activestorage-7.0.2.2/app/models",
 "/home/leonidas/projects/ruby_on_rails/project_name/spec/mailers/previews"]

and running bin/rails r 'pp ActiveSupport::Dependencies._eager_load_paths' outputs the following

{"/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/activestorage-7.0.2.2/app/controllers",
  "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/activestorage-7.0.2.2/app/controllers/concerns",
  "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/activestorage-7.0.2.2/app/jobs",
  "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/activestorage-7.0.2.2/app/models",
  "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/actionmailbox-7.0.2.2/app/controllers",
  "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/actionmailbox-7.0.2.2/app/jobs",
  "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/actionmailbox-7.0.2.2/app/models",
  "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/actiontext-7.0.2.2/app/helpers",
  "/home/leonidas/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/actiontext-7.0.2.2/app/models",
  "/home/leonidas/projects/ruby_on_rails/project_name/app/controllers",
  "/home/leonidas/projects/ruby_on_rails/project_name/app/mailers",
  "/home/leonidas/projects/ruby_on_rails/project_name/app/models",
  "/home/leonidas/projects/ruby_on_rails/project_name/app/serializers",
  "/home/leonidas/projects/ruby_on_rails/project_name/app/workers",
  "/home/leonidas/projects/ruby_on_rails/project_name/lib"}

I was looking into the zeitwerk.rake task in rails and it seems it doesn't check any existing and non empty directories that exist in autoload paths but not in the eager load paths, I don't understand what should be happening here, is autoload_paths not supposed to have the expanded folder structure and only list root folders ? because in my case autoload having nested dirs too and eager_load only having root dirs is printing that warning about not checking the folders inside lib.

CodePudding user response：

Thanks for the updates. The problem is this glob:

config.eager_load_paths  = Dir["#{config.root}/lib/**/"]

Adding to the eager load paths, adds also to the autoload paths (I find this a bit confusing, but it is how it works). Therefore, that glob is adding, for example, lib/project_name to the autoload paths. The key observation to understand what is happening is that when there are nested root paths, the most nested one wins as far as the files and directories below that one is concerned.

So, since lib is in the autoload paths, anything below lib is assumed to be defined in Object. But, since lib/project_name is in the autoload paths too, that subtree also represents Object. It is the same situation like the one you have in app/models and app/models/concerns.

These docs explain it, but they were easy to overlook in this case because it was not obvious your configuration is affected by that point.

The solution is to add only lib:

config.autoload_paths << "#{config.root}/lib"
config.eager_load_paths << "#{config.root}/lib"

Just like that, without wildcards. Technically, the first line is redundant given the second one, but in my view this is subtle and not obvious, I personally like to make both explicit, but as you like here. (There won't be duplicated entries, the final collection passes through uniq.)

Since we are on it, let me also comment something extra. Maybe your application already does this, but just in case, let me say that lib has often files and directories that are not meant to be autoloaded. For example, lib/tasks. It is clean to be deliberate about them and tell the autoloader:

Rails.autoloaders.main.ignore("#{config.root}/lib/tasks")

Otherwise, the autoloader will believe lib/tasks is a namespace, will define an autoload for Tasks in Object, will eager load (== actually define) the module if eager loading is enabled, etc. Conceptually, lib/tasks does not have code to be autoloaded/eager loaded, and the configuration has to reflect that. Same with other similar directories.