How to convert a Rails plugin into a gem

December 12th 2009 — Comments and Reactions

Recently I decided to convert my fork of the auto_complete plugin into a gem; I called it “repeated_auto_complete.” In the end it was very easy to convert a plugin into a gem; all I had to do was:

Make sure there was a code file in the lib folder with the same name as the gem, and
Move or copy the init.rb into a subfolder called “rails.”

This is simple enough, but why do I need to do this? These changes seem rather odd, and also it took me about 3-4 hours of debugging to figure out what I needed to do. The answer has to do with the way the Rails framework loads gems… this is more confusing and complicated than you might think! The rest of this article will show exactly how this works in detail, comparing how gems and plugins are loaded.

The load path works the same way for plugins and gems

Rails treats the load path in the same way for gems as it does for plugins. This is a relief, and also not a surprise since gems and plugins are very similar to each other. The best way to get a sense of how the load path works with plugins and gems is just to inspect it directly in the console. To do this, let’s start by creating a new sample app:

$ rails sample
      create  
      create  app/controllers
      create  app/helpers
      create  app/models
etc…

And now let’s install the auto_complete plugin:

$ cd sample
$ ./script/plugin install git://github.com/rails/auto_complete.git
Initialized empty Git repository in .git/
warning: no common commits
remote: Counting objects: 13, done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 13remote:  (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (13/13), done.

If we start a Rails console we can use the command in bold to just look at the load path:

$ ./script/console 
Loading development environment (Rails 2.3.5)
>> $LOAD_PATH.each { |path| puts path }; nil
.../gems/activesupport-2.3.5/lib/active_support/vendor/i18n-0.1.3/lib
.../gems/activesupport-2.3.5/lib/active_support/vendor/tzinfo-0.3.12
.../gems/activesupport-2.3.5/lib/active_support/vendor/memcache-client-1.7.4
/Users/pat/rails-apps/sample/app/controllers/
/Users/pat/rails-apps/sample/app
/Users/pat/rails-apps/sample/app/models
/Users/pat/rails-apps/sample/app/controllers
/Users/pat/rails-apps/sample/app/helpers
/Users/pat/rails-apps/sample/lib
/Users/pat/rails-apps/sample/vendor/plugins/auto_complete/lib
/Users/pat/rails-apps/sample/vendor
.../gems/rails-2.3.5/lib/../builtin/rails_info/
.../gems/rails-2.3.5/lib
etc…

Here we can see the various application paths for my new sample app, as well as the paths of a few of the gems found on my laptop. For clarity, I've shortened the path to my gems folder, and there are many more gems that I’m not showing here. The line in bold indicates that the lib folder for the auto_complete plugin is included in the load paths array, allowing Rails to look inside the auto_complete plugin in order to find missing constants.

Now if I remove the auto_complete plugin…

$ rm -rf vendor/plugins/auto_complete

… and install the repeated_auto_complete gem (from gemcutter):

$ gem sources -a http://gemcutter.org
http://gemcutter.org added to sources
$ sudo gem install repeated_auto_complete
Password:
Successfully installed repeated_auto_complete-0.1.0
1 gem installed
Installing ri documentation for repeated_auto_complete-0.1.0...
Installing RDoc documentation for repeated_auto_complete-0.1.0...

… and add a call to config.gem in config/environment.rb:

Rails::Initializer.run do |config|
…
  config.gem "repeated_auto_complete"
…
end

… and view the load path again:

$ ./script/console 
Loading development environment (Rails 2.3.5)
>> $LOAD_PATH.each { |path| puts path }; nil
.../gems/activesupport-2.3.5/lib/active_support/vendor/i18n-0.1.3/lib
.../gems/activesupport-2.3.5/lib/active_support/vendor/tzinfo-0.3.12
.../gems/activesupport-2.3.5/lib/active_support/vendor/memcache-client-1.7.4
/Users/pat/rails-apps/sample/app/controllers/
/Users/pat/rails-apps/sample/app
/Users/pat/rails-apps/sample/app/models
/Users/pat/rails-apps/sample/app/controllers
/Users/pat/rails-apps/sample/app/helpers
/Users/pat/rails-apps/sample/lib
.../gems/repeated_auto_complete-0.1.0/lib
/Users/pat/rails-apps/sample/vendor
.../gems/rails-2.3.5/lib/../builtin/rails_info/
.../gems/rails-2.3.5/lib
etc…

In bold I can see the gem’s lib folder appear just as the plugin’s lib folder did earlier. In fact, it even appears at the same position in the array so classes should be loaded in exactly the same way for a gem as they were for a plugin.

For a gem, you need to have the expected code file in your lib folder

This next issue caused me some serious headaches… hopefully this explanation will save you some time. To explore how gems are loaded by Rails, let’s unpack my “repeated_auto_complete” gem that I just installed above:

$ rake gems:unpack
(in /Users/pat/rails-apps/sample)
Unpacked gem: '/Users/pat/rails-apps/sample/vendor/gems/repeated_auto_complete-0.1.0'

Now I have a local copy of the gem’s code in my vendor/gems directory. Next, let’s see what happens when I delete the “repeated_auto_complete.rb” file from the lib folder – in other words, the code file with the same name as the gem:

$ rm vendor/gems/repeated_auto_complete-0.1.0/lib/repeated_auto_complete.rb 
$ ./script/console 
Loading development environment (Rails 2.3.5)
no such file to load -- repeated_auto_complete
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
.../gems/activesupport-2.3.5/lib/active_support/dependencies.rb:156:in `require'
.../gems/activesupport-2.3.5/lib/active_support/dependencies.rb:521:in `new_constants_in'
.../gems/activesupport-2.3.5/lib/active_support/dependencies.rb:156:in `require'
.../gems/rails-2.3.5/lib/rails/gem_dependency.rb:208:in `load'
.../gems/rails-2.3.5/lib/initializer.rb:307:in `load_gems'
.../gems/rails-2.3.5/lib/initializer.rb:307:in `each'
.../gems/rails-2.3.5/lib/initializer.rb:307:in `load_gems'
.../gems/rails-2.3.5/lib/initializer.rb:164:in `process'
.../gems/rails-2.3.5/lib/initializer.rb:113:in `send'
.../gems/rails-2.3.5/lib/initializer.rb:113:in `run'
/Users/pat/rails-apps/sample/config/environment.rb:9
etc…

Now we get an error trying to load the Rails environment! The reason why is simple: when Rails loads each gem specified in the environment.rb file with a call to config.gem, it tries to load a code file with exactly the same name. Let’s take a look at line 307 of initializer.rb which appears in the stack trace above:

def load_gems
  unless $gems_rake_task
    @configuration.gems.each { |gem| gem.load }
  end
end

What’s going on here is:

load_gems is a method of the Rails::Initializer object. This is the class that we refer to in environment.rb.
@configuration is the configuration object that was yielded to the initializer block in environment.rb… in order words the value of the “config” variable that we used in our call to “config.gem”
@configuration.gems is an array of GemDependency objects; each one created by a config.gem call. If you’re interested, you can see these are created at line 811 in initializer.rb.
For each GemDependency object Rails calls “load.”

Let’s take a look at the GemDependency.load method: line 208 in gem_dependency.rb, also in the stack trace above:

def load
  return if @loaded || @load_paths_added == false
  require(@lib || name) unless @lib == false
  @loaded = true
rescue LoadError
  puts $!.to_s
  $!.backtrace.each { |b| puts b }
end

When the GemDependency object was created earlier, two of its attributes were loaded with values as follows:

name: this is set to the name of the gem – “repeated_auto_complete” in my example
@lib: this is set to the value of the “:lib” option provided to the config.gem call.

So if you read the code above, you’ll see that Rails allows for three possible cases when loading a gem:

config.gem ‘repeated_auto_complete’ – in this case Rails will call require “repeated_auto_complete” and fail if a code file with that name is not present in the load path. This is what just happened to us above.
config.gem ‘repeated_auto_complete’, :lib => ‘something_else’ – in this case Rails will call require “something_else” and fail if a code file with that name is not present in the load path.
config.gem ‘repeated_auto_complete’, :lib => false – in this case Rails will not call require at all for this gem.

Note that for plugins none of this is an issue: Rails simply adds the plugin's lib folder to the load path array and that's it. But when you convert a plugin into a gem, you need to decide which variation of config.gem your users will have to put in environment.rb.

Init.rb has to move

The next thing Rails does after loading each plugin or gem is to execute a file called init.rb. If you’re the author of a gem or plugin this gives you a chance to initialize your code… for example to add certain modules you’ve written to classes in the application, etc. But as I mentioned at the beginning, if you’re writing a gem or converting a plugin into a gem, you need to be sure the init.rb file is located inside a folder called “rails.” Let’s see if we can find out how Rails does this; first let’s restore the original gem’s code:

$ rm -rf vendor/gems/repeated_auto_complete-0.1.0
$ rake gems:unpack
(in /Users/pat/rails-apps/sample)
Unpacked gem: '/Users/pat/rails-apps/sample/vendor/gems/repeated_auto_complete-0.1.0'

And now let’s edit the init.rb file, located at vendor/gems/repeated_auto_complete-0.1.0/rails/init.rb:

puts caller
ActionController::Base.send :include, AutoComplete
ActionController::Base.helper AutoCompleteMacrosHelper
ActionView::Helpers::FormBuilder.send :include, AutoCompleteFormBuilderHelper

I added the first line in bold: “puts caller.” This will display a stack trace leading to this file when we startup the sample application:

$ ./script/console 
Loading development environment (Rails 2.3.5)
.../gems/rails-2.3.5/lib/rails/plugin.rb:158:in `evaluate_init_rb'
.../gems/activesupport-2.3.5/lib/active_support/core_ext/kernel/reporting.rb:11:in `silence_warnings'
.../gems/rails-2.3.5/lib/rails/plugin.rb:154:in `evaluate_init_rb'
.../gems/rails-2.3.5/lib/rails/plugin.rb:48:in `load'
.../gems/rails-2.3.5/lib/rails/plugin/loader.rb:38:in `load_plugins'
.../gems/rails-2.3.5/lib/rails/plugin/loader.rb:37:in `each'
.../gems/rails-2.3.5/lib/rails/plugin/loader.rb:37:in `load_plugins'
.../gems/rails-2.3.5/lib/initializer.rb:369:in `load_plugins'
.../gems/rails-2.3.5/lib/initializer.rb:165:in `process'
.../gems/rails-2.3.5/lib/initializer.rb:113:in `send'
.../gems/rails-2.3.5/lib/initializer.rb:113:in `run'
/Users/pat/rails-apps/sample/config/environment.rb:9

This time I’ve bolded the “plugin.rb” file; if you look at line 152 in plugin.rb you’ll see this:

def evaluate_init_rb(initializer)
  if has_init_file?
    silence_warnings do
      # Allow plugins to reference the current configuration object
      config = initializer.configuration
      eval(IO.read(init_path), binding, init_path)
    end
  end
end

So this just calls “eval()” on the init.rb file, assuming that “init_path” indicates the path of this file, executing the plugin’s or gem’s initialization code. If you poke around a bit inside of plugin.rb, you’ll see this code for the Rails:Plugin class, which represents each plugin that Rails finds in your application:

def classic_init_path
  File.join(directory, 'init.rb')
end
def gem_init_path
File.join(directory, 'rails', 'init.rb')
end
def init_path
File.file?(gem_init_path) ? gem_init_path : classic_init_path
end

If we read the definition of init_path, we see that it uses either rails/init.rb or init.rb, whichever it finds first. This seems to indicate that for a Rails plugin, you can place init.rb either in the “rails” subfolder, or in the main plugin folder, and that it will find and use the copy in the “rails” folder if you happen to have both.

However, for a gem things don’t work this way. You can see why if you look down towards the bottom of the plugin.rb file:

class GemPlugin < Plugin
  # Initialize this plugin from a Gem::Specification.
  def initialize(spec, gem)
    directory = spec.full_gem_path
    super(directory)
    @name = spec.name
  end
  def init_path
    File.join(directory, 'rails', 'init.rb')
  end
end

It turns out that Rails uses a different class to represent gems, called “GemPlugin” (what a confusing name!). In this case we can see that init_path is defined to be the path rails/init.rb and nothing else. This means that gems intended to be used in a Rails application must put their init.rb file in the rails folder.

To summarize this logic:

Rails plugins can place init.rb either in the root plugin folder, or in a subfolder called “rails.” If they have both, the rails folder copy will be used.
Rails gems must place their init.rb file in a “rails” subfolder.

The actual reason why Rails was implemented this way was that possibly a gem might be used by more than one Ruby framework (e.g. Merb, Sinatra, etc.) and might have different init.rb code for each framework. But a Rails plugin can only be used in a Rails application. Finally, Rails has allowed for plugins to work in the original manner with init.rb in the root folder, or for a plugin to be a gem at the same time, with init.rb in the rails folder.