Code record/playback using Rails 3 generators

July 27th 2010 — Comments and Reactions

For a while I’ve been thinking that writing a Rails generator is a fairly difficult thing to do. First you need to learn about Thor and the Rails generator system: what sort of Ruby class you need to write, how to handle arguments, how to run commands like “copy_file”, etc. Then you need to write ERB files to produce the code that you’d like to generate, which is always a chore.

So last night I wrote a gem called generate_from_diff that let’s you create Rails 3 generators automatically using a code record/playback model. Here’s how it works:

You make a code change in some Rails project, and commit it to your Git repo.
You run a command from my gem to extract this code change from the Git repo into a new Rails 3 generator.
You later run this new generator in any other Rails 3 app, and your recorded code changes are “played back,” or applied to your new project, using the Unix patch utility written by Larry Wall.

You’ve created a Rails 3 generator without ever writing a single line of generator code!

Disclaimer: I just got this working last night, so it’s still very rough; but if the idea seems worthwhile I’ll clean it up and try to make it more robust and useable. Another disclaimer: none of this will work on Windows since it relies on the Unix patch utility.

Example: recording code into a new generator

Let’s look at an example to try to make this a bit clearer. Support I create a new Rails 3 application:

$ rails new first_app
      create  
      create  README
      create  Rakefile
      create  config.ru
      create  .gitignore
      create  Gemfile
etc...

And let’s run bundle install to be sure I have all the required gems. This is usually not necessary for a new, empty Rails app, but I want to have my Gemfile.lock file created... more on that in a moment.

$ cd first_app
$ bundle install
Fetching source index for http://rubygems.org/
Using rake (0.8.7) 
Using abstract (1.0.0) 
Using activesupport (3.0.0.beta4) 
Using builder (2.1.2) 
etc...

And let’s create a new Git repo here and check the empty application into it:

$ git init
Initialized empty Git repository in /Users/pat/.../first_app/.git/
$ git add .
$ git commit -m"New sample app"

This first Git revision will serve as the baseline for recording my new generator, which I’ll do in a minute. The reason I ran bundle install was to insure that the Gemfile.lock file would be included in the baseline... and so not included in the recorded code change.

Now let’s write some code that I can record into a new generator. Suppose at my company I want to create a controller that returns the build number, diagnostics and some other information about each of my Rails apps. I might do this by creating a new controller as follows:

$ rails generate controller build_info
      create  app/controllers/build_info_controller.rb
      invoke  erb
      create    app/views/build_info
      invoke  test_unit
      create    test/functional/build_info_controller_test.rb
      invoke  helper
etc...

And in this new controller I’ll add a single index action:

class BuildInfoController < ApplicationController
  def index
    render :text => 'Some interesting build info about this app...'
  end
end

Finally, I’ll add a route to send “build_info” requests to this action:

FirstApp::Application.routes.draw do |map|
match 'build_info' => 'build_info#index'
...etc...

This is somewhat silly, but it’s simple enough to use as an example here. Now if I run my app I’ll get this fascinating page:

Next let’s “record” this sample code by using generate_from_diff to create a new Rails generator for it. First, we need to install generate_from_diff:

$ gem install generate_from_diff
Successfully installed generate_from_diff-0.0.1
1 gem installed
Installing ri documentation for generate_from_diff-0.0.1...
Installing RDoc documentation for generate_from_diff-0.0.1...

Next, let’s commit my new controller and routes.rb code changes:

$ git add .

$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#        new file:   app/controllers/build_info_controller.rb
#        new file:   app/helpers/build_info_helper.rb
#        modified:   config/routes.rb
#        new file:   test/functional/build_info_controller_test.rb
#        new file:   test/unit/helpers/build_info_helper_test.rb
#

$ git commit -m"Build info"
Created commit 037ca3b: Build info
 5 files changed, 22 insertions(+), 0 deletions(-)
 create mode 100644 app/controllers/build_info_controller.rb
 create mode 100644 app/helpers/build_info_helper.rb
 create mode 100644 test/functional/build_info_controller_test.rb
 create mode 100644 test/unit/helpers/build_info_helper_test.rb

One last detail: we need to edit the Gemfile to load generate_from_diff into this application:

source 'http://rubygems.org'
gem 'generate_from_diff'
gem 'rails', '3.0.0.beta4'
etc...

Finally we create our new generator by just running this command:

$ rails generate generator_from_diff build_info HEAD~1 HEAD
    create  lib/generators/build_info
    create  lib/generators/build_info/build_info_generator.rb
    create  lib/generators/build_info/USAGE
       run  git diff --no-prefix HEAD~1 HEAD from "."

Ok - what happened here was that I ran a generator called “generator_from_diff” that is located inside the generate_from_diff gem. I provided it with the name of the new generator I want to create: “build_info” in this example. This is similar to how the Rails 3 “generator generator” works: it generates a generator. But next I provide two Git revisions, in this example “HEAD~1” and “HEAD,” the first and second revisions in my Git repo. The first value is the baseline revision: what to compare to. In this example, this is my new, empty Rails application. The second revision is what code to record and save into the new generator, in this example this revision contains all of my controller and routes.rb changes.

Example: playing back code using a generator

Now let’s see if we can use this new Rails generator to copy the build_info controller and route into a different Rails app. First, let’s create a second, new Rails application:

$ cd ..
$ rails new second_app
    create  
    create  README
    create  Rakefile
    create  config.ru
...etc...
$ cd second_app

And next, let’s copy the new generator we just created in the first app, over to this new app:

$ mkdir lib/generators
$ cp -r ../first_app/lib/generators/build_info lib/generators

And now we can just run our new generator to playback the code changes that I recorded above:

$ rails generate build_info
      gsub  lib/generators/build_info/build_info.patch
       run  patch -p0 < /Users/pat/.../second_app/lib/generators/build_info/build_info.patch from "."
  patching file app/controllers/build_info_controller.rb
  patching file app/helpers/build_info_helper.rb
  patching file config/routes.rb
  patching file test/functional/build_info_controller_test.rb
  patching file test/unit/helpers/build_info_helper_test.rb

That’s it! Now I can run the second app and see the same build status page that we had before:

How does this actually work?

Here’s what is going on under the hood. First, when you record your code changes into the new generator like this:

$ rails generate generator_from_diff build_info HEAD~1 HEAD

... the “generator_from_diff” code actually runs the “git diff” command like this:

$ git diff HEAD~1 HEAD
diff --git a/app/controllers/build_info_controller.rb
           b/app/controllers/build_info_controller.rb
new file mode 100644
index 0000000..c44d83e
--- /dev/null
+++ b/app/controllers/build_info_controller.rb
@@ -0,0 +1,5 @@
+class BuildInfoController < ApplicationController
+  def index
...etc...

This produces a list of all the text changes that were made from one revision (HEAD~1) to another (HEAD). These are then saved into a file called “build_info.patch,” saved inside the new generator.

Later, the text differences, the “patch,” are applied to whatever new or existing files are found relative to the current directory when you run the generator. This copies the new controller file as well as the new route inside of routes.rb into the other application. The patch file is applied using this command:

patch -p0 < lib/generators/build_info/build_info.patch

I use patch instead of git apply to avoid the need to match revision id’s; these will be different from one repo to another.

Ok sounds interesting - so where are you going with this next?

I think it’s cool to be able to “record” Rails generators without writing any code. If this seems like a useful idea, then I’ll spend some more time to clean it up and make it more robust. For example, I’m thinking of adding some code to warn you before the patch is run if there are unexpected files present, or if some expected files are missing.

Next, I’m considering enhancing the gem to perform search/replace using arguments or options that you specify when recording the generator. For example, suppose you recorded a series of code changes that had to do with a model called “Person.” But imagine that you want to be able to playback those code changes in a target application that might have a different model name, “User” instead of “Person” for example. Then the gem could search/replace on the patch file, both when it’s recorded and again when it’s played back, to cause the generated code to use User instead of Person.