Working with J2EE applications is something like wandering in a jungle: you never quite know what wild animal you’ll find around the next corner… it might be an ORM tool like Hibernate; it could be an application framework like Spring or Struts with lots of confusing XML files, or it might just be a long list of obscure JAR files that you need to find and download… whatever it is, you’re guaranteed to spend countless hours wasting time learning things you really didn’t want to know. This article will show how you can get your J2EE application under control by using JRuby and RSpec… but first, let’s take a quick look at what RSpec is and how it’s normally used with Ruby.
Using RSpec with Ruby
If you are a Ruby developer, you would probably write code similar to this to add up an array of numbers:
class Calculator
def self.sum numbers
numbers.inject do |sum, x|
sum+x
end
end
end
This is simple enough to run:
$ irb
irb(main):001:0> require 'calculator.rb'
=> true
irb(main):002:0> Calculator.sum [2, 3]
=> 5
One of the best things about Ruby are all of the different testing tools available to you – for example, you could write a test for this method using RSpec like this:
require 'calculator.rb'
describe "Calculator" do
it "should add numbers correctly" do
Calculator.sum([1, 2]).should == 3
Calculator.sum([2, 2]).should == 4
Calculator.sum([2, 3, 4]).should == 9
end
end
And then run the spec as follows:
$ spec calculator_spec.rb
.
Finished in 0.001703 seconds
1 example, 0 failures
RSpec allows you to write test code that is readable, and also behavior-oriented; that is, the tests reflect the way an end user might actually behave. In fact, RSpec is really just the first step towards behavior-driven-development. The Ruby community also benefits from other tools such as WebRat, Cucumber, etc., that can make testing very easy and effective.
A J2EE sample app
Let’s rewrite the “sum” Ruby method above using Java. To make this feel more like an actual J2EE application, we’ll use a service class that will perform the actual sum operation for us, and set it up with the Spring framework. We can start by writing an XML file called “ApplicationContext.xml” for Spring to use, and declare a bean called “calculatorService:”
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN"
"http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
<bean id="calculatorService"
class="calculator.CalculatorServiceImpl">
</bean>
</beans>
In a real J2EE app, we would probably have an interface called “CalculatorService” like this:
package calculator;
public interface CalculatorService {
public int sum(int[] array);
}
And then we’d implement it using a concrete class, like this:
package calculator;
public class CalculatorServiceImpl implements CalculatorService {
public int sum(int[] array)
{
int sum = 0;
for (int number: array)
{
sum += number;
}
return sum;
}
}
And finally, let’s write a simple Java command line client for this so we can test running it from the command line:
package calculator;
import org.springframework.context.support.ClassPathXmlApplicationContext;
public class CalculatorApp {
public static void main(String[] args) throws Exception {
int[] array = { 2, 3 };
ClassPathXmlApplicationContext application_context =
new ClassPathXmlApplicationContext("ApplicationContext.xml");
CalculatorService calculator =
(CalculatorService)application_context.getBean("calculatorService");
System.out.println("2 + 3 is: " + Integer.toString(calculator.sum(array)));
}
}
This will also help us figure out how to write the ruby spec later. If you run this from Eclipse or your favorite Java IDE, you’ll get:
Jun 25, 2009 12:21:58 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@3479e304: display name [org.springframework.context.support.ClassPathXmlApplicationContext@3479e304]; startup date [Thu Jun 25 12:21:58 EDT 2009]; root of context hierarchy
Jun 25, 2009 12:21:58 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [ApplicationContext.xml]
Jun 25, 2009 12:21:58 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.context.support.ClassPathXmlApplicationContext@3479e304]: org.springframework.beans.factory.support.DefaultListableBeanFactory@604788d5
Jun 25, 2009 12:21:58 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@604788d5: defining beans [calculatorService]; root of factory hierarchy
2 + 3 is: 5
This is starting to feel like an actual “Enterprise” J2EE application now that we have an XML config file and lots of confusing information being logged!
Running RSpec with JRuby
Now that we have a J2EE application to test, let’s get started with JRuby. First, you will need to download and install JRuby. This is really just a matter of downloading the TAR file and then placing the JRuby bin folder on your path. Test that you have it setup properly by running this command:
$ jruby --version
jruby 1.1.5 (ruby 1.8.6 patchlevel 114) (2008-11-03 rev 7996) [x86_64-java]
Now let’s update our Ruby spec from above and get it to work with Java. Here’s what we had for Ruby:
require 'calculator.rb'
describe "Calculator" do
it "should add numbers correctly" do
Calculator.add([1, 2]).should == 3
Calculator.add([2, 2]).should == 4
Calculator.add([2, 3, 4]).should == 9
end
end
The key to using JRuby to test Java is to add this line at the top of the spec:
require 'java'
This tells JRuby that we want to allow the Ruby code to call Java directly. But what code should we try to call? The nice thing about the Spring framework is that is makes a series of “beans,” i.e. Java objects, available to us. The reason I went to the trouble of adding the Spring framework to this sample app is that for a real J2EE application using Spring to create and load a Java object will be the simplest path towards testing your target application. What I did while testing a real J2EE application was to:
- Identify what business logic I wanted to test
- Look for the Java object that was the top-level, simplest interface to that business logic under the user interface layer
- Find a bean in the ApplicationContext.xml file that corresponded to that Java object. In my case, there actually was no such bean, and I had to slightly modify the application I was testing by adding a new <bean> tag to one of the Spring XML files.
If your application is not using Spring, you might simply be able to create a Java object directly from your Ruby code, or in the worst case scenario you might have to make Java code changes to the target J2EE app to break dependencies that the object you’d like to test has on other objects, interfaces, services, etc., allowing you to create it in isolation. Michael Feathers has written an entire book on dependency breaking techniques.
Back to this sample app: here in our Ruby spec we can follow the same pattern that I used above in the command line Java client: calling “ClassPathXmlApplicationContext” to get the application context, and then creating a bean with getBean. The other thing we need to add are “include_class” directives that indicate to JRuby which Java classes should be loaded from the classpath and made available to the Ruby script. Here’s the spec code with the new JRuby code changes in bold:
require 'java'
include_class 'org.springframework.context.ApplicationContext'
include_class
'org.springframework.context.support.ClassPathXmlApplicationContext'
describe "Calculator" do
it "should add numbers correctly" do
application_context =
ClassPathXmlApplicationContext.new "ApplicationContext.xml"
calculator = application_context.getBean "calculatorService"
calculator.sum([1, 2]).should == 3
calculator.sum([2, 2]).should == 4
calculator.sum([2, 3, 4]).should == 9
end
end
Now let’s try to run it using JRuby. Another trick with JRuby is knowing how to use the “-S” option – this allows you to run a Ruby command like “gem” or “spec,” but inside a JRuby session. So here’s how to run our new spec using JRuby:
$ jruby -S spec calculator_spec.rb
(eval):1:in `include_class': cannot load Java class
org.springframework.context.ApplicationContext (NameError)
from /Users/pat/src/jruby-1.1.5/lib/ruby/site_ruby/1.8/builtin/javasupport/core_ext/object.rb:38:in `eval'
from /Users/pat/src/jruby-1.1.5/lib/ruby/site_ruby/1.8/builtin/javasupport/core_ext/object.rb:67:in `include_class'
from /Users/pat/src/jruby-1.1.5/lib/ruby/site_ruby/1.8/builtin/javasupport/core_ext/object.rb:38:in `each'
from /Users/pat/src/jruby-1.1.5/lib/ruby/site_ruby/1.8/builtin/javasupport/core_ext/object.rb:38:in `include_class'
from calculator_spec.rb:2
So what is the error message all about? This just means that JRuby wasn’t able to find the ApplicationContext class from Spring on the classpath. But what is the classpath anyway? It would be nice to be able to simply specify the classpath using a command line option, the way you do with a java command line using “-cp” for example. But for JRuby you need to specify the classpath as an environment setting. To make this easier, I wrote a simple shell script for running a JRuby spec:
BASE=`pwd`
CLASSPATH=$BASE/lib/spring-2.5.1.jar
export CLASSPATH
jruby -S spec $1
This just gets the current working directory and constructs the classpath setting, indicating where to find the Spring JAR file. Finally we export the classpath value and call JRuby. This will work on the Mac and Linux; for Windows you would need something slightly different. Anyway, now I can run my specs like this:
$ ./jruby-spec.sh calculator_spec.rb
F
1)
NativeException in 'Calculator should add numbers correctly'
java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
org/springframework/util/ClassUtils.java:73:in `<clinit>'
org/springframework/core/io/DefaultResourceLoader.java:52:in `<init>'
org/springframework/context/support/AbstractApplicationContext.java:198:in `<init>'
org/springframework/context/support/AbstractRefreshableApplicationContext.java:80:in `<init>'
org/springframework/context/support/AbstractXmlApplicationContext.java:58:in `<init>'
org/springframework/context/support/ClassPathXmlApplicationContext.java:119:in `<init>'
org/springframework/context/support/ClassPathXmlApplicationContext.java:66:in `<init>'
sun/reflect/NativeConstructorAccessorImpl.java:-2:in `newInstance0'
sun/reflect/NativeConstructorAccessorImpl.java:39:in `newInstance'
sun/reflect/DelegatingConstructorAccessorImpl.java:27:in `newInstance'
java/lang/reflect/Constructor.java:513:in `newInstance'
org/jruby/javasupport/JavaConstructor.java:226:in `new_instance'
org/jruby/java/invokers/ConstructorInvoker.java:100:in `call'
org/jruby/java/invokers/ConstructorInvoker.java:180:in `call'
etc...
Oops – it turns out that Spring actually requires the Apache Commons logging JAR file to be on the classpath also. This sample app is definitely reminding me of a complex J2EE app running on WebSphere or WebLogic! Let’s add commons-logging-1.0.4.jar to the classpath in my shell script and try again:
$ ./jruby-spec.sh calculator_spec.rb
Jun 25, 2009 1:58:15 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@4c6c3e: display name [org.springframework.context.support.ClassPathXmlApplicationContext@4c6c3e]; startup date [Thu Jun 25 13:58:15 EDT 2009]; root of context hierarchy
Jun 25, 2009 1:58:15 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [ApplicationContext.xml]
F
1)
NativeException in 'Calculator should add numbers correctly'
org.springframework.beans.factory.BeanDefinitionStoreException:
IOException parsing XML document from class path resource
[ApplicationContext.xml]; nested exception is java.io.FileNotFoundException:
class path resource [ApplicationContext.xml] cannot be opened because
it does not exist
org/springframework/beans/factory/xml/XmlBeanDefinitionReader.java:334:in `loadBeanDefinitions'
org/springframework/beans/factory/xml/XmlBeanDefinitionReader.java:295:in `loadBeanDefinitions'
Once again, I’ve forgotten something in my class path: this time it’s the ApplicationContext.xml file which I passed into ClassPathXmlApplicationContext in the spec. As the name implies, I need to put the XML file on the classpath in order for Spring to be able to find it. Let’s cut to the chase and put everything I need onto the classpath… here’s the final version of jruby-spec.sh (the classpath line split into two for readability):
BASE=`pwd`
CLASSPATH=$BASE/bin:$BASE/resources:$BASE/lib/spring-2.5.1.jar:
$BASE/lib/commons-logging-1.0.4.jar
export CLASSPATH
jruby -S spec $1
Now JRuby will be able to find everything that it needs: Spring, Apache Common Logging, the ApplicationContext.xml file, and also the application’s class files saved under “bin” by Eclipse. Now if I run the spec once more it should all work:
$ ./jruby-spec.sh calculator_spec.rb
Jun 25, 2009 2:02:22 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@1717d968: display name [org.springframework.context.support.ClassPathXmlApplicationContext@1717d968]; startup date [Thu Jun 25 14:02:22 EDT 2009]; root of context hierarchy
Jun 25, 2009 2:02:22 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [ApplicationContext.xml]
Jun 25, 2009 2:02:22 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.context.support.ClassPathXmlApplicationContext@1717d968]: org.springframework.beans.factory.support.DefaultListableBeanFactory@5a21fdc8
Jun 25, 2009 2:02:22 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@5a21fdc8: defining beans [calculatorService]; root of factory hierarchy
F
1)
TypeError in 'Calculator should add numbers correctly'
for method sum expected [[I]; got: [org.jruby.RubyArray];
error: argument type mismatch calculator_spec.rb:8:
Finished in 0.70559 seconds
1 example, 1 failure
What’s this error all about? I do have the correct classpath now, but what does “RubyArray” mean? I thought I passed an array of integers into the calculator service:
calculator.sum([1, 2]).should == 3
Well it turns out that Ruby and JRuby aren’t quite the same thing. Since JRuby is actually a Java application itself, it implements each Ruby class with a corresponding Java class. For the Ruby Array class, JRuby has created a class called “org.jruby.RubyArray.” When you call Java code and pass in Ruby objects, JRuby actually provides the Java equivalent of these Ruby objects to the Java code. That’s why we get an error here; our calculator service doesn’t expect a RubyArray – it expects int[] instead.
To avoid this error, we need to convert the RubyArray into a normal Java array, using a JRuby method called “to_java()”, like this:
calculator.sum([1, 2].to_java(:int)).should == 3
To_java takes a symbol as a parameter, which indicates what type each element of the Ruby array should be converted into. Now our spec will pass!
$ ./jruby-spec.sh calculator_spec.rb
Jun 25, 2009 2:15:37 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@1717d968: display name [org.springframework.context.support.ClassPathXmlApplicationContext@1717d968]; startup date [Thu Jun 25 14:15:37 EDT 2009]; root of context hierarchy
Jun 25, 2009 2:15:37 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [ApplicationContext.xml]
Jun 25, 2009 2:15:37 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.context.support.ClassPathXmlApplicationContext@1717d968]: org.springframework.beans.factory.support.DefaultListableBeanFactory@5a21fdc8
Jun 25, 2009 2:15:37 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@5a21fdc8: defining beans [calculatorService]; root of factory hierarchy
.
Finished in 0.28237 seconds
1 example, 0 failures
Taking a step back, I think what we’ve done is quite interesting; we have:
- Pulled a J2EE application out of Eclipse, IntelliJ or whatever IDE you project uses, and started to run it from the command line instead. For me, avoiding a confusing and complex IDE makes the application easier to work with… more like a Rails app.
- Documented what the classpath needs to be and what JAR files the application requires in an understandable text file format, rather than having it hidden away in a confusing Eclipse dialog box.
- Exposed the J2EE application’s business logic to testing below the user interface layer – nothing here involves a web browser or user interface testing tool like Selenium.
- Started to apply behavior driven development to a J2EE application, using the best tools available, which happen to be written in Ruby.
In a future post, I’ll try to take the next step of using Cucumber features to test this same J2EE sample app. It will be interesting to find out if Cucumber works will with JRuby.
Tags:java·jruby·rails
I just updated my fork of the auto_complete plugin to support Rails 2.3 nested attributes. Thanks to Anthony Frustagli for the code and ideas that I used as the basis for this fix.
Basic usage:
To use auto_complete on a complex form with nested attributes, just call “text_field_for_auto_complete” right on the form builder object yielded by form_for or fields_for, like in this example:
<% parent_form.fields_for :children do |child_form| %>
<%= child_form.text_field_with_auto_complete :name, {},
{ :method => :get, :skip_style => true } %>
<% end %>
If you have Rails 2.3, this code will iterate over each child object and display a text field with auto complete support. My plugin will generate HTML and Javascript that works even when repeated in a loop like this. Also note that I’ve left off the object name parameter from text_field_with_auto_complete. It’s not needed now, since the object is indicated by the surrounding call to fields_for. The other parameters are optional and are taken unchanged from the original auto_complete plugin:
- “:method => :get” indicates GET requests should be used by the AJAX calls to load the pick list values, avoiding problems with CSRF protection.
- And “:skip => :true” indicates that the inline CSS stylesheet used by the auto complete drop down Prototype code should be skipped. Since we’re iterating over child objects we don’t want the same CSS code repeated once for each; instead include it once in a parent object’s call to text_field_for_auto_complete or else just include it manually somewhere.
That’s it – it should just work. If you’re interested in learning more about how to use nested attributes and what my plugin is actually doing, read on…
Details:
To learn more, let’s take a look at a simple nested attribute example, using the Projects/Tasks models from Ryan Bates' complex forms screen cast:
class Project < ActiveRecord::Base
has_many :tasks
accepts_nested_attributes_for :tasks, :allow_destroy => true
end
class Task < ActiveRecord::Base
belongs_to :project
end
A project has many tasks, and each task belongs to a project. Here I’ve also declared that each project “accepts nested attributes for” tasks. This is a new method added to ActiveRecord in Rails 2.3… for lots of examples and explanation just take a look directly at the new nested_attributes.rb code file in Rails 2.3. In a nutshell, “accepts_nested_attributes_for” tells ActiveRecord that the project model should be able to save the attributes of the associated task model objects when a project is saved. This means that when I submit my project form, it can also contain a series of task fields as well. For example, my view code might look something like this:
<% form_for @project do |project_form| %>
<p>
<%= project_form.label :name, "Project:" %>
<%= project_form.text_field :name %>
</p>
<% project_form.fields_for :tasks do |task_form| %>
<%= task_form.label :name, "Task:" %>
<%= task_form.text_field :name %>
<% end %>
<% end %>
This displays a name text field for the project, and then calls “fields_for” again right on the form builder yielded by form_for. This is new for Rails 2.3. In earlier versions of Rails you had to explicitly iterate over the child objects and call fields_for for each one. Now in Rails 2.3, you can call fields_for as a method of the parent form and it will automatically iterate over all of the child objects and call fields_for. If we take a look at the HTML generated by this example form, we’ll find something like:
<input id="project_name" name="project[name]"
size="30" type="text" value="Some project" />
<input id="project_tasks_attributes_0_id"
name="project[tasks_attributes][0][id]" type="hidden" value="1" />
<input id="project_tasks_attributes_0_name"
name="project[tasks_attributes][0][name]" type="text" value="Task one" />
<input id="project_tasks_attributes_1_id"
name="project[tasks_attributes][1][id]" type="hidden" value="2" />
<input id="project_tasks_attributes_1_name"
name="project[tasks_attributes][1][name]" type="text" value="Task two" />
I’ve simplified this to make it more readable. You can see the iteration by project_form.fields_for :tasks, and that for each task there’s an <input> tag for the “name” field, along with another hidden <input> tag containing the task’s “id” attribute. The most important detail here is the name given to each of these tags: “project[tasks_attributes][0][name]” for example. Since the tasks are nested attributes of the project, they are displayed using the PARENT_OBJECT[CHILD_OBJECTS_attributes][INDEX][FIELD] pattern, while for the project we get the simple OBJECT[FIELD] pattern. This is the key to making nested attributes work. In our project model, when we called “has_many :tasks”, Rails defined some new methods for us on the Project class to handle tasks: tasks, tasks=, task_ids, task_ids= and a couple of others as well. Now with Rails 2.3, when we call “accepts_nested_attributes_for :tasks” Rails defines another new method for Project called tasks_attributes= in order to process all of the new nested parameters for tasks when the complex project form is submitted. This is the reason for the “_attributes” in the naming pattern used in the form.
Now… how do we get auto complete to work for this form? The problem with auto complete on a complex form has always been that the Javascript and HMTL used by the Prototype library assumes that the <input> tag, <div> tag and related Javascript code would be unique on the HTML page. If you just call the text_field_with_auto_complete macro from the standard auto_complete plugin like this…
<% project_form.fields_for :tasks do |task_form| %>
<%= text_field_with_auto_complete :task, :name, {},
{ :method => :get, :skip_style => true } %>
<% end %>
… it will not work. The first problem is that text_field_with_auto_complete does not know that fields_for is iterating over the child tasks, or which task is currently being processed in the iteration. But even if you were able to identify the current task object somehow, you would still get HTML like this:
<input id="task_name" name="task[name]" size="30" type="text" />
<div class="auto_complete" id="task_name_auto_complete"></div>
<script type="text/javascript">
//<![CDATA[
var task_name_auto_completer = new Ajax.Autocompleter('task_name',
'task_name_auto_complete', '/projects/auto_complete_for_task_name',
{method:'get'})
//]]>
</script>
…
<input id="task_name" name="task[name]" size="30" type="text" />
<div class="auto_complete" id="task_name_auto_complete"></div>
<script type="text/javascript">
//<![CDATA[
var task_name_auto_completer = new Ajax.Autocompleter('task_name',
'task_name_auto_complete', '/projects/auto_complete_for_task_name',
{method:'get'})
//]]>
</script>
Now the <input id=“task_name”> tag is repeated on the same page, and the Javascript call to Ajax.Autocompleter('task_name', … ) will not work since the browser will not be able to identify which <input> tag to use.
If you use my plugin instead of the original auto_complete plugin…
$ rm -rf vendor/plugins/auto_complete
$ ./script/plugin install git://github.com/patshaughnessy/auto_complete.git
Initialized empty Git repository in /Users/pat/rails-app/vendor/plugins/auto_complete/.git/
remote: Counting objects: 20, done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 20 (delta 5), reused 0 (delta 0)
Unpacking objects: 100% (20/20), done.
From git://github.com/patshaughnessy/auto_complete
* branch HEAD -> FETCH_HEAD
… and restart your Rails app, then you can change your view to call text_field_with_auto_complete as a method of the form builder, like this:
<% project_form.fields_for :tasks do |task_form| %>
<%= task_form.text_field_with_auto_complete :name, {},
{ :method => :get, :skip_style => true } %>
<% end %>
Note that I’ve also dropped :task as a parameter since that’s implicit in the call to fields_for. In fact, since text_field_with_auto_complete is now a method of the FormBuilder object (“task_form”), it has access to the task object currently being processed in the iteration. Now if you refresh the same form you’ll instead get this HTML instead:
<input id="project_tasks_attributes_0_name"
name="project[tasks_attributes][0][name]"
size="30" type="text" value="Task one" />
<div class="auto_complete" id="project_tasks_attributes_0_name_auto_complete">
</div><script type="text/javascript">
//<![CDATA[
var project_tasks_attributes_0_name_auto_completer =
new Ajax.Autocompleter('project_tasks_attributes_0_name',
'project_tasks_attributes_0_name_auto_complete',
'/projects/auto_complete_for_task_name',
{method:'get', paramName:'task[name]'})
//]]>
</script>
…
<input id="project_tasks_attributes_1_name"
name="project[tasks_attributes][1][name]"
size="30" type="text" value="Task two" />
<div class="auto_complete" id="project_tasks_attributes_1_name_auto_complete">
</div><script type="text/javascript">
//<![CDATA[
var project_tasks_attributes_1_name_auto_completer =
new Ajax.Autocompleter('project_tasks_attributes_1_name',
'project_tasks_attributes_1_name_auto_complete',
'/projects/auto_complete_for_task_name',
{method:'get', paramName:'task[name]'})
//]]>
</script>
This looks much better, and will actually work for the following reasons:
- The <input> tags have the correct names, using the PARENT_OBJECT[CHILD_OBJECTS_attributes][INDEX][FIELD] pattern from fields_for. This means that the field values will be processed properly by ActiveRecord when the form is submitted.
- My changes to the auto_complete plugin have picked up the child object index, 0 and 1 in this example, and included it in the <input> tag’s id, the <div> tag id and well as the associated Javascript code that calls Ajax.Autocompleter. Since all of the tag id’s are unique, the auto complete behavior works properly again for each text field.
- The original “task” class name and “name” field name are passed unchanged into the Ajax calls to the server. This means that in your controller you can continue to use “auto_complete_for :task, :name” as usual, without worrying about the complex form and the fact that the task fields are repeated multiple times, etc.:
Ajax.Autocompleter('project_tasks_attributes_1_name',
'project_tasks_attributes_1_name_auto_complete',
'/projects/auto_complete_for_task_name',
{method:'get', paramName:'task[name]'})
Here the third parameter to Ajax.Autocompleter, "/projects/auto_complete_for_task_name", is the AJAX URL which you need to account for in routes.rb, and paramName:'task[name]' tells the auto_complete_for handler in your controller to get the task names as usual, and protects the server side code from all of the complexity around the tag id, names, child object index, etc.
Tags:
I’ve forked the auto_complete plugin to support repeated text fields in a complex form; see http://patshaughnessy.net/repeated_auto_complete for more details.
If you had downloaded my plugin in the past, I’ve just made a couple of changes that will require some simple code changes to your app:
- You no longer need to or are able to use “auto_complete_form_for” or “auto_complete_fields_for.” I decided this was confusing and unnecessary. Now my plugin just mixes the text_field_with_auto_complete method right into the standard FormBuilder class. Just use form_for or fields_for as usual.
- I also dropped the object name parameter from text_field_with_auto_complete. Since text_field_with_auto_complete is a method of the form builder, the target object is indicated by the surrounding call to fields_for or form_for and so doesn’t need to be repeated. Now using form.text_field_with_auto_complete is very similar to using form.text_field or the other form builder methods: you just need to specify the column/field name.
So if you are using my old plugin with a Rails 2.2 or earlier app like this:
<% for person in @group.people %>
<% auto_complete_fields_for "group[person_attributes][]", person do |form| %>
Person <%= person_form.label :name %><br />
<%= form.text_field_with_auto_complete :person, :name, {},
{:method => :get} %>
<% end %>
<% end %>
… you should drop “auto_complete_” and “:person” and just use code like this instead:
<% for person in @group.people %>
<% fields_for "group[person_attributes][]", person do |form| %>
Person <%= person_form.label :name %><br />
<%= form.text_field_with_auto_complete :name, {},
{:method => :get} %>
<% end %>
<% end %>
And if you have Rails 2.3 or later and are using nested attributes, this would become:
<% form_for @group do |group_form| -%>
<% group_form.fields_for :people do |person_form| %>
Person <%= person_form.label :name %><br />
<%= person_form.text_field_with_auto_complete :name, {},
{ :method => :get, :skip_style => true } %>
<% end %>
<% end %>
Tags:
In part 1 of this series, I showed how to create a simple Rails web site that uses the Paperclip plugin from Thoughtbot to upload and display image files. Then in part 2, I went on to change the sample app to download the image files through a Rails controller and not just through a direct call to Apache. This would be useful if you wanted to implement security for file attachments or for a variety of other reasons.
This time I’d like to show how to modify the same sample application to save the file attachments in a database BLOB column, instead of on your web server’s file system. To jump ahead and just get the working code, look at the “part3” folder in the github repo: http://github.com/patshaughnessy/paperclip-sample-app.
But before we actually work on the sample app, a disclaimer: Don’t try this at home! Serving file content directly from the file system via Apache or some other web server will always be faster and simpler than loading the file attachments from a database table… Apache and other web servers were designed to load and serve files quickly, and there’s normally no need to issue an expensive SQL query or to make another network connection to a database server just to send files to a web browser.
So why in the world would you ever want to pay the extra performance penalty and move the files into a database table? Here are a couple of reasons:
- Your client or employer wants you to. Some companies insist on using a commercial, “enterprise” RDMS system to save file contents for one reason or another. My employer, for example, has many years of experience using Oracle and is comfortable managing large numbers of files that are stored in an Oracle table, while the thought of managing, replicating, backing up, etc., files that are simply saved on a Linux file system seems much more complex and unfamiliar.
- Security. If you need to encrypt the contents of super-secret file attachments, storing them in a database might be an easier solution if you’re willing to spend money on a database server like Oracle. And database encryption aside, generally information in a database server can be more easily protected and audited than files on your web server's file system can be.
Anyway, let’s move on and actually change the sample app to save the files in a BLOB column. The first thing we will need to do is to use the version of Paperclip that I modified; the actual Paperclip plugin from Thoughtbot does not support storing files in a BLOB column. I added some code to Paperclip – a new “storage module” – to make this possible. See my code changes to learn more.
So let’s delete the original plugin and install my version:
$ cd /path/to/paperclip-sample-app
$ rm -rf vendor/plugins/paperclip
$ ./script/plugin install git://github.com/patshaughnessy/paperclip.git
Now that we have the modified plugin installed, let’s go ahead and create the BLOB columns that we will use to save the files. I tried to design the database storage module to be easy to use; one of the decisions I made was around what these BLOB columns should be called. I decided by default to use “[attachment]_file” as the name for the primary file attachment, and “[attachment]_[style]_file” for the other styles. If you want to use other column names, you just need to specify the names in the call to “has_attached_file” in the model. See my usage post for more info.
For this sample app I’ll go ahead and use the default column names: “avatar_file,” “avatar_small_file” and “avatar_thumb_file.” Here’s how to create those columns for a MySQL database. First create a new migration as usual:
$ ./script/generate migration add_attachments_blob_avatar_to_user
exists db/migrate
create db/migrate/20090528173400_add_attachments_blob_avatar_to_user.rb
… and then edit the new migration file and add the bolded code to it:
class AddAttachmentsBlobAvatarToUser < ActiveRecord::Migration
def self.up
execute 'ALTER TABLE users ADD COLUMN avatar_file LONGBLOB'
execute 'ALTER TABLE users ADD COLUMN avatar_small_file LONGBLOB'
execute 'ALTER TABLE users ADD COLUMN avatar_thumb_file LONGBLOB'
end
def self.down
remove_column :users, :avatar_file
remove_column :users, :avatar_small_file
remove_column :users, :avatar_thumb_file
end
end
Normally to create a BLOB column you would use “add_column :users, :avatar_file, :binary.” This would work fine for Oracle and other database servers. MySQL, however, supports four different types of BLOBs: TINYBLOB (256 bytes), BLOB (64k bytes), MEDIUMBLOB (16MB) and LONGBLOB (4GB). (See http://dev.mysql.com/doc/refman/5.0/en/storage-requirements.html for more information.) If we used the Rails migrations :binary column type, then we would get normal BLOBs and have an upper limit for the file attachment size of 64k, which is not normally enough. Unfortunately, there’s no way to specify LONGBLOB for MySQL using Rails migrations, so you have to use the “execute” migration and write an actual SQL statement to add the column.
Next go ahead and run your migrations and create the new columns:
$ rake db:migrate
(in /path/to/paperclip-sample-app)
== AddAttachmentsBlobAvatarToUser: migrating =================================
-- execute("ALTER TABLE users ADD COLUMN avatar_file LONGBLOB")
-> 0.0585s
-- execute("ALTER TABLE users ADD COLUMN avatar_small_file LONGBLOB")
-> 0.0266s
-- execute("ALTER TABLE users ADD COLUMN avatar_thumb_file LONGBLOB")
-> 0.0116s
== AddAttachmentsBlobAvatarToUser: migrated (0.0973s) ========================
Now we just need to tell Paperclip to use database storage instead of the default file system storage and we’re ready to try our app and see if it saves the files into the new BLOB columns. Add the bolded parameter to has_attached_file in the User model:
class User < ActiveRecord::Base
has_attached_file :avatar,
:storage => :database,
:styles => { :thumb => "75x75>", :small => "150x150>" },
:url => '/:class/:id/:attachment?style=:style'
end
Note that I also removed the “:path” parameter; this value would be ignored by the database storage module anyway since the files will be stored in the DB. Let’s try it out! Start up the sample app, and re-edit a user record to upload a new image file:
Select a file, click “Update” to submit the form and the file will be processed by ImageMagick, and saved into our new BLOB columns by the database storage module in Paperclip…
What? Where’s the image? It turns out that we still need to make a code change to the UsersController to download the image file from the BLOB column instead of from the file system. I’ll get to this in a moment. But first, let’s look at the console and see if the new files were saved into the database properly:
$ ./script/console
Loading development environment (Rails 2.3.2)
>> User.first
=> #<User id: 1, name: "Mickey Mouse", email: "mickey@disney.com",
created_at: "2009-05-28 17:27:00", updated_at: "2009-05-28 17:37:42",
avatar_file_name: "mickey-mouse.jpg", avatar_content_type: "image/jpeg",
avatar_file_size: 137233, avatar_updated_at: "2009-05-28 17:37:42",
avatar_file: "\377???JFIF\000\001\002\001\001h\001h\000\000\377?\021\b\002\210\001?\001\021\000\002\021\001\003\021\001\377?\204\000\001\001\001\001\001\001\001...",
avatar_small_file: "\377???JFIF\000\001\001\001\001h\001h\000\000\377?C\000\003\002\002\002\002\002\003\002\002\002\003\003\003\003\004\006\004\004\004\004\004\b\006\006\005\006...",
avatar_thumb_file: "\377???JFIF\000\001\001\001\001h\001h\000\000\377?C\000\003\002\002\002\002\002\003\002\002\002\003\003\003\003\004\006\004\004\004\004\004\b\006\006\005\006...">
Here you can see the three new BLOB columns, avatar_file, avatar_small_file and avatar_medium_file, and the first few bytes of each column’s value. In fact, if you’re a real geek you’ll notice that the first few bytes contain “JFIF,” which is probably the image file type specification (not sure)… so we know we are seeing the binary contents of the three versions of the Mickey image here. Great!
Actually, not so great: there’s a subtle problem here we need to worry about. I actually typed in a very simple ActiveRecord command, “User.first,” and it loaded all three of the image files’ contents into memory just so that I could inspect the value of the user record in the console. This was convenient now, since I was actually interested in knowing whether or not each of the files was saved properly in the DB. However, this is potentially a big performance problem in general. Imagine if the files were very large… it could take a long time for all 3 of the files to be fetched by a SQL query and returned to the Rails ActiveRecord object. Do I really want or need this to happen every time I access a User record? Usually when I load a User record it’s just because I need to check the value of one of the metadata columns, like the user’s name, email address or something else. And if I do need one of the files, why should I have to load all three files? Spending the time required to load the file contents for each image style is a big performance penalty that isn’t usually necessary.
The solution I came up for this problem was to enable Paperclip to add a method to your model class called “select_without_file_columns_for” that you can use as a named scope or, even better, a default scope. It returns a :select scope hash that will exclude the BLOB columns from the SQL query that ActiveRecord issues to load each record. If you’re using Rails 2.3 or higher, you can use select_without_file_columns_for as a default scope in your model like this:
class User < ActiveRecord::Base
has_attached_file :avatar, :storage => :database,
:styles => { :thumb => "75x75>", :small => "150x150>" },
:url => '/:class/:id/:attachment?style=:style'
default_scope select_without_file_columns_for(:avatar)
end
To learn more about default scopes and to see a couple of other examples, read this: http://ryandaigle.com/articles/2008/11/18/what-s-new-in-edge-rails-default-scoping … or http://m.onkey.org/2009/3/24/default-scopes-and-inheritance-to-the-rescue. Let’s see how this works in the console again after adding the default scope:
$ ./script/console
Loading development environment (Rails 2.3.2)
>> User.first
=> #<User id: 1, name: "Mickey Mouse", email: "mickey@disney.com",
created_at: "2009-05-28 17:27:00", updated_at: "2009-05-28 17:37:42",
avatar_file_name: "mickey-mouse.jpg", avatar_content_type: "image/jpeg",
avatar_file_size: 137233, avatar_updated_at: "2009-05-28 17:37:42">
Cool. Now the BLOB columns are not displayed. “Default scope” refers to the fact that the SQL used by ActiveRecord by default to load records is automatically changed. I didn’t have to pay the price of loading each of the files, and I was still able to load all of the other User columns using ActiveRecord the way I normally would. To see what happened, execute the “select_without_file_columns_for” method directly in the console:
>> User.select_without_file_columns_for :avatar
=> {:select=>"id,name,email,created_at,updated_at,avatar_file_name,
avatar_content_type,avatar_file_size,avatar_updated_at"}
It‘s a lot more common to use a default or named scope with :conditions (to modify the WHERE clause) or :order (to modify the ORDER BY clause) but in this case I’ve used :select to specify which columns should be loaded by ActiveRecord (the SELECT portion of the SQL). If you look at the hash, you’ll see all of the User columns listed, except for avatar_file, avatar_small_file and avatar_thumb_file.
If you’re using Rails 2.2 or earlier default scope is not available yet, and you will need to use a named scope, like this:
class User < ActiveRecord::Base
has_attached_file :avatar,
:storage => :database,
:styles => { :thumb => "75x75>", :small => "150x150>" },
:url => '/:class/:id/:attachment?style=:style'
named_scope :without_file_data, select_without_file_columns_for(:avatar)
end
And you will need to use the named scope explicitly to avoid loading the files, like this:
$ ./script/console
Loading development environment (Rails 2.3.2)
>> User.without_file_data.first
=> #<User id: 1, name: "Mickey Mouse", email: "mickey@disney.com",
created_at: "2009-05-30 12:14:03", updated_at: "2009-05-30 12:14:03",
avatar_file_name: "mickey-mouse.jpg", avatar_content_type: "image/jpeg",
avatar_file_size: 137233, avatar_updated_at: "2009-05-30 12:14:03">
A good way to understand exactly what ActiveRecord is doing is to use a Ruby trick and open up the User class and add some debug code to it. Try entering this code into your console:
>> class User < ActiveRecord::Base
>> def self.find_by_sql(sql)
>> puts "DEBUG: #{sql}"
>> super
>> end
>> end
=> nil
What this code does is open up our User model class, and override the “find_by_sql” ActiveRecord method to display the SQL statement before calling the original base class method to execute it. Find_by_sql is the method that the various different ActiveRecord find methods all call once they have constructed a SQL statement. For example, find :first, find :all, first, last… all of these eventually call find_by_sql.
Assuming we have the version of User with the named_scope, we can see what SQL statements are issued with or without the select_without_file_columns_for scope:
>> User.first
DEBUG: SELECT * FROM `users` LIMIT 1
=> #<User id: 1, name: "Mickey Mouse", email: "mickey@disney.com",
created_at: "2009-05-28 17:27:00", updated_at: "2009-05-28 17:37:42",
avatar_file_name: "mickey-mouse.jpg", avatar_content_type: "image/jpeg",
avatar_file_size: 137233, avatar_updated_at: "2009-05-28 17:37:42",
avatar_file: "\377???JFIF\000\001\002\001\001h\001h\000\000\377?\021\b\002\210\001?\001\021\000\002\021\001\003\021\001\377?\204\000\001\001\001\001\001\001\001...",
avatar_small_file: "\377???JFIF\000\001\001\001\001h\001h\000\000\377?C\000\003\002\002\002\002\002\003\002\002\002\003\003\003\003\004\006\004\004\004\004\004\b\006\006\005\006...",
avatar_thumb_file: "\377???JFIF\000\001\001\001\001h\001h\000\000\377?C\000\003\002\002\002\002\002\003\002\002\002\003\003\003\003\004\006\004\004\004\004\004\b\006\006\005\006...">
>> User.without_file_data.first
DEBUG: SELECT id,name,email,created_at,updated_at,avatar_file_name,
avatar_content_type, avatar_file_size,avatar_updated_at FROM `users` LIMIT 1
=> #<User id: 1, name: "Mickey Mouse", email: "mickey@disney.com",
created_at: "2009-05-28 17:27:00", updated_at: "2009-05-28 17:37:42",
avatar_file_name: "mickey-mouse.jpg", avatar_content_type: "image/jpeg",
avatar_file_size: 137233, avatar_updated_at: "2009-05-28 17:37:42">
Here we can see that ActiveRecord will load all of the User columns that are listed above in the hash we pass to named_scope… all of the columns except for the BLOB columns. This is different from what ActiveRecord does by default, which is a simple SELECT * FROM …. statement.
Enough about ActiveRecord internals… let’s get back to the sample app and finish it up:
So why didn’t the image appear here properly? The reason is that in UsersController I’m still using the code that accesses the file on the file system and streams it to the client using send_file (see my last post for more info):
def avatars
user = User.find(params[:id])
style = params[:style] ? params[:style] : 'original'
send_file user.avatar.path(style),
:type => user.avatar_content_type
end
Obviously send_file is no longer going to work for us. Instead we need to use a similar function in ActionController::Streaming called send_data, which takes the binary data directly as a parameter instead of a file. And to access the file’s contents, I’ve added a method called “file_contents” to Paperclip that returns the actual file contents for the given style, or for the original style by default. Here’s how to put it all together; replace the “avatars” method in UsersController with this new version instead:
def avatars
user = User.find(params[:id])
style = params[:style] ? params[:style] : 'original'
send_data user.avatar.file_contents(style),
:type => user.avatar_content_type
end
Only the line in bold has changed. We just call user.avatar.file_contents, pass in the specified style and then pass along the data to send_data.
If you restart the app and refresh your browser, now you should see the image again:
Now we are seeing the binary image file data loaded from the BLOB column by ActiveRecord and streamed down to the browser by send_data.
One last detail about this: since usually everyone will use the same controller code to load the file contents for a given style from the BLOB, and then pass it along to send_data, I enabled Paperclip to add another utility method, this time to your controller, to make this even easier:
class UsersController < ApplicationController
downloads_files_for :user, :avatar
etc…
If you call “downloads_files_for” from your controller like this and specify the model and file attachment, it will automatically generate the correct controller method for you, and call it “avatars” or whatever the plural version of your attachment name is. I chose the name to make it easy and natural to create a route to it in routes.rb. No need to even think about send_data or how to get the file contents from Paperclip at all! Nothing could be simpler. However, if you need to implement security or some other business rules around downloading files, then you might need to add that business logic to the “avatars” method above. Either way, it’s very simple.
Tags:paperclip·rails
Last time I wrote about how to quickly setup a Rails application using scaffolding that allows users to upload image files and then display them using the Paperclip plugin. Paperclip does the simplest thing possible by default: it saves the file attachments right on the file system of your web server, allowing you to download them to users easily using Apache or whatever web server you have installed.
Today I’d like to take that sample app one step further and show how to use a Rails controller to download the files, instead of directly through Apache. To get the finished code just go to http://github.com/patshaughnessy/paperclip-sample-app and look at the “part2” folder.
There are a variety of reasons why you might want to do this, including:
- Security: you don’t want to expose all of the file attachments to all users of your web site. Instead, you want to implement some business rules and show some files to some users, but not to others.
- Auditing/logging: you want to keep track of who is viewing which files, or how many times they are viewing them.
- You want to hide the actual location of the files from users, and instead map the files to URLs in some other pattern or manner.
- Or, you might want not want to store the files on your web server’s file system at all, but instead in a database table or somewhere else. In Part 3 of this series, I’ll show how to do this…
The common thread here is that you want to execute some Ruby code every time a users accesses a file, and the way to do that is by routing the download requests through a controller.
Let’s pick up where we left off last time:
If we take a look at some of the HTML source for this page:
<h1>Listing users</h1>
<table>
<tr>
<th>Photo</th>
<th>Name</th>
<th>Email</th>
</tr>
<tr>
<td><img alt="Mickey-mouse"
src="/system/avatars/1/thumb/mickey-mouse.jpg?1242395876" /></td>
<td>Mickey Mouse</td>
<td>mickey@disney.com</td>
<td><a href="/users/1">Show</a></td>
<td><a href="/users/1/edit">Edit</a></td>
… we can see that Paperclip’s “url” function which we called in index.html.erb is returning a pointer to the actual location of the file on the web server’s hard drive, under the public/system folder:
$ find public/system
public/system
public/system/avatars
public/system/avatars/1
public/system/avatars/1/original
public/system/avatars/1/original/mickey-mouse.jpg
public/system/avatars/1/small
public/system/avatars/1/small/mickey-mouse.jpg
public/system/avatars/1/thumb
public/system/avatars/1/thumb/mickey-mouse.jpg
Now let’s say that we want to implement some simple security around these images…. reason #1 from my list above. The first thing we’ll need to do, then, is to remove the image files from the public folder and instead save them in some non-public place on our web server, for example:
$ mkdir non-public
$ mv public/system non-public/.
Now let’s double check that Apache can’t find the files in their new location:
Great! We see a missing image as expected. No users can see the files unless we run some bit of Ruby code to enable access. Now… how can we use a controller to download the files through Rails, instead of through Apache? The first thing we need to do is add a route to routes.rb for accessing the files.
If you open up routes.rb and look at what the scaffolding generator created for us, you’ll see this:
ActionController::Routing::Routes.draw do |map|
map.resources :users
...etc...
map.connect ':controller/:action/:id'
map.connect ':controller/:action/:id.:format'
end
The map.resources line indicates that our application supports a series of routes that handle the four REST actions: GET, POST, PUT and DELETE. The best way to get a handle on how the routes work is by running “rake routes” to list out all the URL patterns that Rails will match on:
$ rake routes
(in /Users/pat/rails-apps/paperclip-sample-app)
users GET /users(.:format) {:action=>"index", :controller=>"users"}
POST /users(.:format) {:action=>"create", :controller=>"users"}
new_user GET /users/new(.:format) {:action=>"new", :controller=>"users"}
edit_user GET /users/:id/edit(.:format) {:action=>"edit", :controller=>"users"}
user GET /users/:id(.:format) {:action=>"show", :controller=>"users"}
PUT /users/:id(.:format) {:action=>"update", :controller=>"users"}
DELETE /users/:id(.:format) {:action=>"destroy", :controller=>"users"}
/:controller/:action/:id
/:controller/:action/:id(.:format)
The last two lines are the “default routes,” which connect any URL matching the pattern controller/action/id to the corresponding controller. We could just go ahead and use the default routes, but to learn a bit more about how map.resources works, let’s create a new URL pattern for our users that we’ll use in a minute to download the avatar file attachments with. Edit routes.rb and add the :member parameter in bold:
ActionController::Routing::Routes.draw do |map|
map.resources :users, :member => { :avatars => :get }
etc...
If we now re-run rake routes, we can see that a new URL pattern was created for us that we can use to download the avatar images via a GET request:
$ rake routes
(in /Users/pat/rails-apps/paperclip-sample-app)
users GET /users(.:format) {:action=>"index", :controller=>"users"}
POST /users(.:format) {:action=>"create", :controller=>"users"}
new_user GET /users/new(.:format) {:action=>"new", :controller=>"users"}
edit_user GET /users/:id/edit(.:format) {:action=>"edit", :controller=>"users"}
avatars_user GET /users/:id/avatars(.:format) {
:action=>"avatars",
:controller=>"users"
}
user GET /users/:id(.:format) {:action=>"show", :controller=>"users"}
PUT /users/:id(.:format) {:action=>"update", :controller=>"users"}
DELETE /users/:id(.:format) {:action=>"destroy", :controller=>"users"}
/:controller/:action/:id
/:controller/:action/:id(.:format)
Now to get the avatar for user 7, for example, we can issue a URL like this:
http://localhost:3000/users/7/avatars
… and the request will be routed to the “avatars” action in the “users” controller (plural since a user might have more than one style of avatar). So now let’s go right ahead and implement the avatars method and add some code to download a file to the client. The way to do that is to use ActionController::Streaming::send_file. It’s simple enough; we just need to pass the file’s path to send_file as well as the MIME content type which the client uses as a clue for deciding how to display the file, and that’s it! Let’s hard code these values for now and see if it all works (update the path here for your machine):
class UsersController < ApplicationController
def avatars
send_file '/path/to/non-public/system/avatars/1/original/mickey-mouse.jpg',
:type => 'image/jpeg'
end
Now if you type http://localhost:3000/users/1/avatars into your browser you should see the mickey image again. If not, then double check your code changes, where the files are actually located on the hard drive now and also try stopping and reloading the Rails app since changes to routes.rb are cached and are only loaded when Rails is initialized.
Instead of hard coding the path in the avatars method, we obviously need to be able to handle requests for any avatar file attachment for any user record. Before we enhance our code to do this, let’s take a few minutes to configure Paperclip and tell it where the files are now stored on the file system, and which URL we have configured our routes.rb file to use. This will make our work coding in the controller a lot easier, and also indicate to Paperclip where new file attachments should be uploaded to. To do this, we need to add a couple of parameters to our call to has_attached_file in our User model (user.rb), shown here in bold (again, update the path for your machine):
class User < ActiveRecord::Base
has_attached_file :avatar,
:styles => { :thumb => "75x75>", :small => "150x150>" },
:path => '/path/to/non-public/system/avatars/1/original/mickey-mouse.jpg',
:url => 'users/1/avatars'
end
Just to take one step at a time, I’ve hard coded the URL and path again here in the model. But now we can generalize our code in UserController to handle any user, like this:
def avatars
user = User.find(params[:id])
send_file user.avatar.path, :type => user.avatar_content_type
end
Now we can test http://localhost:3000/users/1/avatars again to be sure that we haven’t broken anything. If it’s all working, lets’ proceed to clean up the hard coding in user.rb. It turns out that Paperclip uses the same interpolations idea that we saw above in routes.rb. So I can use symbols like :rails_root, :id, :style, etc., and they will be evaluated to the values I expect and need. Here’s the finished code in my model:
has_attached_file :avatar,
:styles => { :thumb => "75x75>", :small => "150x150>" },
:path =>
':rails_root/non-public/system/:attachment/:id/:style/:basename.:extension',
:url => '/:class/:id/:attachment'
If we open up the console and take a look at our user object there, we can see that Paperclip is substituting the correct values for each of the symbols I’ve provided:
$ ./script/console
Loading development environment (Rails 2.3.2)
>> User.first.avatar.url
=> "/users/1/avatars?1242395876" (time stamp appended here automatically)
>> User.first.avatar.path
=> "/Users/pat/.../non-public/system/avatars/1/original/mickey-mouse.jpg"
Now let’s go back and test our original application again with our new code:
- The new route in routes.rb
- The new action in UserController
- And the :url and :path parameters added to has_attached_file in the User model.
Oops… we see the large image again. It doesn’t work! What happened? Well, it turns our that we forgot one detail in our new avatars method in UserController. Our code always returns the default, or “original” style of the file attachment, but in the users index view we actually display the thumbnail image using image_tag user.avatar.url(:thumb). So our controller code needs to be able to handle requests for other styles as well. To do this, we need to pass the requested style somehow. The simplest thing to do is just to add the style as a URL parameter to the download request, like this:
http://localhost:3000/users/1/avatar?style=thumb
(We could also add the style to the URL's path, but that would require another change to routes.rb.) To make this work, first I need to tell Paperclip about how I want to handle the style value in the URL:
has_attached_file :avatar,
:styles => { :thumb => "75x75>", :small => "150x150>" },
:path =>
':rails_root/non-public/system/:attachment/:id/:style/:basename.:extension',
:url => '/:class/:id/avatar?style=:style'
And now I need to handle this new parameter in my controller:
def avatars
user = User.find(params[:id])
style = params[:style] ? params[:style] : 'original'
send_file user.avatar.path(style),
:type => user.avatar_content_type
end
I don’t need to change anything in index.html.erb since there we call image_tag user.avatar.url(:thumb) which picks up the new URL pattern from Paperclip.
And now, if I’ve got all of this correct, I should be able to finally see the thumbnail image again on the index page:
And finally we have files being downloaded by Ruby code present in the UserController class. If we actually wanted to implement security, logging or some other sort of logic we would just add code to the avatars method in UserController. For example, avatars could return a 401 (unauthorized) error if the user wasn’t logged in, or didn’t have access to view Mickey’s image for some reason.
That’s it for now; next time I’ll modify this sample app once more to demonstrate how we can store the image files in a database table, instead of in the “non-public” folder or anywhere on the file system.
Tags:paperclip·rails
I love scaffolding. Many experienced Rails developers scoff at the idea of using scaffolding to generate Rails code: it’s ugly; it probably means you don’t understand how to write the code yourself; it generates a lot more code than you need, etc., etc. However, for a beginning Rails developer working on her/his own like me who isn’t surrounded by a team of Ruby experts, scaffolding is an essential tool and can help to get started in the right direction. Also, even for experienced Rubyists scaffolding can be a great way to quickly (minutes, not hours or days) get a simple app up and running to use for demos, UI wireframes, spiking some technical issue, etc.
This post will demonstrate how to use scaffolding to create a new Rails app from scratch that uses the Paperclip plugin to upload and display an image file. Feel free to copy/paste pieces of code from the narrative below and use them in your app, or you can just skip to the chase and get the finished version from github and run that on your machine.
There are a lot of other good tutorials out there about this; see:
I’ll take on the risk of repeating material that’s already out there in order to show how easy it is to get a working Paperclip application up and running using scaffolding. The fact that just a few commands and lines of code are required illustrates just how simple and powerful Paperclip’s design is. In my next post, I’ll proceed to change this sample app to demonstrate how to save the uploaded files in a database column instead of on the web server’s file system, using my modified version of Paperclip.
FYI At the time I wrote this, Rails was at version 2.3.2:
$ rails --version
Rails 2.3.2
Let’s get started by creating a new Rails application:
$ rails paperclip-sample-app
create
create app/controllers
create app/helpers
create app/models
create app/views/layouts
create config/environments
create config/initializers
create config/locales
create db
create doc
create lib
create lib/tasks
create log
etc...
Before we go any farther, let’s setup our database.yml file and create a new MySQL database to use with the sample app. Replace the contents of config/database.yml with this:
development:
adapter: mysql
database: paperclip_sample_app_development
username: root
password:
host: localhost
Enter the proper username and password for MySQL if they are not “root” and null. And then run this from the command line:
$ cd paperclip-sample-app
$ rake db:create
(in /Users/pat/rails-apps/paperclip-sample-app)
Ok, now we have a MySQL database to work with. Next, let’s go ahead and install the Paperclip plugin. The best thing to do is just to get the latest version from github; Thoughtbot frequently updates it with bug fixes, enhancements, etc.:
$ ./script/plugin install git://github.com/thoughtbot/paperclip.git
Initialized empty Git repository in /Users/pat/rails-apps/paperclip-sample-app/vendor/plugins/paperclip/.git/
remote: Counting objects: 62, done.
remote: Compressing objects: 100% (50/50), done.
remote: Total 62 (delta 6), reused 39 (delta 4)
Unpacking objects: 100% (62/62), done.
From git://github.com/thoughtbot/paperclip
* branch HEAD -> FETCH_HEAD
Now that we have an empty, shell application created and the Paperclip plugin installed, we can use scaffolding to add some working code to it. Let’s use the same “user” and “avatar” example Thoughtbot does on the Paperclip project page. The idea is that the sample will contain a table of users, and each user will have an avatar image displayed in the web site. So to get started, I’ll just create a new “user” model with string columns for the name and email address:
$ ./script/generate scaffold user name:string email:string
exists app/models/
exists app/controllers/
exists app/helpers/
create app/views/users
exists app/views/layouts/
exists test/functional/
exists test/unit/
create test/unit/helpers/
exists public/stylesheets/
create app/views/users/index.html.erb
create app/views/users/show.html.erb
etc...
Now we need to generate the database columns necessary for Paperclip on our new model object using script/generate:
$ ./script/generate paperclip user avatar
exists db/migrate
create db/migrate/20090430084151_add_attachments_avatar_to_user.rb
And let’s go ahead and create the users table using db:migrate:
$ rake db:migrate
(in /Users/pat/rails-apps/paperclip-sample-app)
== CreateUsers: migrating ====================================================
-- create_table(:users)
-> 0.0031s
== CreateUsers: migrated (0.0032s) ===========================================
== AddAttachmentsAvatarToUser: migrating =====================================
-- add_column(:users, :avatar_file_name, :string)
-> 0.0063s
-- add_column(:users, :avatar_content_type, :string)
-> 0.0069s
-- add_column(:users, :avatar_file_size, :integer)
-> 0.0085s
-- add_column(:users, :avatar_updated_at, :datetime)
-> 0.0081s
== AddAttachmentsAvatarToUser: migrated (0.0311s) ============================
You can see that the Paperclip generator created columns in the users table called “avatar_file_name,” “avatar_content_type,” “avatar_file_size” and “avatar_updated_at.” Now we have our database schema setup. The next step is to just modify the code that was generated for us by the scaffolding and make the changes necessary for Paperclip. The first thing to do is to add a line to the user model and indicate that it has a file attachment called “avatar.” To do this, open app/models/user.rb and just add this one line:
class User < ActiveRecord::Base
has_attached_file :avatar
end
And then edit the new user form (app/views/users/new.html.erb) and add a file field to use to upload files. There are actually two code changes you need to make: first you need to set the HTML form to encode the uploaded file data (and other fields) using MIME multiple part syntax, and then second you need to actually add the file upload field. Here’s the finished new.html.erb file with these two changes in bold:
<h1>New user</h1>
<% form_for(@user, :html => { :multipart => true }) do |f| %>
<%= f.error_messages %>
<p>
<%= f.label :name %><br />
<%= f.text_field :name %>
</p>
<p>
<%= f.label :email %><br />
<%= f.text_field :email %>
</p>
<p>
<%= f.label :avatar %><br />
<%= f.file_field :avatar %>
</p>
<p>
<%= f.submit 'Create' %>
</p>
<% end %>
<%= link_to 'Back', users_path %>
Also make the same changes to the edit form that was generated by the scaffolding: app/views/users/edit.html.erb. The best thing to do would be to include the same ERB file (maybe called “_form.html.erb”) in both the new and edit form files. Ideally the scaffolding generator would have done this for us…
Now if we run our application we can upload an image file and attach it to a user:
If you submit this form, the image file will be uploaded to the server and saved on the file system. By default, Paperclip saves files inside a “system” folder it creates in your Rails app’s public folder. Let’s take a look at my public folder and see where the file went:
$ find public/system
public/system
public/system/avatars
public/system/avatars/1
public/system/avatars/1/original
public/system/avatars/1/original/mickey-mouse.jpg
This is one of the nice things about Paperclip: it just works. I don’t have to think about or worry about where the files are going to go; Thoughtbot has chosen simple default values that make sense. Here we can see that there are a series of folders created that correspond to the attachment name, model primary key and also the “style” of the attachment (more on that below).
If you want to or need to save the files in some other place on your server’s file system you can specify different options to has_attached_file in your model; see this write up for an example: http://travisonrails.com/2009/01/11/Changing-Paperclip-File-Storage-Location. Paperclip also supports saving the files in Amazon’s S3 storage service, and in my next post I’ll demonstrate how to save the file data inside the database itself, right in the users table in this example.
I’m almost done; now I just need to display the uploaded image somewhere; the simplest thing to do is just to add an image tag to the users show page. Again, my changes to the standard scaffolding code are in bold:
<p>
<b>Name:</b>
<%=h @user.name %>
</p>
<p>
<b>Email:</b>
<%=h @user.email %>
</p>
<p>
<b>Avatar:</b>
<%= image_tag @user.avatar.url %>
</p>
<%= link_to 'Edit', edit_user_path(@user) %> |
<%= link_to 'Back', users_path %>
Now we can see the image for our new user:
Since this image is bigger that what I would like, I can take advantage of Paperclip “styles” feature to generate a smaller version of it. To do that you will need to be sure you have ImageMagick installed on your server, which is what Paperclip uses behind the scenes to modify image files. Then all you need to do is just add two “styles” to your model, like this:
class User < ActiveRecord::Base
has_attached_file :avatar,
:styles => {
:thumb => "75x75>",
:small => "150x150>"
}
end
The strings we pass in are actually options for ImageMagick's "convert" command; see it’s documentation for more details. And now in the show ERB we can just specify the “small” style in the image tag instead:
<%= image_tag @user.avatar.url(:small) %>
To see it, first I need to re-edit and re-upload the image (remember to add the file field code to edit.html.erb just like for new.html.erb):
Now when this form is submitted we will see the smaller image:
And let’s take another look at the public/system folder and see what Paperclip and ImageMagick have done for us:
$ find public/system
public/system
public/system/avatars
public/system/avatars/1
public/system/avatars/1/original
public/system/avatars/1/original/mickey-mouse.jpg
public/system/avatars/1/small
public/system/avatars/1/small/mickey-mouse.jpg
public/system/avatars/1/thumb
public/system/avatars/1/thumb/mickey-mouse.jpg
Again, this is very simple and just works! As a last step, let’s add the thumbnail image to the users index page so we can see Mickey without even clicking on that user record. This is as simple as editing app/views/users/index.html.erb and adding a new table column:
<table>
<tr>
<th>Photo</th>
<th>Name</th>
<th>Email</th>
</tr>
<% @users.each do |user| %>
<tr>
<td><%= image_tag user.avatar.url(:thumb) %></td>
<td><%=h user.name %></td>
<td><%=h user.email %></td>
<td><%= link_to 'Show', user %></td>
<td><%= link_to 'Edit', edit_user_path(user) %></td>
<td><%= link_to 'Destroy', user, :confirm => 'Are you sure?', :method => :delete %></td>
</tr>
<% end %>
</table>
And now we just need to refresh the index page since the thumb image file was already generated:
And there you have it: a working file upload web site written in minutes. This was made possible by Rails scaffolding, and Paperclip's simple, elegant design.
Tags:paperclip·rails
Back in February I wrote an implementation of a new storage module for Paperclip that supports saving file attachments in a database table. My original implementation saved the file attachments in a separate database table, which was internally managed using a “has_many” relationship from the target model.
This month I decided to rewrite the code to use a simpler, single table approach. Instead of saving the files in a separate table, each file is saved in a BLOB column in the same table as the target model. This makes the database storage module easier and more intuitive to use, and moves it closer to the original intent of the Paperclip design.
Code: http://github.com/patshaughnessy/paperclip
New usage:
- Install and use my version of the plugin:
script/plugin install git://github.com/patshaughnessy/paperclip.git
- In your model specify the "database" storage option; for example:
has_attached_file :avatar, :storage => :database
The file will be stored in a column called [attachment name]_file (e.g. "avatar_file") by default. To specify a different BLOB column name, use :column, like this:
has_attached_file :avatar,
:storage => :database,
:column => 'avatar_data'
- If you have defined different styles, these files will be stored in additional columns called [attachment name]_[style name]_file (e.g. "avatar_thumb_file") by default. To specify different column names for each style, use :column in the style definition, like this:
has_attached_file :avatar,
:storage => :database,
:styles => {
:medium => {
:geometry => "300x300>",
:column => 'medium_file'
},
:thumb => {
:geometry => "100x100>",
:column => 'thumb_file'
}
}
- You need to create these new columns in your migrations or you'll get an exception. Example:
add_column :users, :avatar_file, :binary
add_column :users, :avatar_medium_file, :binary
add_column :users, :avatar_thumb_file, :binary
Note the migration for the "binary" column type will not work for LONGBLOBs in MySQL. Here's an example migration for MySQL:
execute 'ALTER TABLE users ADD COLUMN avatar_file LONGBLOB'
execute 'ALTER TABLE users ADD COLUMN avatar_medium_file LONGBLOB'
execute 'ALTER TABLE users ADD COLUMN avatar_thumb_file LONGBLOB'
-
To avoid performance problems loading all of the BLOB columns every time you access your ActiveRecord object, a class method is provided on your model called “select_without_file_columns_for.” This is set to a hash that will instruct ActiveRecord::Base.find to load all of the columns except the BLOB/file data columns, for example:
{:select=>"id,name,avatar_file_name,avatar_content_type,..."}
If you’re using Rails 2.3, you can specify this as a default scope:
default_scope select_without_file_columns_for(:avatar)
Or if you’re using Rails 2.1 or 2.2 you can use it to create a named scope:
named_scope :without_file_data, select_without_file_columns_for(:avatar)
- By default, URLs will be set to this pattern:
/:relative_root/:class/:attachment/:id?style=:style
Example:
/app-root-url/users/avatars/23?style=original
The idea here is that to retrieve a file from the database storage, you will need some controller's code to be executed. Once you pick a controller to use for downloading, you can add this line to generate the download action for the default URL/action (the plural attachment name), "avatars" in this example:
downloads_files_for :user, :avatar
Or you can write a download method manually if there are security, logging or other requirements. If you prefer a different URL for downloading files you can specify that in the model; e.g.:
has_attached_file :avatar,
:storage => :database,
:url =>'/users/show_avatar/:id/:style'
- Add a route for the download to the controller which will handle downloads, if necessary. The default URL, /:relative_root/:class/:attachment/:id?style=:style, will be matched by the default route: :controller/:action/:id
For now, I’ve overwritten my code from February. I believe this implementation is cleaner and easier to use; if anyone did download and use my code from February let me know and I can help you migrate the file data from the separate table back to columns in primary table. I will be writing a script to do this for my own application.
When I have time in the next few days or weeks, I’ll post a sample application that illustrates all of these steps and shows exactly how to do all of this. If anyone has any questions or suggestions please let me know.
Tags:paperclip·rails
I just updated my customized version of the auto_complete plugin to allow you to provide a named scope to auto_complete_for, in order to filter the auto_complete pick list options differently than the plugin does by default. The updated code is now on github:
http://github.com/patshaughnessy/auto_complete
This is based on the ideas from my last post, Andrew Ng’s original post and my friend Alex’s suggestion to use named scopes instead of manually modifying the find options. Here’s an example of how to use it taken from the auto_complete sample app I posted in January:
- Add a named scope to your target model: For example suppose tasks belong to projects and have a named scope “by_project” which joins on the projects table and returns the tasks belonging to the project with the given name:
class Task < ActiveRecord::Base
belongs_to :project
named_scope :by_project,
lambda { |project_name| {
:include => :project,
:conditions => [ "projects.name = ?", project_name ]
} }
end
- In the controller, pass a block to auto_complete_for to specify that a named scope should be used to generate the competion options. Here the “by_project” named scope will be used to handle the task auto complete requests, using the “project” HTTP parameter:
auto_complete_for :task, :name do | items, params |
items.by_project(params['project'])
end
- In the view, optionally specify additional parameters you might want to pass into your named scope: in my sample app I have a field called “project_name” elsewhere on my form:
<% fields_for_task task do |f| -%>
…
<%= f.text_field_with_auto_complete :task,
:name,
{},
{
:method => :get,
:with =>"value + '&project=' + $F('project_name')"
} %>
…
<% end -%>
So how does this work? Let’s take a look at my new implementation of auto_complete_for:
def auto_complete_for(object, method, options = {})
define_method("auto_complete_for_#{object}_#{method}") do
model = object.to_s.camelize.constantize
find_options = {
:conditions => [ "LOWER(#{model.quoted_table_name}.#{method}) LIKE ?",
'%' + params[object][method].downcase + '%' ],
:order => "#{model.quoted_table_name}.#{method} ASC",
:limit => 10 }.merge!(options)
@items = model.scoped(find_options)
@items = yield(@items, params) if block_given?
render :inline => "<%= auto_complete_result @items, '#{method}' %>"
end
end
One minor change I made here was to call “quoted_table_name” on the given model to specify the table name in the SQL generated to retrieve the auto complete results later. This was needed in case, like in my sample application, the controller specifies a named scope that joins with another table containing columns with the same name as the target model. If this isn’t the case, adding the table name to the SQL is harmless.
However, the most important 2 lines here are in bold: first we call a function called “scoped” to create an anonymous named scope based on the default auto_complete options “find_options:”
@items = model.scoped(find_options)
The exciting thing about this line, which Alex explained in his blog post, is that the use of named scopes delays the corresponding SQL statement from being executed until later when we actually access the query results in auto_complete_result. What happens instead is that an ActiveRecord:: NamedScope::Scope object is created, containing a temporary cache of the find options.
A good way to understand how this works is to try it in the Rails console:
complex-form-examples pat$ ./script/console
Loading development environment (Rails 2.1.0)
>> find_options = {
?> :conditions => [ "LOWER(`tasks`.name) LIKE ?", '%t%' ],
?> :order => "`tasks`.name ASC",
?> :limit => 10 }
=> {:order=>"name ASC", :conditions=>["LOWER(name) LIKE ?", "%t%"], :limit=>10}
>> Task.scoped(find_options)
=> [#<Task id: 4, project_id: 2, name: "Task 2a", due_at: nil, created_at: "2009-04-02 16:21:54",
updated_at: "2009-04-02 16:21:54">, #<Task id: 5, project_id: 2, name: "Task 2b", due_at: nil,
created_at: "2009-04-02 16:21:54", updated_at: "2009-04-02 16:21:54">, #<Task id: 6, project_id: 2,
name: "Task 2c", due_at: nil, created_at: "2009-04-02 16:21:54", updated_at: "2009-04-02
16:21:54">, #<Task id: 1, project_id: 1, name: "Task One", due_at: nil, created_at: "2009-04-02
16:21:30", updated_at: "2009-04-02 16:21:30">, #<Task id: 3, project_id: 1, name: "Task
Three", due_at: nil, created_at: "2009-04-02 16:21:30", updated_at: "2009-04-02 16:21:30">,
#<Task id: 2, project_id: 1, name: "Task Two", due_at: nil, created_at: "2009-04-02 16:21:30",
updated_at: "2009-04-02 16:21:30">]
Wait a minute! I thought the actual SQL execution was delayed by named scopes until I needed to access the results? Here the console has already displayed the query results, so the SQL statement must have been executed already. How and why did this happen? In this case, when you enter an expression into the Rails console and press ENTER, the expression is evaluated and then the “inspect” method is called on it. The problem is that the named scopes implementation has delegated the “inspect” method to another method, which executes the SQL statement and loads the query results.
We can use a trick in the console to open the ActiveRecord::NamedScope::Scope class and override the inspect method so the SQL is not executed, and prove to ourselves that “scoped()” actually does return a named scope object without executing the SQL statement:
>> module ActiveRecord
>> module NamedScope
>> class Scope
>> def inspect
>> super # Avoids calling ActiveRecord::Base.find and calls Object.inspect
>> end
>> end
>> end
>> end
=> nil
>> Task.scoped(find_options)
=> #<ActiveRecord::NamedScope::Scope:0x21e65d4
@proxy_options={:conditions=>["LOWER(`tasks`.name) LIKE ?", "%t%"],
:order=>"`tasks`.name ASC", :limit=>10},
@proxy_scope=Task(id: integer, project_id: integer, name: string,
due_at: datetime, created_at: datetime, updated_at: datetime)>
So here we can see that “scoped” returns an ActiveRecord::NamedScope::Scope object, and that it has two interesting instance variables: proxy_scope and proxy_options. The first of these, proxy_options, contains the find options that were passed into the scoped() function, or into the “named_scope” declaration in your model. The second value, proxy_scope, indicates the parent scope or context in which this named scope object’s SQL statement should be run. In this example, that is the Task model itself. The named scope object is essentially a cache of the query options that will be user later when the query is executed.
Let’s see how this works in the auto_complete plugin. Back again to the new implementation of auto_complete_for, we have:
@items = model.scoped(find_options)
@items = yield(@items, params) if block_given?
The first line generates a ActiveRecord::NamedScope::Scope object, which is then passed into the block provided by the controller code, if any. Let’s take a look at my sample app’s implementation of the controller:
auto_complete_for :task, :name do | items, params |
items.by_project(params['project'])
end
This is a good example of the second very cool feature of named scopes: that they are composable… in other words, that two or more named scopes can be combined together to form a single SQL statement that is executed only once! Let’s return to the same Rails console session with our redefined “inspect” method and see if we can understand a bit more about this:
>> Task.scoped(find_options).by_project 'Project One'
=> #<ActiveRecord::NamedScope::Scope:0x21d1864
@proxy_options={
:conditions=>["projects.name = ?", "Project One"],
:include=>:project},
@proxy_scope=
#<ActiveRecord::NamedScope::Scope:0x21d1a30
@proxy_options={
:conditions=>["LOWER(`tasks`.name) LIKE ?", "%t%"],
:order=>"`tasks`.name ASC",
:limit=>10},
@proxy_scope=Task(id: integer, project_id: integer, name: string, due_at: datetime, created_at: datetime, updated_at: datetime)
>
>
Now we can see that calling scoped(find_options).by_project just returns a chain of two named scopes: the first scope object with @proxy_scope set to the second one, and the second one with @proxy_scope set to the base model class. Later when this SQL query is executed, the code in NamedScope and ActiveRecord::Base will simply walk this chain of objects, accumulate the options into a single hash, convert the hash to SQL and execute it.
In auto_complete_for after the controller’s block returns, the “@items” value in auto_complete_for above is set to the parent/child named scope chain of objects, and then passed into auto_complete_result:
render :inline => "<%= auto_complete_result @items, '#{method}' %>"
Inside of auto_complete_result the @items value is used as if it were an array… like this:
def auto_complete_result(entries, field, phrase = nil)
return unless entries
items = entries.map { |entry|
content_tag("li",
phrase ? highlight(entry[field], phrase) : h(entry[field]))
}
content_tag("ul", items.uniq)
end
… to generate the HTML needed for the Prototype library’s implementation of the auto complete drop down box. The interesting thing here is that the SQL statement is executed as soon as the call to “map” is executed, accessing the elements of the “@items” array. This works because the ActiveRecord::NamedScope::Scope class redirects or delegates the [] and other array methods to ActiveRecord::Base.find. Here's the single SQL that is executed with the combined, accumulated query options:
SELECT `tasks`.`id` AS t0_r0, `tasks`.`project_id` AS t0_r1,
`tasks`.`name` AS t0_r2, `tasks`.`due_at` AS t0_r3,
`tasks`.`created_at` AS t0_r4, `tasks`.`updated_at` AS t0_r5,
`projects`.`id` AS t1_r0, `projects`.`name` AS t1_r1,
`projects`.`created_at` AS t1_r2, `projects`.`updated_at` AS t1_r3
FROM `tasks`
LEFT OUTER JOIN `projects` ON `projects`.id = `tasks`.project_id
WHERE ((projects.name = 'Project One') AND
(LOWER(`tasks`.name) LIKE '%t%'))
ORDER BY `tasks`.name ASC LIMIT 10
In fact, in the original version of the auto_complete plugin before my changes to it for named scopes, the value passed into auto_complete_result was a simple array. The fact that named scopes are used now is entirely hidden from this code!
One last note here about named scope: as described in a comment in named_scope.rb from the Rails source code, the “proxy_options” method provides a convenient way to test the behavior of named scopes without actually checking the results of an actual SQL query. Here’s one of the tests I wrote for my new version of auto_complete_for:
def test_default_auto_complete_for
get :auto_complete_for_some_model_some_field,
:some_model => { :some_field => "some_value" }
default_auto_complete_find_options = @controller.items.proxy_options
assert_equal "`some_models`.some_field ASC",
default_auto_complete_find_options[:order]
assert_equal 10, default_auto_complete_find_options[:limit]
assert_equal ["LOWER(`some_models`.some_field) LIKE ?", "%some_value%"],
default_auto_complete_find_options[:conditions]
end
Since I didn’t want to go to the trouble of setting up an actual in-memory database using SQLite, or to introduce mocha or some other mocking framework to the auto_complete tests, all I had to do was just call @controller.items.proxy_options and check that the find options are as expected. (I also had to expose “items” in the mock controller using attr_reader.) I have another test that checks that the controller’s block is called and it’s named scope options are present as expected… this test uses the proxy_scope method to walk up the chain to the parent named scope and get it’s proxy_options. See auto_complete_test.rb for details.
Tags:auto_complete·rails
I’ve written a lot here during the past few months about the auto_complete plugin and how to get it to work with repeated text fields on a complex form. Back in January I modified Ryan Bates’s complex forms sample app to illustrate how to use my version of the auto complete plugin to handle repeated text fields. Here’s what the form looks like in that sample app:
Here as the user types into a “Task” text field, a list of all of the existing task names in the system that match the characters typed in by the user are displayed in a drop down list. But what if I didn’t want to display all of the matching task names? What if I wanted to display only the tasks for a given project? Or if I wanted to filter the task names in some other way based on other field values?
In this simple example, what if I only wanted to display Tasks 2a, 2b and 2c, since they belonged to Project Two?
Today I took a look at this problem and expected to see a number of simple solutions, but instead I was surprised to find that it is fairly difficult to do this. I got started by reading this nice solution from Andrew Ng (nice work Andrew!). Andrew explains how to get the Prototype javascript code to pass an additional HTTP parameter to the server when the user types into an autocomplete field. This additional parameter can then be used to filter the list of auto complete options differently. I’ll let you read the details, but basically Andrew found that you can use a Javascript callback function like this to load a value from another field on your form, and pass it to the server in the Ajax request as an additional query string parameter:
<script type="text/javascript">
new Ajax.Autocompleter(
'task_name',
'task_name_auto_complete',
'/projects/auto_complete_for_task_name',
{ callback: function(e, qs) {
return qs + '&project=' + $F('project_name');
}
}
);
</script>
(I’ve renamed the variables to use my project/tasks example.) What I didn’t like about this was the need to manually code all of this javascript; there must be a way to get the auto_complete plugin to do this instead… and there is! If you look at the definition of text_field_with_auto_complete in auto_complete_macros_helper.rb, you’ll see that it takes both tag_options and completion_options as parameters, and eventually calls auto_complete_field with the completion_options. Here’s what auto_complete_field looks like in the auto_complete plugin:
def auto_complete_field(field_id, options = {})
function = "var #{field_id}_auto_completer = new Ajax.Autocompleter("
function << "'#{field_id}', "
etc...
js_options = {}
js_options[:tokens] = etc...
js_options[:callback] =
"function(element, value) { return #{options[:with]} }" if options[:with]
js_options[:indicator] = etc...
js_options[:select] = etc...
js_options[:paramName] = etc...
etc...
function << (', ' + options_for_javascript(js_options) + ')')
javascript_tag(function)
end
If you look closely at the line I bolded above, you’ll see that we can actually generate Andrew’s Javascript callback function automatically by simply passing in a value for the “with” completion option when we call text_field_with_auto_complete in our view, like this:
text_field_with_auto_complete :task, :name, {},
{
:method => :get,
:with =>"value + '&project=' + $F('project_name')"
}
Again, this line of Javascript code is called when the user types into the task name field, and appends “&project=XYZ” to the query string for the Ajax request. “XYZ” is the name of the project typed in by the user on the same form, loaded with prototype’s “$F” (Form element get value) function. The “:method => :get” completion option is used to avoid problems with CSRF protection; see http://www.ruby-forum.com/topic/128970. If you look at your server’s log file, you’ll see HTTP requests that look something like this now:
127.0.0.1 - - [13/Mar/2009:16:17:03 EDT] "
GET /projects/auto_complete_for_task_name?task%5Bname%5D=T&project=Project%20Two
HTTP/1.1" 200 57
Here we can see the “auto_complete_for_task_name” route is called and given two request parameters: “task[name]” and “project”. The task name is the standard parameter generated by the autocompleter javascript, and “project” is the additional parameter created by the callback function generated by the :with option.
Now… how do we handle the “project” parameter in our controller code? Without modifying the auto_complete plugin itself, you would have to write your own controller method and not use the “auto_complete_for” macro at all. Andrew shows how to do this on his blog. What I want to explore here now is whether there’s a way to change the auto_complete_for method to allow for customizations of the query used to load the auto complete options.
To understand the problem a bit better, let’s take a look at how “auto_complete_for” is implemented in the auto_complete plugin:
def auto_complete_for(object, method, options = {})
define_method("auto_complete_for_#{object}_#{method}") do
find_options = {
:conditions => [ "LOWER(#{method}) LIKE ?", '%' +
params[object][method].downcase + '%' ],
:order => "#{method} ASC",
:limit => 10 }.merge!(options)
@items = object.to_s.camelize.constantize.find(:all, find_options)
render :inline => "<%= auto_complete_result @items, '#{method}' %>"
end
end
When this is called as your Rails application initializes, it adds a new method to your controller called something like “auto_complete_for_task_name” with your model and column names instead. What we want to do is filter the query results differently, by using a new HTTP parameter – so we need to modify the “conditions” hash passed into find :all. At first I tried to do this by passing in different values for the “options” parameter, since that’s merged with the default options and then passed into find :all. However, the problem with this approach is that whatever you pass in using “options” will not have access to the request parameters, since it’s passed in when the controller is initialized, and not when the HTTP request is received.
So the solution is to pass in a block that is evaluated when the request is received, and when the generated method is actually called. I wrote a variation on auto_complete_for called “filtered_auto_complete_for:”
def filtered_auto_complete_for(object, method)
define_method("auto_complete_for_#{object}_#{method}") do
find_options = {
:conditions => [ "LOWER(#{method}) LIKE ?", '%' +
params[object][method].downcase + '%' ],
:order => "#{method} ASC",
:limit => 10 }
yield find_options, params
@items = object.to_s.camelize.constantize.find(:all, find_options)
render :inline => "<%= auto_complete_result @items, '#{method}' %>"
end
end
Filtered_auto_complete_for takes a block and evaluates it when the actual HTTP Ajax request is received from the auto complete Javascript. The block is provided with the find options hash and also the request parameters. This enables the controller’s block to modify the find options in any way it would like, possibly using the HTTP request parameters provided. I’ve also removed the options parameter since that’s not necessary any more.
As an example, here’s my sample app’s controller code:
class ProjectsController < ApplicationController
# Handle auto complete for project names as usual:
auto_complete_for :project, :name
# For task name auto complete, only display tasks
# that belong to the given project:
filtered_auto_complete_for :task, :name do | find_options, params|
find_options.merge!(
{
:include => :project,
:conditions => [ "LOWER(tasks.name) LIKE ? AND projects.name = ?",
'%' + params['task']['name'].downcase + '%',
params['project'] ],
:order => "tasks.name ASC"
}
)
end
def index
@projects = Project.find(:all)
end
etc...
The code in this sample block modifies the find_options by adding “:include => :project”. This causes ActiveRecord to use a JOIN to include columns from the project table in the query (in this sample app Project has_many Tasks, and each Task belongs_to a Project). Then it matches on the project name, in addition to the portion of the task name typed in by the user so far. This limits the auto complete values to just the tasks that belong to the given project:
When I have time during the next few days I’ll add “filtered_auto_complete_for” to my forked version of the auto_complete plugin… first I need to write some unit tests for it, and be sure it works as intended. After that, I’ll post this sample app back on github and you can try it yourself.
Tags:auto_complete·rails
Update April 2009:
I just rewrote this to use the same database table as the target model to save the file data, and not a separate database table. The “create_table” migration I describe here below will no longer work; instead you should create one or more columns in the same table as the target model to hold the file data. Please read this update for more details.
Paperclip from Thoughtbot is a fantastic bit of code that allows you to easily upload files to your Rails app and later manage them as just another attribute on your model object. If you’re not familiar with Paperclip you should start by reading Thoughtbot’s Paperclip intro page; Ryan Bates also did a screen cast on Paperclip usage. By default it supports saving the file attachments on the server file system, and also there’s an option for saving the files in Amazon’s S3 service. One reason I decided to use Paperclip in a recent project was that the implementation and usage were both much simpler and cleaner than attachment_fu, the other popular Rails plugin for file upload and management.
One thing that Thoughtbot decided not to implement was the ability to store files in a database table, rather than on the file system. It doesn’t make a lot of sense to do this for most normal web application deployments, since serving files via Apache directly from the file system is obviously much faster and avoids the need to call your Rails stack at all for download requests. However, I work in an Enterprise IT environment that has a lot of experience with Oracle, and finds it easier to manage file attachments with a database table. I also have requirements around file encryption, security, etc.
Since Paperclip doesn’t include a database storage option, I decided to write one. Here’s what I came up with: http://github.com/patshaughnessy/paperclip
I added a new storage module called Paperclip::Storage::Database. See lib/paperclip/storage.rb for details; Paperclip::Storage::Database is at the bottom of the file. I’d love any feedback or suggestions about the usage/design of how the database storage option would work with your application, or on the implementation itself.
I’ll be blogging here in the coming weeks with a detailed explanation of how Paperclip database storage works, and a working sample application that illustrates how to use it.
For now, here’s the usage description from lib/paperclip/storage.rb for specifying database storage for your Paperclip app. You need to follow these steps in addition to the standard Paperclip setup steps from Thoughtbot.
- In your model specify the "database" storage option; for example:
has_attached_file :avatar, :storage => :database
The files will be stored in a new database table named with the plural attachment name
by default, "avatars" in this example.
You need to create this new storage table with at least these columns:
- file_contents
- style
- the primary key for the parent model (e.g. user_id)
Note the "binary" migration will not work for the LONGBLOG type in MySQL for the
file_contents column. You may need to craft a SQL statement for your migration,
depending on which database server you are using. Here's an example migration for MySQL:
create_table :avatars do |t|
t.string :style
t.integer :user_id
t.timestamps
end
execute 'ALTER TABLE avatars ADD COLUMN file_contents LONGBLOB'
You can optionally specify any storage table name you want as follows:
has_attached_file :avatar, :storage => :database,
:database_table => 'avatar_files'- By default, URLs will be set to this pattern:
/:relative_root/:class/:attachment/:id?style=:style
Example:
/app-root-url/users/avatars/23?style=original
The idea here is that to retrieve a file from the database storage, you will need some
controller's code to be executed. Once you pick a controller to use for downloading, you can add this line
to generate the download action for the default URL/action (the plural attachment name),
"avatars" in this example:
downloads_files_for :user, :avatar
Or you can write a download method manually if there are security, logging or other
requirements. If you prefer a different URL for downloading files you can specify that in the model; e.g.:
has_attached_file :avatar, :storage => :database,
:url => '/users/show_avatar/:id/:style'
- Add a route for the download to the controller which will handle downloads, if necessary.
The default URL (/:relative_root/:class/:attachment/:id?style=:style) will be matched by
the default route:
map.connect ':controller/:action/:id'
Tags:paperclip·paperclip·rails·rails
Update June 2009: I just added support to my version of auto_complete to support Rails 2.3 nested attributes; for more details see: http://patshaughnessy.net/repeated_auto_complete. This sample app will still work since the git submodule commands below refer to the old version of my auto_complete plugin; I’ll try to write another sample app or tutorial in the next week or two illustrating how to use Rails 2.3 to build this same app… in a much simpler way! If you download my version of auto_complete now from github, please refer to my description of the usage changes.
In his “Complex Forms” series (part 1, part 2 and part 3) Ryan Bates does a fantastic job explaining how to create a complex form containing a series of parent/child text fields while still using simple, clean code. Ryan also pushed the sample application from the screen cast onto github, here: http://github.com/ryanb/complex-form-examples
Here’s what Ryan’s sample complex form looks like:
One problem I ran into while using Ryan’s suggestions on a complex form I was writing was how to get auto complete behavior to work properly using the auto_complete plugin for fields that are repeated, like the “task” field here. As I explained in a previous blog post, this causes a lot of problems for the auto_complete plugin since the <input id=””> attributes are no longer unique, breaking the javascript used for auto complete. I was able to solve the problem by modifying the auto_complete plugin to generate unique <input id=””> attributes, among other things.
Here I want to take some time to show how to use my modified auto_complete plugin, using the same sample application from Ryan’s screencast. To get started, let’s clone the git repository for the sample app - this command refers to my fork of Ryan's complex-form-examples repository: http://github.com/patshaughnessy/complex-form-examples
$ git clone git://github.com/patshaughnessy/complex-form-examples.git
Initialized empty Git repository in /Users/pat/rails-apps/complex-form-examples/.git/
remote: Counting objects: 192, done.
remote: Compressing objects: 100% (122/122), done.
remote: Total 192 (delta 71), reused 159 (delta 58)
Receiving objects: 100% (192/192), 86.19 KiB | 68 KiB/s, done.
Resolving deltas: 100% (71/71), done.
Ryan had saved various versions of the sample app in different git branches, so to avoid confusion I’ve saved my auto complete related changes in a branch called “auto_complete.” So next you should switch to that branch:
$ git checkout origin/auto_complete
Note: moving to "origin/auto_complete" which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
git checkout -b <new_branch_name>
HEAD is now at 4f3e908... Sample app code changes for auto_complete
Now you will see my changes in Ryans’ code, except for one more detail: I saved my version of the auto_complete plugin in this git repository as a submodule. To get the plugin’s code for this sample app you need to run these commands:
$ git submodule init
Submodule 'vendor/plugins/auto_complete' (git://github.com/patshaughnessy/auto_complete.git) registered for path 'vendor/plugins/auto_complete'
$ git submodule update
Initialized empty Git repository in /Users/pat/rails-apps/complex-form-examples/vendor/plugins/auto_complete/.git/
remote: Counting objects: 22, done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 22 (delta 5), reused 0 (delta 0)
Receiving objects: 100% (22/22), 7.65 KiB, done.
Resolving deltas: 100% (5/5), done.
Submodule path 'vendor/plugins/auto_complete': checked out '0814a25a754a235c5cf6f7a258fa405059a5ca6f'
(Note that normally to install the plugin in your app you would just run “script/plugin install git://github.com/patshaughnessy/auto_complete.git” – the submodule is only present in this sample app.) Now to setup and run the application you just need to:
- Enter your MySQL details in config/database.yml
- Run rake db:migrate
- Run script/server to launch the app
If you enter a few records you should be able to see the auto complete drop down, even for the repeated field:
Let’s review the changes I’ve made to Ryan’s code aside from adding my modified version of auto_complete to vendor/plugins. First, I added the standard auto_complete handlers to projects_controller.rb for both the project and task fields:
class ProjectsController < ApplicationController
auto_complete_for :project, :name
auto_complete_for :task, :name
…
Next I modified the project text field to use auto complete (in views/projects/_form.html.erb):
<p>
<%= f.label :name, "Project:" %>
<%= text_field_with_auto_complete :project, :name, {}, {:method => :get } %>
</p>
These two changes enable auto complete for the single project text field, just the same way you would with any text field and the standard auto_complete plugin. However, to get auto complete to work with the repeated tasks field, we need to use changes I’ve made to auto_complete. First, in helpers/projects_helper.rb change the “fields_for_task” method to use my new auto_complete_fields_for method, like this:
def fields_for_task(task, &block)
new_or_existing = task.new_record? ? 'new' : 'existing'
prefix = "project[#{new_or_existing}_task_attributes][]"
auto_complete_fields_for(prefix, task, &block)
end
This causes my code in auto_complete to provide a custom form builder object, which we can use in the view as follows (views/projects/_task.html.erb):
<% fields_for_task task do |f| -%>
<%= error_messages_for :task, :object => task %>
<%= f.label :name, "Task:" %>
<%= f.text_field_with_auto_complete :task, :name, {}, {:method => :get } %>
<%= link_to_function "remove", "$(this).up('.task').remove()" %>
<% end -%>
Here I’ve called “text_field_with_auto_complete” as a method on the “f” form builder object yielded by fields_for_task. This will cause the auto complete script and HTML to be generated with unique <input id=””> attributes, allowing the auto complete behavior to work properly.
One other change I made was also to helpers/projects_helper.rb:
def add_task_link(name)
link_to_remote "Add a task", :url => {
:controller => "projects",
:action => "add_task_script"
}
end
Here I’ve changed Ryan’s “link_to_function” call to “link_to_remote.” As Ryan explains in part 2 of his complex forms screen cast, link_to_function avoids an AJAX call to the server to obtain the HTML for each new task <input> tag, avoiding unnecessary load on the server since all of the task fields are the same. However, with my changes to auto_complete the HTML generated for the task field contains random numbers which are different for each copy of the field… meaning that we do need a separate call to the server to obtain the task field HTML and script. To handle the call from link_to_remote, I’ve added a new file, views/projects/add_task_script.rjs:
page.insert_html :bottom, :tasks, :partial => 'task', :object => Task.new
… which works essentially the same way as described by Ryan, but is called each time the user clicks “Add a task.”
The last change I made to the sample app is in routes.rb; these changes are required to allow the controller to map the Ajax requests, and to insure that these requests use GET, and not POST HTTP requests:
map.connect 'projects/auto_complete_for_project_name',
:controller => 'projects',
:action => 'auto_complete_for_project_name'
map.connect 'projects/auto_complete_for_task_name',
:controller => 'projects',
:action => 'auto_complete_for_task_name'
map.connect 'projects/add_task_script',
:controller => 'projects',
:action => 'add_task_script'
map.resources :projects,
:collection => {
:auto_complete_for_project_name => :get,
:auto_complete_for_task_name => :get
}
This certainly seems very ugly, and probably could be simplified! But for now, we need this code to avoid problems with CRSF protection; see http://www.ruby-forum.com/topic/128970.
Tags:auto_complete·rails
Update June 2009: I just added support to my version of auto_complete to support Rails 2.3 nested attributes; for more details see: http://patshaughnessy.net/repeated_auto_complete. The basic ideas below still apply, but my implementation of auto_complete has changed, and I’ve also simplified the usage.
In October I described how the auto_complete plugin doesn’t work when text fields are repeated more than once on a complex form. I went on to write a plugin called “repeated_auto_complete” which modified the way the standard auto_complete plugin works and fixed this problem by adding random numbers to <input id=""> attributes among other changes.
Since it’s much cleaner to have a single auto_complete plugin rather than two separate plugins, I’ve merged my changes to auto_complete into the original version, and pushed them to github as a new fork: http://github.com/patshaughnessy/auto_complete
To install and use my modified version of auto_complete first remove the standard auto_complete plugin from your app if necessary, and install with:
script/plugin install git://github.com/patshaughnessy/auto_complete.git
To use auto complete in a complex form, you write “auto_complete_fields_for” or “auto_complete_form_for” in your view, and then call text_field_with_auto_complete on the form builder object, as follows:
<% for person in @group.people %>
<% auto_complete_fields_for "group[person_attributes][]", person do |form| %>
Person <%= person_form.label :name %><br />
<%= form.text_field_with_auto_complete :person, :name, {},
{:method => :get} %>
<% end %>
<% end %>
To understand my changes to the plugin, let’s first look at how the original auto_complete works. If you add this line to your view:
<%= text_field_with_auto_complete :project, :name, {}, {:method => :get } %>
…then you get HTML and script that looks like this (style sheet omitted):
<input id="project_name" name="project[name]" size="30" type="text" />
<div class="auto_complete" id="project_name_auto_complete"></div>
<script type="text/javascript">
//<![CDATA[
var project_name_auto_completer = new Ajax.Autocompleter('project_name',
'project_name_auto_complete', '/projects/auto_complete_for_project_name',
{method:'get'})
//]]>
</script>
The original text_field_with_auto_complete method looked like this:
def text_field_with_auto_complete(object, method, tag_options = {},
completion_options = {})
(completion_options[:skip_style] ? "" : auto_complete_stylesheet) +
text_field(object, method, tag_options) +
content_tag("div", "", :id => "#{object}_#{method}_auto_complete",
:class => "auto_complete") +
auto_complete_field(
"#{object}_#{method}",
{
:url => { :action => "auto_complete_for_#{object}_#{method}" }
}.update(completion_options))
end
You can see that it calls “text_field” in ActionView::Helpers::FormHelper to generate the actual <input> tag for the form, in addition to generating the HTML and script needed for the auto completion behavior.
What I wanted to achieve in the modified plugin was to allow the view to contain code like this:
<% auto_complete_fields_for task do |f| %>
<%= f.label :name, "Task:" %>
<%= f.text_field_with_auto_complete :task, :name, {}, {:method => :get } %>
<% end %>
To make this work, we need a new version of text_field_with_auto_complete that calls text_field from ActionView::Helpers::FormBuilder, and not ActionView::Helpers::FormHelper, generating an <input> tag similar to what this call would generate:
<% fields_for task do |f| %>
<%= f.text_field :name %>
<% end %>
To do this, I first refactored the original text_field_with_auto_complete in auto_complete_macros_helper.rb:
def text_field_with_auto_complete(object, method, tag_options = {},
completion_options = {})
auto_complete_field_with_style_and_script(object, method, tag_options,
completion_options) do
text_field(object, method, tag_options)
end
end
def auto_complete_field_with_style_and_script(object, method,
tag_options = {},
completion_options = {})
(completion_options[:skip_style] ? "" : auto_complete_stylesheet) +
yield +
content_tag("div", "", :id => "#{object}_#{method}_auto_complete",
:class => "auto_complete") +
auto_complete_field(
"#{object}_#{method}",
{
:url => { :action => "auto_complete_for_#{object}_#{method}" }
}.update(completion_options))
end
Here I’ve introduced a new utility function called “auto_complete_field_with_style_and_script” that generates the same Javascript and style sheet for the view as before, but instead calls a block to generate the actual text field. Then I changed text_field_with_auto_complete to call this, providing a block to make the call to “text_field” in ActionView::Helpers::FormHelper with the proper names and options.
Now my new form builder class in auto_complete_form_helper.rb contains a version of text_field_with_auto_complete that looks like this:
def text_field_with_auto_complete(object,
method,
tag_options = {},
completion_options = {})
unique_object_name = "#{object}_#{Object.new.object_id.abs}"
completion_options_for_original_name =
{
:url => { :action => "auto_complete_for_#{object}_#{method}"},
:param_name => "#{object}[#{method}]"
}.update(completion_options)
@template.auto_complete_field_with_style_and_script(
unique_object_name,
method,
tag_options,
completion_options_for_original_name
) do
text_field(method,
{
:id => "#{unique_object_name}_#{method}"
}.update(tag_options))
end
end
Here the call to auto_complete_field_with_style_and_script passes a block that calls the other text_field from ActionView::Helpers::FormBuilder (note the “object” parameter is not present as above).
To allow the text field to be repeated on a complex form, I insure the object’s name is unique by adding a random number to it (“unique_object_name”). This unique name is then passed into both auto_complete_field_with_style_and_script and text_field, insuring that the <input> and related Javascript all work without problems, even if the text field is repeated more than once on the same form.
The last important detail here is that the completion options passed into auto_complete_field_with_style_and_script are generated using the original, unchanged (non-unque) object name, so that the Ajax calls to the server are made using the original name. This means no changes are required on the server side, and the same single line of code in your controller still works as usual:
auto_complete_for :task, :name
Next time I’ll post a sample application that uses this new plugin, and explain what changes you will need to make to your own application for auto_complete in a complex form.
Tags:auto_complete·rails
Update April 2009:
I’ve started working with Mark Bennett on phpunit_setup.inc and moved it up to a github repository. Mark has adapted it to work with Drupal 5, and we’re working on a number of other related ideas as well. I’ll post updates here about our progress.
This month I’ve been experimenting with using testing ideas from Ruby on Rails while developing a Drupal module. To read more, see:
If you want to try this out yourself, follow these instructions:
Edit settings.php and use an array of two values for $db_url:
$db_url["default"] = 'mysql://user:password@localhost/drupal;
$db_url["test"] = 'mysql://user:password@localhost/drupal_test';
Create a new test database in MySQL:
CREATE DATABASE drupal_test DEFAULT CHARACTER SET utf8
COLLATE utf8_unicode_ci;- Download and save phpunit_setup.inc somewhere in your Drupal application; for example in the “includes” folder.
- Include phpunit_setup.inc at the top of each of your PHPUnit test classes. See one of the two articles above for example PHPUnit tests.
Execute your PHPUnit test class from the root folder of your Drupal app:
$ cd /path/to/your/drupal-site
$ phpunit YourClass modules/your_module/YourClassFileName.php
PHPUnit 3.2.21 by Sebastian Bergmann.
..
Time: 0 seconds
OK (2 tests)
For more information about how to run PHPUnit with Drupal, see: Writing your first PHPUnit test in Drupal.
Tags:drupal
Last time I wrote about how to use an entirely separate MySQL database to hold test data for Drupal unit tests, similar to the approach that the Ruby on Rails framework uses for test data. In this post I’ll look at another Rails innovation that can be applied equally well to unit testing with Drupal: running each unit test in a separate database transaction.
First let’s take a quick look at what database transactions are, and how we would use them while running unit tests. In a nutshell, a database transaction is just a way to group a series of SQL operations together and insuring that they are all run together as a single unit – either all of them are executed, or none of them. Let’s take an example. Here are some of the SQL statements Drupal executes when you save a new node in the database:
BEGIN
INSERT INTO node_revisions (nid, uid, title, body, teaser, log, timestamp...
INSERT INTO node (vid, type, language, title, uid, status, created, changed...
UPDATE node_revisions SET nid = 2 WHERE vid = 2
COMMIT
Normally Drupal does not use transactions, but I've inserted the “BEGIN” and “COMMIT” commands here as an example: the transaction starts with the BEGIN command, and ends with the COMMIT command. When MySQL receives the COMMIT command, it allows other database clients (future Drupal HTTP requests, or possibly future command line unit tests) to see the new inserted and updated node data. However, if the transaction were rolled back like this:
BEGIN
INSERT INTO node_revisions (nid, uid, title, body, teaser, log, timestamp...
INSERT INTO node (vid, type, language, title, uid, status, created, changed...
UPDATE node_revisions SET nid = 2 WHERE vid = 2
ROLLBACK
… then none of the changes would be made to the node and node_revisions tables. Instead when MySQL receives the ROLLBACK command it will discard the changes and these tables will appear the same way they did before the transaction started. Therefore, by running each unit test in a separate transaction and rolling it back at the end of each test, we can insure that any changes made to the database by that test are discarded… before the next test is run. Below I’ll explain how to actually do this with PHPUnit and Drupal.
But first let me quickly mention another huge benefit to using transactions with unit test suites: test performance. To learn more about why your tests will run a lot faster using MySQL transactions, read this great article by Mike Clark from the period when Rails 1.0 was released, way back in 2005. What Mike wrote about Rails in 2005 is still true today for Drupal: your unit tests will run faster because fewer SQL statements are required. You won’t need to execute DELETE SQL statements to remove the data after each test since rolling back each transaction accomplishes the same thing.
Now let’s get it to work with Drupal… first let me add a second unit test to my simple PHPUnit test class from last time:
<?php
require_once './includes/phpunit_setup.inc';
class TestDataExampleTest2 extends PHPUnit_Framework_TestCase
{
public function create_test_blog_post()
{
$node = new stdClass();
$node->title = "This is a blog post";
$node->body = "This is the body of the post";
$node->type = "Story";
$node->promote = 1;
node_save($node);
return $node;
}
public function test_there_is_one_post()
{
$this->create_test_blog_post();
$this->assertEquals(1, db_result(db_query("SELECT COUNT(*) FROM {NODE}")));
}
public function test_there_are_two_posts()
{
$this->create_test_blog_post();
$this->create_test_blog_post();
$this->assertEquals(2, db_result(db_query("SELECT COUNT(*) FROM {NODE}")));
}
}
?>
In this example, I use the phpunit_setup.inc file I wrote in my last post to clear out and setup a new Drupal schema each time I run the tests. Even though I have a clean test database each time I run PHPUnit, without using transactions one of these two unit tests will fail since each one creates its own test data, and assumes no other test data exist in the node table:
$ phpunit TestDataExampleTest2
modules/test_data_module/TestDataExampleTest2.php
PHPUnit 3.2.21 by Sebastian Bergmann.
.F
Time: 0 seconds
There was 1 failure:
1) test_there_are_two_posts(TestDataExampleTest2)
Failed asserting that <string:3> matches expected value <integer:2>.
/Users/pat/htdocs/drupal4/modules/test_data_module/TestDataExampleTest2.php:24
FAILURES!
Tests: 2, Failures: 1.
Here the second test fails since the blog post created in the first test is still present in the database. The simplest way to start a new database transaction before each test is run, and to rollback after each test is completed, is with the PHPUnit setup/teardown methods as follows:
public function setup()
{
db_query("BEGIN");
}
public function teardown()
{
db_query("ROLLBACK");
}
If you add these functions to the “TestDataExampleTest2” class above both tests should now pass since the ROLLBACK call will delete the nodes created by each test each time teardown is called…
$ phpunit TestDataExampleTest2
modules/test_data_module/TestDataExampleTest2.php
PHPUnit 3.2.21 by Sebastian Bergmann.
.F
…
FAILURES!
Tests: 2, Failures: 1.
Wait… what happened? It failed!
The problem is that MySQL does not support transactions using the MyISAM database engine, which is what Drupal uses by default. What we need to do is to convert all of the Drupal MySQL tables to use the InnoDB database engine instead. Unfortunately, there are many implications to using InnoDB vs. MyISAM in Drupal or with any MySQL based application. See "MySQL InnoDB: performance gains as well as some pitfalls" to read more. Specifically, there can be performance issues and degradation when using InnoDB incorrectly, or depending on the type of application you have. Drupal was actually designed and developed with MyISAM in mind, and not InnoDB, although there is some chance this might change for Drupal 7 someday.
Despite all of this, using InnoDB in a test database is a great idea since you will get all of the benefits of isolating tests from each other without having to worry about how InnoDB will effect your production site’s performance. In fact, the performance of your tests will actually be dramatically improved, as Mike Clark explained.
With all of this in mind, I wrote some code to convert the newly created Drupal tables in the test database from MyISAM to InnoDB right after we clear out and reload the test database. Here’s how it works; this code is from phpunit_setup.inc, which I included at the top of my PHPUnit test file:
function enable_mysql_transactions()
{
convert_test_tables_to_innodb();
db_query("SET AUTOCOMMIT = 0");
}
function convert_test_tables_to_innodb()
{
each_table('convert_to_innodb');
}
function each_table($table_callback)
{
global $db_url;
$url = parse_url($db_url['test']);
$database = substr($url['path'], 1);
$result = db_query("SELECT table_name FROM information_schema.tables
WHERE table_schema = '$database'");
while ($table = db_result($result)) {
$table_callback($table);
}
}
function convert_to_innodb($table)
{
db_query("ALTER TABLE $table ENGINE = INNODB");
}
This iterates over the Drupal tables in the test database and executes ALTER TABLE … ENGINE = INNODB on each one. The SET AUTOCOMMIT=0 command is used to prevent SQL statements from being committed immediately after they are executed, and to allow the InnoDB transactions to work properly.
To repeat and summarize how to employ and separate MySQL test database and transactions in your PHPUnit tests for Drupal, just follow these steps:
Edit settings.php and use an array of two values for $db_url:
$db_url["default"] = 'mysql://user:password@localhost/drupal;
$db_url["test"] = 'mysql://user:password@localhost/drupal_test';
Create a new test database in MySQL:
CREATE DATABASE drupal_test DEFAULT CHARACTER SET utf8
COLLATE utf8_unicode_ci;- Download and save phpunit_setup.inc somewhere in your Drupal application; for example in the “includes” folder.
- Include phpunit_setup.inc at the top of each of your PHPUnit test classes.
Execute your PHPUnit test class from the root folder of your Drupal app:
$ cd /path/to/your/drupal-site
$ phpunit YourClass modules/your_module/YourClassFileName.php
PHPUnit 3.2.21 by Sebastian Bergmann.
..
Time: 0 seconds
OK (2 tests)
Here’s my finished test class:
<?php
require_once './includes/phpunit_setup.inc';
class TestDataExampleTest2 extends PHPUnit_Framework_TestCase
{
public function setup()
{
db_query("BEGIN");
}
public function teardown()
{
db_query("ROLLBACK");
}
public function create_test_blog_post()
{
$node = new stdClass();
$node->title = "This is a blog post";
$node->body = "This is the body of the post";
$node->type = "Story";
$node->promote = 1;
node_save($node);
return $node;
}
public function test_there_is_one_post()
{
$this->create_test_blog_post();
$this->assertEquals(1, db_result(db_query("SELECT COUNT(*) FROM {NODE}")));
}
public function test_there_are_two_posts()
{
$this->create_test_blog_post();
$this->create_test_blog_post();
$this->assertEquals(2, db_result(db_query("SELECT COUNT(*) FROM {NODE}")));
}
}
?>
Tags:drupal
In my last few posts I used Test Driven Development (TDD) to write a very simple Drupal module and showed how TDD helped to keep my custom code decoupled from the Drupal framework. This time I want to take a closer look at the biggest headache I ran into while using TDD with Drupal: handling the test data. When I started to write unit tests for my example module I ran into trouble creating test data inside each test, since I found that PHPUnit stopped executing the test each time there was a failing assertion, meaning that test data weren’t cleaned up after a test failure. I was able to avoid this problem by creating and deleting the test data before and after each test was run using the setup/teardown methods from PHPUnit. But this solution brings along different problems with it:
- In setup() I need to create all of the test data that every test will use since it is called every time, which becomes a performance problem as the number of tests increases.
- Teardown() still won’t be called if there are any PHP syntax errors in my test or production code, which happens a lot if I’m really using TDD.
- There’s no way to create different test data for different tests
- Worst of all, any existing data in my Drupal database might cause the tests to fail, and vice-versa: the test data might interfere with my development work.
We need a better approach for handling test data. Rather than reinventing the wheel, let’s take a look at the Ruby on Rails framework for some inspiration and see if we can emulate the way Rails handles test data using PHP and Drupal. How does Rails handle test data? First of all, each Rails application has multiple, different databases setup: one for development, one for production, and a third for testing at a minimum. Every time you run a unit test in Rails, the test database is manipulated as follows:
- Rails deletes the existing contents of the test database, if any.
- Rails loads your test database with an empty copy of your application’s database schema (tables, columns, indices, etc.).
- Finally Rails runs each of the unit tests targeting this empty test database, by default each test within a separate database transaction (more on this in my next post).
How can we do this with Drupal? If you take a close look at the SimpleTest module, you’ll see that it uses some tricks to create a test copy of the Drupal schema using the “database prefix” feature of Drupal. While this works fine, I decided to see if I could directly follow the Rails pattern of having a completely separate MySQL database to use for testing. Let’s use PHPUnit directly on Drupal from the command line again as I did before. Here’s a very simple PHPUnit test:
<?php
require_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);
class TestDataExampleTest extends PHPUnit_Framework_TestCase
{
public function create_test_blog_post()
{
$node = new stdClass();
$node->title = "This is a blog post";
$node->body = "This is the body of the post";
$node->type = "Story";
$node->promote = 1;
node_save($node);
return $node;
}
public function test_there_is_one_post()
{
$this->create_test_blog_post();
$this->assertEquals(1, db_result(db_query("SELECT COUNT(*) FROM {NODE}")));
}
}
?>
As I explained in December, we have to run the unit test from the Drupal root folder as follows (replace “modules/test_data_module/TestDataExampleTest.php” with the path to the test file):
$ cd /path/to/my-drupal-site
$ phpunit TestDataExampleTest modules/test_data_module/TestDataExampleTest.php
The “test_there_is_one_post” unit test will create a test blog post record in the node table, and then count the number of nodes in the database and assert that there is exactly one. Obviously this will fail if there are any existing node records in my Drupal database, or if I even just run the test more than once:
$ phpunit TestDataExampleTest modules/test_data_module/TestDataExampleTest.php
PHPUnit 3.2.21 by Sebastian Bergmann.
F
Time: 0 seconds
There was 1 failure:
1) test_there_is_one_post(TestDataExampleTest)
Failed asserting that <string:6> matches expected value <integer:1>.
/Users/pat/htdocs/drupal4/TestDataExampleTest.php:19
FAILURES!
Tests: 1, Failures: 1.
This failure is actually good: this test is intentionally dependent on the contents of the test database. Later if we can get this test to pass then we know we have properly initialized the contents of the database without resorting to the setup/teardown solution from last time.
Let’s get started by creating a real test database using the MySQL command line:
mysql> CREATE DATABASE drupal_test DEFAULT CHARACTER SET utf8
COLLATE utf8_unicode_ci;
Query OK, 1 row affected (0.00 sec)
Now, how can we get Drupal to use this database instead of the normal one? Let’s try converting the $db_url value in settings.php into an array, like this:
$db_url["default"] = 'mysql://user:password@localhost/drupal;
$db_url["test"] = 'mysql://user:password@localhost/drupal_test';
Here I’ve renamed the original $db_url variable to $db_url[“default”], and created a new entry for the test database. Now Drupal can run against either the original database, or the test database as we wish.
The next step is to load the test database with an empty copy of the Drupal database schema. In the Rails world, there are Ruby functions that export the development database schema, and then reload it into the test database. In Drupal, the database schema is created automatically by PHP functions when you install the application for the first time. The SimpleTest module also does the same thing before running its tests. Looking at code in install.php from the Drupal installation process, and also in drupal_web_test_case.php from SimpleTest I came up with this solution:
function create_test_drupal_schema()
{
include_once './includes/install.inc';
drupal_install_system();
drupal_install_modules(drupal_verify_profile('default', 'en'));
$task = 'profile';
default_profile_tasks($task, '');
menu_rebuild();
actions_synchronize();
_drupal_flush_css_js();
variable_set('user_mail_status_activated_notify', FALSE);
$account = user_load(1);
$merge_data = array('name' => 'admin', 'pass' => 'test', 'roles' => array(),
'status' => 1);
user_save($account, $merge_data);
}
Here drupal_install_system() and drupal_install_modules() will create most of the empty tables we need just the way they do when you install Drupal. The other calls I took from DrupalWebTestCase->setup() in SimpleTest to create some initial data Drupal requires to function properly, like the menu for example. The last few lines I wrote to setup the admin user properly, and to avoid sending emails to the admin during this process.
The last piece of the puzzle is to find a way to clear out the test database before each test run. To do that, I wrote this code to iterate over all of the tables in the test database and drop them:
function drop_test_tables()
{
each_table('drop');
}
function each_table($table_callback)
{
global $db_url;
$url = parse_url($db_url['test']);
$database = substr($url['path'], 1);
$result = db_query("SELECT table_name FROM information_schema.tables
WHERE table_schema = '$database'");
while ($table = db_result($result)) {
$table_callback($table);
}
}
function drop($table)
{
db_query("DROP TABLE $table");
}
To put it all together we just need to call db_set_active(“test”) and call all of this code before our test runs:
db_set_active("test");
drop_test_tables();
create_test_drupal_schema();
Here db_set_active(“test”) tells Drupal we want to use the test database instead of the actual database. After switching to the test database we drop any existing tables that may exist there, and then create a new, empty Drupal schema.
To avoid cluttering my PHPUnit test file, and to be able to reuse this code in many PHPUnit tests, I moved the test database setup into a new include file called: phpunit_setup.inc. I also added some validation code to phpunit_setup.inc to perform a sanity check so you don’t accidentally drop all of the tables in your main Drupal database, and to make it easier to avoid mistakes with $db_url in settings.php. The code requires that "test" be present in the test database name. I also added code to enable database transactions in the test database, which I will discuss in my next post.
To try this out on your system, just download phpunit_setup.inc and then include it at the top of your PHPUnit test file, like this:
<?php
require_once './includes/phpunit_setup.inc';
class TestDataExampleTest extends PHPUnit_Framework_TestCase
{
public function create_test_blog_post()
{
$node = new stdClass();
$node->title = "This is a blog post";
$node->body = "This is the body of the post";
$node->type = "Story";
$node->promote = 1;
node_save($node);
return $node;
}
public function test_there_is_one_post()
{
$this->create_test_blog_post();
$this->assertEquals(1, db_result(db_query("SELECT COUNT(*) FROM {NODE}")));
}
}
?>
The require_once statement above assumes you downloaded phpunit_setup.inc into the includes folder. If you put it somewhere else, just update require_once as necessary. Now the test passes every time:
$ cd /path/to/my-drupal-site
$ phpunit TestDataExampleTest modules/test_data_module/TestDataExampleTest.php
PHPUnit 3.2.21 by Sebastian Bergmann.
.
Time: 0 seconds
OK (1 test)
One important detail I’ve glossed over here is that the test only passes because it is the only test I’m running at all in this database. If there were a second test with it’s own test data and assumptions about what data were present, then there would be failures depending on what data each test expected, and which test ran first. Next time, I’ll show how Rails solved this problem using database transactions and show how to use them with Drupal unit tests.
Tags:drupal
