Pat Shaughnessy

Update November 2009: My View Mapper gem now supports generating scaffolding code for a complex form with auto_complete behavior like the one I describe below right inside your application, using your models and attributes. For more info see: http://patshaughnessy.net/2009/11/25/scaffolding-for-auto-complete-on-a-complex-nested-form. You can read more about my fork of the auto_complete plugin here: http://patshaughnessy.net/repeated_auto_complete.

In his “Complex Forms” series (part 1, part 2 and part 3) Ryan Bates does a fantastic job explaining how to create a complex form containing a series of parent/child text fields while still using simple, clean code. Ryan also pushed the sample application from the screen cast onto github, here: http://github.com/ryanb/complex-form-examples

Here’s what Ryan’s sample complex form looks like:

One problem I ran into while using Ryan’s suggestions on a complex form I was writing was how to get auto complete behavior to work properly using the auto_complete plugin for fields that are repeated, like the “task” field here. As I explained in a previous blog post, this causes a lot of problems for the auto_complete plugin since the <input id=””> attributes are no longer unique, breaking the javascript used for auto complete. I was able to solve the problem by modifying the auto_complete plugin to generate unique <input id=””> attributes, among other things.

Here I want to take some time to show how to use my modified auto_complete plugin, using the same sample application from Ryan’s screencast. To get started, let’s clone the git repository for the sample app - this command refers to my fork of Ryan's complex-form-examples repository: http://github.com/patshaughnessy/complex-form-examples

$ git clone git://github.com/patshaughnessy/complex-form-examples.git
Initialized empty Git repository in /Users/pat/rails-apps/complex-form-examples/.git/
remote: Counting objects: 192, done.
remote: Compressing objects: 100% (122/122), done.
remote: Total 192 (delta 71), reused 159 (delta 58)
Receiving objects: 100% (192/192), 86.19 KiB | 68 KiB/s, done.
Resolving deltas: 100% (71/71), done.

Ryan had saved various versions of the sample app in different git branches, so to avoid confusion I’ve saved my auto complete related changes in a branch called “auto_complete.” So next you should switch to that branch:

$ cd complex-form-examples
$ git checkout origin/auto_complete
Note: moving to "origin/auto_complete" which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
  git checkout -b <new_branch_name>
HEAD is now at 4f3e908... Sample app code changes for auto_complete

Now you will see my changes in Ryans’ code, except for one more detail: I saved my version of the auto_complete plugin in this git repository as a submodule. To get the plugin’s code for this sample app you need to run these commands:

$ git submodule init
Submodule 'vendor/plugins/auto_complete' (git://github.com/patshaughnessy/auto_complete.git) registered for path 'vendor/plugins/auto_complete'
$ git submodule update
Initialized empty Git repository in /Users/pat/rails-apps/complex-form-examples/vendor/plugins/auto_complete/.git/
remote: Counting objects: 22, done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 22 (delta 5), reused 0 (delta 0)
Receiving objects: 100% (22/22), 7.65 KiB, done.
Resolving deltas: 100% (5/5), done.
Submodule path 'vendor/plugins/auto_complete': checked out '0814a25a754a235c5cf6f7a258fa405059a5ca6f'

(Note that normally to install the plugin in your app you would just run “script/plugin install git://github.com/patshaughnessy/auto_complete.git” – the submodule is only present in this sample app.) Now to setup and run the application you just need to:

1. Enter your MySQL details in config/database.yml
2. Run rake db:migrate
3. Run script/server to launch the app

If you enter a few records you should be able to see the auto complete drop down, even for the repeated field:

Let’s review the changes I’ve made to Ryan’s code aside from adding my modified version of auto_complete to vendor/plugins. First, I added the standard auto_complete handlers to projects_controller.rb for both the project and task fields:

class ProjectsController < ApplicationController
  auto_complete_for :project, :name
  auto_complete_for :task, :name
…

Next I modified the project text field to use auto complete (in views/projects/_form.html.erb):

<p>
  <%= f.label :name, "Project:" %>
  <%= text_field_with_auto_complete :project, :name, {}, {:method => :get } %>
</p>

These two changes enable auto complete for the single project text field, just the same way you would with any text field and the standard auto_complete plugin. However, to get auto complete to work with the repeated tasks field, we need to use changes I’ve made to auto_complete. First, in helpers/projects_helper.rb change the “fields_for_task” method to use my new auto_complete_fields_for method, like this:

def fields_for_task(task, &block)
  new_or_existing = task.new_record? ? 'new' : 'existing'
  prefix = "project[#{new_or_existing}_task_attributes][]"
  auto_complete_fields_for(prefix, task, &block)
end

This causes my code in auto_complete to provide a custom form builder object, which we can use in the view as follows (views/projects/_task.html.erb):

<% fields_for_task task do |f| -%>
  <%= error_messages_for :task, :object => task %>
  <%= f.label :name, "Task:" %>
  <%= f.text_field_with_auto_complete :task, :name, {}, {:method => :get } %>
  <%= link_to_function "remove", "$(this).up('.task').remove()" %>
<% end -%>

Here I’ve called “text_field_with_auto_complete” as a method on the “f” form builder object yielded by fields_for_task. This will cause the auto complete script and HTML to be generated with unique <input id=””> attributes, allowing the auto complete behavior to work properly.

One other change I made was also to helpers/projects_helper.rb:

def add_task_link(name)
  link_to_remote "Add a task", :url => {
                                 :controller => "projects",
                                 :action => "add_task_script"
                               }
end

Here I’ve changed Ryan’s “link_to_function” call to “link_to_remote.” As Ryan explains in part 2 of his complex forms screen cast, link_to_function avoids an AJAX call to the server to obtain the HTML for each new task <input> tag, avoiding unnecessary load on the server since all of the task fields are the same. However, with my changes to auto_complete the HTML generated for the task field contains random numbers which are different for each copy of the field… meaning that we do need a separate call to the server to obtain the task field HTML and script. To handle the call from link_to_remote, I’ve added a new file, views/projects/add_task_script.rjs:

page.insert_html :bottom, :tasks, :partial => 'task', :object => Task.new

… which works essentially the same way as described by Ryan, but is called each time the user clicks “Add a task.”

The last change I made to the sample app is in routes.rb; these changes are required to allow the controller to map the Ajax requests, and to insure that these requests use GET, and not POST HTTP requests:

map.connect 'projects/auto_complete_for_project_name',
            :controller => 'projects',
            :action => 'auto_complete_for_project_name'
map.connect 'projects/auto_complete_for_task_name',
            :controller => 'projects',
            :action => 'auto_complete_for_task_name'
map.connect 'projects/add_task_script',
            :controller => 'projects',
            :action => 'add_task_script'
map.resources :projects,
              :collection => {
                :auto_complete_for_project_name => :get,
                :auto_complete_for_task_name => :get
              }

This certainly seems very ugly, and probably could be simplified! But for now, we need this code to avoid problems with CRSF protection; see http://www.ruby-forum.com/topic/128970.

Update June 2009: I just added support to my version of auto_complete to support Rails 2.3 nested attributes; for more details see: http://patshaughnessy.net/repeated_auto_complete. The basic ideas below still apply, but my implementation of auto_complete has changed, and I’ve also simplified the usage.

In October I described how the auto_complete plugin doesn’t work when text fields are repeated more than once on a complex form. I went on to write a plugin called “repeated_auto_complete&rdquo; which modified the way the standard auto_complete plugin works and fixed this problem by adding random numbers to <input id=""> attributes among other changes.

Since it’s much cleaner to have a single auto_complete plugin rather than two separate plugins, I’ve merged my changes to auto_complete into the original version, and pushed them to github as a new fork: http://github.com/patshaughnessy/auto_complete

To install and use my modified version of auto_complete first remove the standard auto_complete plugin from your app if necessary, and install with:

script/plugin install git://github.com/patshaughnessy/auto_complete.git

To use auto complete in a complex form, you write “auto_complete_fields_for” or “auto_complete_form_for” in your view, and then call text_field_with_auto_complete on the form builder object, as follows:

<% for person in @group.people %>
  <% auto_complete_fields_for "group[person_attributes][]", person do |form| %>
    Person <%= person_form.label :name %><br />
    <%= form.text_field_with_auto_complete :person, :name, {},
                                           {:method => :get}  %>
  <% end %>
<% end %>

To understand my changes to the plugin, let’s first look at how the original auto_complete works. If you add this line to your view:

<%= text_field_with_auto_complete :project, :name, {}, {:method => :get } %>

…then you get HTML and script that looks like this (style sheet omitted):

<input id="project_name" name="project[name]" size="30" type="text" />
<div class="auto_complete" id="project_name_auto_complete"></div>
<script type="text/javascript">
//<![CDATA[
var project_name_auto_completer = new Ajax.Autocompleter('project_name',
'project_name_auto_complete', '/projects/auto_complete_for_project_name',
{method:'get'})
//]]>
</script>

The original text_field_with_auto_complete method looked like this:

def text_field_with_auto_complete(object, method, tag_options = {},
                                  completion_options = {})
    (completion_options[:skip_style] ? "" : auto_complete_stylesheet) +
    text_field(object, method, tag_options) +
    content_tag("div", "", :id => "#{object}_#{method}_auto_complete",
                :class => "auto_complete") +
    auto_complete_field(
        "#{object}_#{method}",
        { 
          :url => { :action => "auto_complete_for_#{object}_#{method}" }
        }.update(completion_options))
  end

You can see that it calls “text_field” in ActionView::Helpers::FormHelper to generate the actual <input> tag for the form, in addition to generating the HTML and script needed for the auto completion behavior.

What I wanted to achieve in the modified plugin was to allow the view to contain code like this:

<% auto_complete_fields_for task do |f| %>
  <%= f.label :name, "Task:" %>
  <%= f.text_field_with_auto_complete :task, :name, {}, {:method => :get } %>
<% end %>

To make this work, we need a new version of text_field_with_auto_complete that calls text_field from ActionView::Helpers::FormBuilder, and not ActionView::Helpers::FormHelper, generating an <input> tag similar to what this call would generate:

<% fields_for task do |f| %>
  <%= f.text_field :name %>
<% end %>

To do this, I first refactored the original text_field_with_auto_complete in auto_complete_macros_helper.rb:

def text_field_with_auto_complete(object, method, tag_options = {},
                                  completion_options = {})
  auto_complete_field_with_style_and_script(object, method, tag_options,
                                            completion_options) do
    text_field(object, method, tag_options)
  end
end

def auto_complete_field_with_style_and_script(object, method,
                                              tag_options = {},
                                              completion_options = {})
  (completion_options[:skip_style] ? "" : auto_complete_stylesheet) +
  yield +
  content_tag("div", "", :id => "#{object}_#{method}_auto_complete",
              :class => "auto_complete") +
  auto_complete_field(
    "#{object}_#{method}",
    {
      :url => { :action => "auto_complete_for_#{object}_#{method}" } 
    }.update(completion_options))
end

Here I’ve introduced a new utility function called “auto_complete_field_with_style_and_script” that generates the same Javascript and style sheet for the view as before, but instead calls a block to generate the actual text field. Then I changed text_field_with_auto_complete to call this, providing a block to make the call to “text_field” in ActionView::Helpers::FormHelper with the proper names and options.

Now my new form builder class in auto_complete_form_helper.rb contains a version of text_field_with_auto_complete that looks like this:

def text_field_with_auto_complete(object,
                                  method,
                                  tag_options = {},
                                  completion_options = {})
  unique_object_name = "#{object}_#{Object.new.object_id.abs}"
  completion_options_for_original_name =
    {
      :url => { :action => "auto_complete_for_#{object}_#{method}"},
      :param_name => "#{object}[#{method}]"
    }.update(completion_options)
  @template.auto_complete_field_with_style_and_script(
        unique_object_name,
        method,
        tag_options,
        completion_options_for_original_name
  ) do
    text_field(method,
               {
                 :id => "#{unique_object_name}_#{method}"
               }.update(tag_options))
  end
end

Here the call to auto_complete_field_with_style_and_script passes a block that calls the other text_field from ActionView::Helpers::FormBuilder (note the “object” parameter is not present as above).

To allow the text field to be repeated on a complex form, I insure the object’s name is unique by adding a random number to it (“unique_object_name”). This unique name is then passed into both auto_complete_field_with_style_and_script and text_field, insuring that the <input> and related Javascript all work without problems, even if the text field is repeated more than once on the same form.

The last important detail here is that the completion options passed into auto_complete_field_with_style_and_script are generated using the original, unchanged (non-unque) object name, so that the Ajax calls to the server are made using the original name. This means no changes are required on the server side, and the same single line of code in your controller still works as usual:

auto_complete_for :task, :name

Next time I’ll post a sample application that uses this new plugin, and explain what changes you will need to make to your own application for auto_complete in a complex form.

Update April 2009:
I’ve started working with Mark Bennett on phpunit_setup.inc and moved it up to a github repository. Mark has adapted it to work with Drupal 5, and we’re working on a number of other related ideas as well. I’ll post updates here about our progress.

This month I’ve been experimenting with using testing ideas from Ruby on Rails while developing a Drupal module. To read more, see:

• Using MySQL transactions with Drupal unit tests
• Using a test database with Drupal unit tests

If you want to try this out yourself, follow these instructions:

1. Edit settings.php and use an array of two values for $db_url:

   $db_url["default"] = 'mysql://user:password@localhost/drupal;
   $db_url["test"] = 'mysql://user:password@localhost/drupal_test';

2. Create a new test database in MySQL:

   CREATE DATABASE drupal_test DEFAULT CHARACTER SET utf8
                               COLLATE utf8_unicode_ci;

3. Download and save phpunit_setup.inc somewhere in your Drupal application; for example in the “includes” folder.
4. Include phpunit_setup.inc at the top of each of your PHPUnit test classes. See one of the two articles above for example PHPUnit tests.

5. Execute your PHPUnit test class from the root folder of your Drupal app:

   $ cd /path/to/your/drupal-site
   $ phpunit YourClass modules/your_module/YourClassFileName.php 
   PHPUnit 3.2.21 by Sebastian Bergmann.
   ..
   Time: 0 seconds
   OK (2 tests)

For more information about how to run PHPUnit with Drupal, see: Writing your first PHPUnit test in Drupal.

Last time I wrote about how to use an entirely separate MySQL database to hold test data for Drupal unit tests, similar to the approach that the Ruby on Rails framework uses for test data. In this post I’ll look at another Rails innovation that can be applied equally well to unit testing with Drupal: running each unit test in a separate database transaction.

First let’s take a quick look at what database transactions are, and how we would use them while running unit tests. In a nutshell, a database transaction is just a way to group a series of SQL operations together and insuring that they are all run together as a single unit – either all of them are executed, or none of them. Let’s take an example. Here are some of the SQL statements Drupal executes when you save a new node in the database:

BEGIN
INSERT INTO node_revisions (nid, uid, title, body, teaser, log, timestamp...
INSERT INTO node (vid, type, language, title, uid, status, created, changed...
UPDATE node_revisions SET nid = 2 WHERE vid = 2
COMMIT

Normally Drupal does not use transactions, but I've inserted the “BEGIN” and “COMMIT” commands here as an example: the transaction starts with the BEGIN command, and ends with the COMMIT command. When MySQL receives the COMMIT command, it allows other database clients (future Drupal HTTP requests, or possibly future command line unit tests) to see the new inserted and updated node data. However, if the transaction were rolled back like this:

BEGIN
INSERT INTO node_revisions (nid, uid, title, body, teaser, log, timestamp...
INSERT INTO node (vid, type, language, title, uid, status, created, changed...
UPDATE node_revisions SET nid = 2 WHERE vid = 2
ROLLBACK

… then none of the changes would be made to the node and node_revisions tables. Instead when MySQL receives the ROLLBACK command it will discard the changes and these tables will appear the same way they did before the transaction started. Therefore, by running each unit test in a separate transaction and rolling it back at the end of each test, we can insure that any changes made to the database by that test are discarded… before the next test is run. Below I’ll explain how to actually do this with PHPUnit and Drupal.

But first let me quickly mention another huge benefit to using transactions with unit test suites: test performance. To learn more about why your tests will run a lot faster using MySQL transactions, read this great article by Mike Clark from the period when Rails 1.0 was released, way back in 2005. What Mike wrote about Rails in 2005 is still true today for Drupal: your unit tests will run faster because fewer SQL statements are required. You won’t need to execute DELETE SQL statements to remove the data after each test since rolling back each transaction accomplishes the same thing.

Now let’s get it to work with Drupal… first let me add a second unit test to my simple PHPUnit test class from last time:

<?php
require_once './includes/phpunit_setup.inc';
class TestDataExampleTest2 extends PHPUnit_Framework_TestCase
{  
  public function create_test_blog_post()
  {
    $node = new stdClass();
    $node->title = "This is a blog post";
    $node->body = "This is the body of the post";
    $node->type = "Story";
    $node->promote = 1;
    node_save($node);
    return $node;
  }
  public function test_there_is_one_post()
  {
    $this->create_test_blog_post();
    $this->assertEquals(1, db_result(db_query("SELECT COUNT(*) FROM {NODE}")));
  }
  public function test_there_are_two_posts()
  {
    $this->create_test_blog_post();
    $this->create_test_blog_post();
    $this->assertEquals(2, db_result(db_query("SELECT COUNT(*) FROM {NODE}")));
  }
}
?>

In this example, I use the phpunit_setup.inc file I wrote in my last post to clear out and setup a new Drupal schema each time I run the tests. Even though I have a clean test database each time I run PHPUnit, without using transactions one of these two unit tests will fail since each one creates its own test data, and assumes no other test data exist in the node table:

$ phpunit TestDataExampleTest2
          modules/test_data_module/TestDataExampleTest2.php 
PHPUnit 3.2.21 by Sebastian Bergmann.
.F
Time: 0 seconds
There was 1 failure:
1) test_there_are_two_posts(TestDataExampleTest2)
Failed asserting that <string:3> matches expected value <integer:2>.
/Users/pat/htdocs/drupal4/modules/test_data_module/TestDataExampleTest2.php:24
FAILURES!
Tests: 2, Failures: 1.

Here the second test fails since the blog post created in the first test is still present in the database. The simplest way to start a new database transaction before each test is run, and to rollback after each test is completed, is with the PHPUnit setup/teardown methods as follows:

public function setup()
{
  db_query("BEGIN");
}
public function teardown()
{
  db_query("ROLLBACK");
}

If you add these functions to the “TestDataExampleTest2” class above both tests should now pass since the ROLLBACK call will delete the nodes created by each test each time teardown is called…

$ phpunit TestDataExampleTest2
          modules/test_data_module/TestDataExampleTest2.php 
PHPUnit 3.2.21 by Sebastian Bergmann.
.F
…
FAILURES!
Tests: 2, Failures: 1.

Wait… what happened? It failed!

The problem is that MySQL does not support transactions using the MyISAM database engine, which is what Drupal uses by default. What we need to do is to convert all of the Drupal MySQL tables to use the InnoDB database engine instead. Unfortunately, there are many implications to using InnoDB vs. MyISAM in Drupal or with any MySQL based application. See "MySQL InnoDB: performance gains as well as some pitfalls" to read more. Specifically, there can be performance issues and degradation when using InnoDB incorrectly, or depending on the type of application you have. Drupal was actually designed and developed with MyISAM in mind, and not InnoDB, although there is some chance this might change for Drupal 7 someday.

Despite all of this, using InnoDB in a test database is a great idea since you will get all of the benefits of isolating tests from each other without having to worry about how InnoDB will effect your production site’s performance. In fact, the performance of your tests will actually be dramatically improved, as Mike Clark explained.

With all of this in mind, I wrote some code to convert the newly created Drupal tables in the test database from MyISAM to InnoDB right after we clear out and reload the test database. Here’s how it works; this code is from phpunit_setup.inc, which I included at the top of my PHPUnit test file:

function enable_mysql_transactions()
{
  convert_test_tables_to_innodb();
  db_query("SET AUTOCOMMIT = 0");  
}
function convert_test_tables_to_innodb()
{
  each_table('convert_to_innodb');  
} 
function each_table($table_callback)
{
  global $db_url;
  $url = parse_url($db_url['test']);
  $database = substr($url['path'], 1);
  $result = db_query("SELECT table_name FROM information_schema.tables
                      WHERE table_schema = '$database'");
  while ($table = db_result($result)) {
    $table_callback($table);
  }
}
function convert_to_innodb($table)
{
  db_query("ALTER TABLE $table ENGINE = INNODB");
}

This iterates over the Drupal tables in the test database and executes ALTER TABLE … ENGINE = INNODB on each one. The SET AUTOCOMMIT=0 command is used to prevent SQL statements from being committed immediately after they are executed, and to allow the InnoDB transactions to work properly.

To repeat and summarize how to employ and separate MySQL test database and transactions in your PHPUnit tests for Drupal, just follow these steps:

1. Edit settings.php and use an array of two values for $db_url:

   $db_url["default"] = 'mysql://user:password@localhost/drupal;
   $db_url["test"] = 'mysql://user:password@localhost/drupal_test';

2. Create a new test database in MySQL:

   CREATE DATABASE drupal_test DEFAULT CHARACTER SET utf8
                               COLLATE utf8_unicode_ci;

3. Download and save phpunit_setup.inc somewhere in your Drupal application; for example in the “includes” folder.
4. Include phpunit_setup.inc at the top of each of your PHPUnit test classes.

5. Execute your PHPUnit test class from the root folder of your Drupal app:

   $ cd /path/to/your/drupal-site
   $ phpunit YourClass modules/your_module/YourClassFileName.php 
   PHPUnit 3.2.21 by Sebastian Bergmann.
   ..
   Time: 0 seconds
   OK (2 tests)

Here’s my finished test class:

<?php
require_once './includes/phpunit_setup.inc';
class TestDataExampleTest2 extends PHPUnit_Framework_TestCase
{  
  public function setup()
  {
    db_query("BEGIN");
  }
  public function teardown()
  {
    db_query("ROLLBACK");
  }
  public function create_test_blog_post()
  {
    $node = new stdClass();
    $node->title = "This is a blog post";
    $node->body = "This is the body of the post";
    $node->type = "Story";
    $node->promote = 1;
    node_save($node);
    return $node;
  }
  public function test_there_is_one_post()
  {
    $this->create_test_blog_post();
    $this->assertEquals(1, db_result(db_query("SELECT COUNT(*) FROM {NODE}")));
  }
  public function test_there_are_two_posts()
  {
    $this->create_test_blog_post();
    $this->create_test_blog_post();
    $this->assertEquals(2, db_result(db_query("SELECT COUNT(*) FROM {NODE}")));
  }
}
?>

In my last few posts I used Test Driven Development (TDD) to write a very simple Drupal module and showed how TDD helped to keep my custom code decoupled from the Drupal framework. This time I want to take a closer look at the biggest headache I ran into while using TDD with Drupal: handling the test data. When I started to write unit tests for my example module I ran into trouble creating test data inside each test, since I found that PHPUnit stopped executing the test each time there was a failing assertion, meaning that test data weren’t cleaned up after a test failure. I was able to avoid this problem by creating and deleting the test data before and after each test was run using the setup/teardown methods from PHPUnit. But this solution brings along different problems with it:

• In setup() I need to create all of the test data that every test will use since it is called every time, which becomes a performance problem as the number of tests increases.
• Teardown() still won’t be called if there are any PHP syntax errors in my test or production code, which happens a lot if I’m really using TDD.
• There’s no way to create different test data for different tests
• Worst of all, any existing data in my Drupal database might cause the tests to fail, and vice-versa: the test data might interfere with my development work.

We need a better approach for handling test data. Rather than reinventing the wheel, let’s take a look at the Ruby on Rails framework for some inspiration and see if we can emulate the way Rails handles test data using PHP and Drupal. How does Rails handle test data? First of all, each Rails application has multiple, different databases setup: one for development, one for production, and a third for testing at a minimum. Every time you run a unit test in Rails, the test database is manipulated as follows:

• Rails deletes the existing contents of the test database, if any.
• Rails loads your test database with an empty copy of your application’s database schema (tables, columns, indices, etc.).
• Finally Rails runs each of the unit tests targeting this empty test database, by default each test within a separate database transaction (more on this in my next post).

How can we do this with Drupal? If you take a close look at the SimpleTest module, you’ll see that it uses some tricks to create a test copy of the Drupal schema using the “database prefix” feature of Drupal. While this works fine, I decided to see if I could directly follow the Rails pattern of having a completely separate MySQL database to use for testing. Let’s use PHPUnit directly on Drupal from the command line again as I did before. Here’s a very simple PHPUnit test:

<?php
require_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);
class TestDataExampleTest extends PHPUnit_Framework_TestCase
{  
  public function create_test_blog_post()
  {
    $node = new stdClass();
    $node->title = "This is a blog post";
    $node->body = "This is the body of the post";
    $node->type = "Story";
    $node->promote = 1;
    node_save($node);
    return $node;
  }
  public function test_there_is_one_post()
  {
    $this->create_test_blog_post();
    $this->assertEquals(1, db_result(db_query("SELECT COUNT(*) FROM {NODE}")));
  }
}
?>

As I explained in December, we have to run the unit test from the Drupal root folder as follows (replace “modules/test_data_module/TestDataExampleTest.php” with the path to the test file):

$ cd /path/to/my-drupal-site
$ phpunit TestDataExampleTest modules/test_data_module/TestDataExampleTest.php

The “test_there_is_one_post” unit test will create a test blog post record in the node table, and then count the number of nodes in the database and assert that there is exactly one. Obviously this will fail if there are any existing node records in my Drupal database, or if I even just run the test more than once:

$ phpunit TestDataExampleTest modules/test_data_module/TestDataExampleTest.php
PHPUnit 3.2.21 by Sebastian Bergmann.
F
Time: 0 seconds
There was 1 failure:
1) test_there_is_one_post(TestDataExampleTest)
Failed asserting that <string:6> matches expected value <integer:1>.
/Users/pat/htdocs/drupal4/TestDataExampleTest.php:19
FAILURES!
Tests: 1, Failures: 1.

This failure is actually good: this test is intentionally dependent on the contents of the test database. Later if we can get this test to pass then we know we have properly initialized the contents of the database without resorting to the setup/teardown solution from last time.

Let’s get started by creating a real test database using the MySQL command line:

mysql> CREATE DATABASE drupal_test DEFAULT CHARACTER SET utf8
       COLLATE utf8_unicode_ci;
Query OK, 1 row affected (0.00 sec)

Now, how can we get Drupal to use this database instead of the normal one? Let’s try converting the $db_url value in settings.php into an array, like this:

$db_url["default"] = 'mysql://user:password@localhost/drupal;
$db_url["test"] = 'mysql://user:password@localhost/drupal_test';

Here I’ve renamed the original $db_url variable to $db_url[“default”], and created a new entry for the test database. Now Drupal can run against either the original database, or the test database as we wish.

The next step is to load the test database with an empty copy of the Drupal database schema. In the Rails world, there are Ruby functions that export the development database schema, and then reload it into the test database. In Drupal, the database schema is created automatically by PHP functions when you install the application for the first time. The SimpleTest module also does the same thing before running its tests. Looking at code in install.php from the Drupal installation process, and also in drupal_web_test_case.php from SimpleTest I came up with this solution:

function create_test_drupal_schema()
{
  include_once './includes/install.inc';
  drupal_install_system();
  drupal_install_modules(drupal_verify_profile('default', 'en'));
  $task = 'profile';
  default_profile_tasks($task, '');
  menu_rebuild();
  actions_synchronize();
  _drupal_flush_css_js();

  variable_set('user_mail_status_activated_notify', FALSE);
  $account = user_load(1);
  $merge_data = array('name' => 'admin', 'pass' => 'test', 'roles' => array(),
                     'status' => 1);
  user_save($account, $merge_data);  
}

Here drupal_install_system() and drupal_install_modules() will create most of the empty tables we need just the way they do when you install Drupal. The other calls I took from DrupalWebTestCase->setup() in SimpleTest to create some initial data Drupal requires to function properly, like the menu for example. The last few lines I wrote to setup the admin user properly, and to avoid sending emails to the admin during this process.

The last piece of the puzzle is to find a way to clear out the test database before each test run. To do that, I wrote this code to iterate over all of the tables in the test database and drop them:

function drop_test_tables()
{
  each_table('drop');
}
function each_table($table_callback)
{
  global $db_url;
  $url = parse_url($db_url['test']);
  $database = substr($url['path'], 1);
  $result = db_query("SELECT table_name FROM information_schema.tables
                      WHERE table_schema = '$database'");
  while ($table = db_result($result)) {
    $table_callback($table);
  }
}
function drop($table)
{
  db_query("DROP TABLE $table");
}

To put it all together we just need to call db_set_active(“test”) and call all of this code before our test runs:

db_set_active("test");
drop_test_tables();
create_test_drupal_schema();

Here db_set_active(“test”) tells Drupal we want to use the test database instead of the actual database. After switching to the test database we drop any existing tables that may exist there, and then create a new, empty Drupal schema.

To avoid cluttering my PHPUnit test file, and to be able to reuse this code in many PHPUnit tests, I moved the test database setup into a new include file called: phpunit_setup.inc. I also added some validation code to phpunit_setup.inc to perform a sanity check so you don’t accidentally drop all of the tables in your main Drupal database, and to make it easier to avoid mistakes with $db_url in settings.php. The code requires that "test" be present in the test database name. I also added code to enable database transactions in the test database, which I will discuss in my next post.

To try this out on your system, just download phpunit_setup.inc and then include it at the top of your PHPUnit test file, like this:

<?php
require_once './includes/phpunit_setup.inc';
class TestDataExampleTest extends PHPUnit_Framework_TestCase
{  
  public function create_test_blog_post()
  {
    $node = new stdClass();
    $node->title = "This is a blog post";
    $node->body = "This is the body of the post";
    $node->type = "Story";
    $node->promote = 1;
    node_save($node);
    return $node;
  }
  public function test_there_is_one_post()
  {
    $this->create_test_blog_post();
    $this->assertEquals(1, db_result(db_query("SELECT COUNT(*) FROM {NODE}")));
  }
}
?>

The require_once statement above assumes you downloaded phpunit_setup.inc into the includes folder. If you put it somewhere else, just update require_once as necessary. Now the test passes every time:

$ cd /path/to/my-drupal-site
$ phpunit TestDataExampleTest modules/test_data_module/TestDataExampleTest.php
PHPUnit 3.2.21 by Sebastian Bergmann.
.
Time: 0 seconds
OK (1 test)

One important detail I’ve glossed over here is that the test only passes because it is the only test I’m running at all in this database. If there were a second test with it’s own test data and assumptions about what data were present, then there would be failures depending on what data each test expected, and which test ran first. Next time, I’ll show how Rails solved this problem using database transactions and show how to use them with Drupal unit tests.

Many Drupal projects that I’ve been involved with during the past few years have all suffered from different variations on the same problem: the custom PHP code added by me and the development team I was in has ended up being highly coupled with the Drupal framework. At best, our code was hard to understand without knowing a lot about Drupal internals and could never run on its own outside of Drupal. At worst, our code changes were made right inside Drupal core functions and needed to be identified, reimplemented and merged into every newer version of Drupal whenever we decided to upgrade.

In hindsight, I believe that we could have avoided all of these problems if we had only used Test Driven Development (TDD). Using TDD has a lot of benefits for any software project which I won’t go into here (see Wikipedia or Bob Martin's explanation for more information) but I believe the biggest benefit of using TDD specifically on a Drupal development project is that it makes it easy to write PHP code that is not highly coupled with Drupal, is easy to distinguish from Drupal’s core modules, and can even run on its own outside of Drupal entirely.

The same is probably true while developing inside any application development framework.

To explore this topic in detail and find out whether I was right about this, in December I started experimenting with Drupal and TDD. After I wrote my first Drupal PHPUnit test, I started to write a simple example module using TDD, showing how the test first thought process can work with PHP and Drupal. Below I’ll take a look at the finished example module to see whether the module’s code is any less coupled to Drupal, and whether it’s any easier to identify and maintain.

But first let’s take a closer look at the 3 types of PHP code any Drupal web site will contain:

1. Drupal Code: This is the PHP code you download from drupal.org: files such as node.module, common.include, index.php, etc. You can assume it all works properly, was tested by the Drupal community and is used in thousands of other web sites successfully.
2. Custom Code: This is code you write to actually implement the special behavior your business or community needs in its web site. You need to write and maintain this, and you need to test that it works properly since no one else will use it.
3. Connection Code: This is code you have to write to connect your custom code to Drupal. Unfortunately, there’s no way to magically drop your business logic into a Drupal site and see it appear in a web page, ready for users. You need to know something about how Drupal actually works, and you need to connect your Custom Code with the Drupal Code properly.

A good way to understand better the difference between Drupal, Custom and Connection code is to imagine that next year you decide to upgrade your site from Drupal 6.x to Drupal 7, or that you decide to rebuild your site with Ruby on Rails or some other new technology. What you would have to do is:

• Discard the Drupal Code, and replace it with the new framework.
• Discard your Connection Code, and write the necessary new code required to connect to the new framework.
• Take your Custom Code with you, possibly retaining the PHP unchanged for Drupal 7, or else rewriting it using Ruby or some other language.

Clearly you need to know which code is which or else you’ll discard code you really need! TDD can make this easy; the reason why is that using PHPUnit forces you to structure your Custom Code so that it can run outside of Drupal from the command line. Knowing that your Custom Code can run outside of Drupal inside of a unit test suite guarantees that you will be able to take the code with you and later connect it to another framework.

Let’s see how my example module turned out; see one of my previous posts for more background on what this module does and how to set it up. Here’s the finished code:

• My Custom Code: Tdd.php and TddTests.php
• My Connection Code: tdd.module
• And the Drupal “info” file: tdd.info

What I’ve done since my last post while finishing up this simple example is to separate the Custom Code and Connection Code into two separate files. First, here’s my finished Custom Code:

<?php
function tdd_search_for_titles($query, $ascending, $from, $count) {
  $titles = array();
  if ($query != NULL) {
    $sql = "SELECT title FROM {node} WHERE title LIKE '%%%s%%'";
    if ($ascending) {
      $sql .= ' ORDER BY title';
    }
    else {
      $sql .= ' ORDER BY title DESC';
    }
    if ($from || $count) {
      $result = db_query_range($sql, $query, $from, $count);
    }
    else {
      $result = db_query($sql, $query);
    }
    while ($node = db_fetch_object($result)) {
      $titles[] = $node->title;
    }
  }
  return $titles;
}
?>

This probably could be refactored even more into 2 or 3 simpler methods, but the question here is whether this is Custom Code, or Connection Code. I know this is my Custom Code because:

• This is the code that actually implements the behavior I am trying to achieve. In this simple example the code is searching for nodes by words in their titles.
• This code is not at all coupled to the Drupal framework. This function is easy to understand for any PHP developer, even someone who knows nothing about Drupal. The only minor exception here is that it uses 3 simple utility Drupal functions: db_query(), db_query_range() and db_fetch_object(). However, these are self-explanatory; If I were a purist, I could have used the mysql_query() function instead, and have eliminated all of my dependencies on Drupal entirely. The code also assumes the presence of the node table and the title column within it, but the same code would work on any database table containing web pages and their titles with only trivial changes. The point is that the code is easy to understand and would be easy to migrate to another framework.
• This code can run outside the Drupal framework. I know this is the case since I do this when I run PHPUnit on my tests in TddTests.php:

  $ cd ~/htdocs/drupal3
  $ phpunit TddTests modules/tdd/TddTests.php 
  PHPUnit 3.2.21 by Sebastian Bergmann.
  ............
  Time: 0 seconds
  OK (12 tests)

The test code, which I won’t repeat here, is also an essential part of my Custom Code, since it is the only way I have to prove that my function is working properly. It also documents my code’s the desired behavior, and finally will allow me to validate that the code is working if I ever move it to a newer version of Drupal or to some other technology. Also, if you take a look at TddTests.php, you’ll see that the test code is also not highly coupled to Drupal. There are a few references to node_save() and drupal_bootstrap() for example, but most of the test code is pure PHPUnit and has nothing to do with Drupal.

Let’s take a look at my Connection Code, which is in the tdd.module file. I won’t repeat all of it here in this page, but if you look at tdd.module you can see the code does not have these 3 qualities:

• This code has nothing to do with the behavior I’m trying to implement. It simply provides URL parameters from the user’s request to tdd_search_for_titles(), and displays the results from tdd_search_for_titles() in a web page. It also handles the complexity around the Drupal Form API.
• This code is very coupled to the Drupal framework. For someone who doesn’t understand Drupal internals tdd.module is very hard to understand – even for developers with years of PHP experience.
• This code could never run outside of the Drupal framework. This is because it uses functions such as drupal_get_form() and theme() that would be impossible - and pointless - to implement outside of Drupal. More importantly, the functions in tdd.module are called by Drupal at certain times during its processing; none of this would make any sense outside of Drupal.

Here’s the only interesting snippet from my Connection Code:

...
$sortAscending = true;
if (isset($_GET['sort']) && $_GET['sort'] == 'desc') {
  $sortAscending = false;
}
$titles = tdd_search_for_titles($keys, $sortAscending, 0, 10);
$rows = array();
foreach ($titles as $title) {
  $rows[] = array($title);
}
...

This is the actual location in the code where we “connect” from Drupal to my Custom Code, and vice-versa. This few lines are actually passing (connecting) the request parameters onto my Custom Code, and later saving the results in $titles which is parsed and returned to Drupal in the required format.

In my next post I’ll try using TDD again to write a more complex and interesting Drupal module: one that will display monthly archive links - e.g. January 2009 (23) - for a Drupal blog, similar to what you would see in a standard WordPress or B2Evolution blog site.