Hofmockel.org

Our configuration management discipline paid off last week, when we rolled out lots of bugfixes onto our production server. I used my deployment script to test the release-1 branch against our regression tests and copy the source code to the production server after it passed. A quick database update later, and everything worked without a hitch.

Why did it work? Here's what's behind our build system:

• We have a branch for release-1 bugfixes and a trunk branch for new features. It's easier than cherrypicking features to push to production.
• Module .install files handle all behavior-related changes (variables, new modules, etc.). Drupal can automatically apply those changes during the database upgrade process. Even the process of enabling a module is done through an update function in our main module's .install file. Note that the module install files are called in alphabetic order, so you may need to think about the sequence of functions.
• The Simpletest framework for unit and functional testing allows us to write regression tests that minimize risks of updates.
• Updates and tests are always run after a stripped-down copy of the production database is loaded, and are also occasionally tested against a full copy of the production database. This makes sure that the changes will cleanly apply to the production server. Domain Access complicates things a little because the domain name is encoded in the database, so my deployment script also takes care of substituting the correct domain name.
• To automate testing and upgrades, we use the Drupal Shell (drush) module in our local and QA environments.
• I wrote a deployment script webpage that provides a form for managing database changes and source code deployments to our QA server and to our production server.

I think it's pretty darn good. What would make it even awesomer?

• An updated deployment plan. I'm going to work on this document later.
• Testing before commits, instead of after. maybe I can set up a pre-hook on my own system, or use Emacs to do this somehow.
• Buttons to get full copy of database for local, QA. Full copies of the database take much more time, but are useful for getting a realistic feel for the functionality.
• An article on how other people can get up and running with this kind of thing. DeveloperWorks, maybe?

Technorati Tags: deployment

So now that we're doing all our configuration changes in source code, it makes sense to automate database updates as much as I can. Here's something I've added to drush_tools so that I can run all the schema changes from the command-line:

function drush_tools_update($command = '') { ob_start(); require_once 'includes/install.inc'; require_once 'update.php'; $ret = ob_get_contents(); drupal_load_updates(); ob_end_clean(); $list = module_list(); $update_list = array(); foreach ($list as $module) { $updates = drupal_get_schema_versions($module); if ($updates !== FALSE) { $latest = 0; $base = drupal_get_installed_schema_version($module); foreach ($updates as $update) { if ($update > $base) { if ($update > $latest) { $latest = $update; } $update_list[$module][] = $update; } } if ($latest) { sort($update_list[$module]); printf("%-30s %5d -> %5d (%s)\n", $module, $base, $latest, join(', ', $update_list[$module])); } else { printf("%-30s %5d\n", $module, $base); } } } if (count($update_list) == 0) return; if ($command != 'force' && !drush_confirm(t('Do you really want to continue?'))) { drush_die('Aborting.'); } ob_start(); foreach ($update_list as $module => $versions) { foreach ($versions as $v) { print "Running " . $module . "_update_" . $v . "\n"; update_data($module, $v); } } $updates = ob_get_contents(); cache_clear_all('*', 'cache', TRUE); cache_clear_all('*', 'cache_page', TRUE); cache_clear_all('*', 'cache_menu', TRUE); cache_clear_all('*', 'cache_filter', TRUE); drupal_clear_css_cache(); ob_end_clean(); $output = ''; if (!empty($_SESSION['update_results'])) { $output .= "The following queries were executed:\n"; foreach ($_SESSION['update_results'] as $module => $updates) { $output .= "\n" . $module . "\n--------------------------\n"; foreach ($updates as $number => $queries) { $output .= 'Update #'. $number . ":\n"; foreach ($queries as $query) { if ($query['success']) { $output .= "SUCCESS: " . $query['query'] . "\n"; } else { $output .= "FAILURE: " . $query['query'] . "\n"; } } if (!count($queries)) { $output .= "No queries\n"; } } } $output .= "\n"; print $output; unset($_SESSION['update_results']); } }

Technorati Tags: drupal, drush

To keep track of the bugfixes we'll need to make for our next release, I've created a Subversion branch called branches/release-1. Development of new features will continue on trunk, but we'll merge in the bugfixes from release-1 every so often.

There are three environments we deploy to:

Local

Developers should be able to easily test both versions on their local machines.

QA server

We should be able to deploy both versions to a publicly-accessible QA server for acceptance testing.

Production server

We should be able to deploy release-1 (and then later, release-2 and so on) to the production server, preferably after a lot of testing

Editorial changes happen on the production server, where our users update content. We would like to be able to take a snapshot of that database and use that to test our development code on the QA server or in our local development environments. Because we use Domain Access to serve multiple subdomains with shared content, it's not just a matter of using mysqldump to back up the database and copy it over. We also need to replace URLs inside the database, and we need to override domain_root using the $conf array in settings.php.

I'm the only one running Linux, so the other developers don't really benefit from the Makefiles I've defined or the tools I use. For the simpler build system we had before (all development on trunk), I wrote a deployment script that allowed users to:

• Download a stripped copy of the production database with the URLs changed for their local testing environment
• Deploy a stripped copy of the production database to the QA server
• Deploy a specified revision of the source code to the QA server
• Deploy a specified revision of the source code to the production server

The new deployment script needed to allow users to do the same, but for both branches of the code. Both branches of the code would be simultaneously available on the QA server, so the script would need to deploy the code to different directories.

After some fiddling around with the page design (because I care about making interfaces make sense!), I came up with something that looks like this:

  Development Release-1 Local Database QA Database QA Deployment 988987985984983982981980979978977976975974973972971970969968 986963985984983982981980979978977976975974973972971970969968967 Production Deployment None 986963985984983982981980979978977976975974973972971970969968967 Changelog ------- r988 | somegeek | 2008-09-15 13:00:00 -0500 (Mon, 15 Sep 2008) | 2 lines Use dev.qa.example.com ------------ [more changelog entries go here] ------------------------------------------------------------------------ r986 | somegeek | 2008-09-15 11:03:38 -0500 (Mon, 15 Sep 2008) | 3 lines Starting a branch for release-1 ------------------------------------------------------------------------ [more changelog entries go here]

The deployment script allows the user to get a copy of the database, deploy a copy of the database, or deploy specific revisions of branches.

Because I was having a hard time figuring out how to do ssh key-based operations from Apache (which runs as a no-login user), I use two shell scripts to do the dirty work. One shell script connects to the production server, creates a partial backup, copies the information over, and does any necessary replacements. Another shell script takes a domain name and optionally a revision, and deploys the revision from the appropriate branch.

Here's my totally small-scale PHP way to show the revisions log:

$dev_output = shell_exec("svn log $dev_url $details --limit 20"); $dev_revisions = preg_match_all('/r([0-9]+)/', $dev_output, $dev_matches);

where $dev_url is the URL of the trunk in Subversion, and $details contains the username and password specified as options for the Subversion command-line.

I'm going to see if I can get my regression tests running on the server that I've got my deployment script on. Wouldn't that be awesome?

Technorati Tags: configuration management, deployment, drupal, subversion, svn

Hooray! We have running code, and we're about to make another release after our code exits quality assurance. This means, of course, that we'll need some way to differentiate the inevitable bugfixes that the next production release will require, and development of new features.

What's the best way to do this? Making a release branch seems like a good idea. Here's how I did it:

svn copy http://subversion.example.com/myproject/trunk http://subversion.example.com/myproject/branches/release-1

Bugfixes for release-1 will be committed to the release-1 branch, while new development continues on trunk. Bugfixes will be periodically merged into trunk to make it easier to roll the next release, which will be the release-2 branch.

Next, I need to configure my local system so that it's easy to switch back and forth. I could work with a single source tree for local.example.com, but that means switching back and forth. The best thing to do would be to have two separate source code directories: one for trunk, and one for production releases.

svn co http://subversion.example.com/myproject/branches/release-1 /var/www/example.com-prod

For each site, I'll need

• a source code directory
• a database
• entries in /etc/hosts
• entries in my Apache configuration
• a local site directory (ex: sites/dev.local.example.com) with settings.php
• a QA site directory (ex: sites/dev.qa.example.com) with settings.php

I'll also need to update my Makefile to make it easier to work. For example, the Makefile should connect me to the right database depending on which branch I'm on. How do I determine this? One way is to have an unversioned file that overrides some of the Makefile variables, and to include that file in my Makefile. I can do this by adding the following to my Makefile:

-include *.mk

and then creating a dev-local.mk file that changes the values of my variables.

I'll need copies of the production database translated for the different domains, which means I need to update my deploy script and format it a little to make it easier to deal with all these options. Hmm…

This will be fun.

UPDATE: Fixed HTML tags. Thanks for pointing it out!

To make configuration management easier, we decided to make sure that all behavior-related changes are in the source code repository. So when I needed to add the reCAPTCHA module to the project, I needed to figure out how to programmatically install and enable the module with update code in another module's .install file.

Here is some sample code to do so:

/** * Install and enable the captcha module. */ function yourmodule_update_1() { $ret = array(); include_once('includes/install.inc'); module_rebuild_cache(); drupal_install_modules(array('recaptcha')); variable_set('recaptcha_public_key', 'PUBLIC KEY GOES HERE'); variable_set('recaptcha_private_key', 'SECRET KEY GOES HERE'); $ret[] = array( 'success' => true, 'query' => 'Installed recaptcha module and enabled it', ); return $ret; }

I've written about using drush to evaluate PHP statements in the Drupal context using the command line before, and it turns out that Drush is also quite useful for running Simpletest scripts. Drush comes with a module that allows you to display all the available tests with "drush test list", run all the tests with "drush test run", or run specified tests with "drush test run test1,test2".

'Course, I wanted to run groups of tests and tests matching regular expressions, so I defined two new commands:

drush test run re regular-expression

Run all tests matching a regular expression that uses ereg(..) to match.

Ex: drush test run re Example.*

drush test run group group1,group2…

Run all tests matching the given groups

Ex: drush test run group Example

Here's the patch to make it happen:

Index: drush_simpletest.module =================================================================== --- drush_simpletest.module (revision 884) +++ drush_simpletest.module (working copy) @@ -12,9 +12,13 @@ function drush_simpletest_help($section) { switch ($section) { case 'drush:test run': - return t("Usage drush [options] test run.\n\nRun the specified specified unit tests. If is omitted, all tests are run. should be a list of classes separated by a comma. For example: PageCreationTest,PageViewTest."); + return t("Usage drush [options] test run .\n\nRun the specified unit tests. If is omitted, all tests are run. should be a list of classes separated by a comma. For example: PageCreationTest,PageViewTest."); case 'drush:test list': return t("Usage drush [options] test list.\n\nList the available tests. Use drush test run command to run them. "); + case 'drush:test group': + return t("Usage drush [options] test group .\n\nRun all unit tests in the specified groups. For example: drush test group Group1,Group2"); + case 'drush:test re': + return t("Usage drush [options] test re .\n\nRun all unit tests matching this regular expression. For example: drush test re Page.*"); } } @@ -30,10 +34,18 @@ 'callback' => 'drush_test_list', 'description' => 'List the available Simpletest test classes.', ); + $items['test re'] = array( + 'callback' => 'drush_test_re', + 'description' => 'Run one or more Simpletest tests based on regular expressions.', + ); + $items['test group'] = array( + 'callback' => 'drush_test_group', + 'description' => 'Run one or more Simpletest test groups.', + ); return $items; } -function drush_test_list() { +function drush_test_get_list() { simpletest_load(); // TODO: Refactor simpletest.module so we don't copy code from DrupalUnitTests $files = array(); @@ -60,6 +72,11 @@ $rows[] = array($class, $info['name'], truncate_utf8($info['desc'], 30, TRUE, TRUE)); } } + return $rows; +} + +function drush_test_list() { + $rows = drush_test_get_list(); return drush_print_table($rows, 0, TRUE); } @@ -75,3 +92,31 @@ } return $result; } + +function drush_test_re($expression) { + if (!$expression) { + die('You must specify a regular expression.'); + } + $rows = drush_test_get_list(); + $tests = array(); + foreach ($rows as $row) { + if (ereg($expression, $row[0])) { + $tests[] = $row[0]; + } + } + simpletest_run_tests($tests, 'text'); + return $result; +} + +function drush_test_group($groups) { + $rows = drush_test_get_list(); + $tests = array(); + $groups = explode(',', $groups); + foreach ($rows as $row) { + if (in_array($row[1], $groups)) { + $tests[] = $row[0]; + } + } + simpletest_run_tests($tests, 'text'); + return $result; +}

That makes running tests so much easier and more fun!

Technorati Tags: drupal, drush, patch, simpletest, testing

Building on the configuration management strategy I described last time, I wrote some scripts to make it easier for other developers to migrate the production database to the QA server or to get a copy of the production database for their local system. I needed to consider the following factors:

• Domain name changes: Because we use Domain Access to serve multiple subdomains using a single Drupal installation and shared sign-on, we needed to make sure that all instances of the domain root are replaced and the correct domain_root is set. For example, the site URLs might be:
  Production QA Local example.com qa.example.com local.example.com foo.example.com foo.qa.example.com foo.local.example.com bar.example.com bar.qa.example.com bar.local.example.com
• Privacy concerns: The QA database and local copies of the database should not contain sensitive user information. All users aside from the administrator should be removed from the system, and all content on site should be limited to editorial content.
• Efficiency: We don't need data like access logs or watchdog logs in QA or for local testing. This saves quite a lot of time and space during the data migration process.

Here's how I did it:

1. I identified tables I needed to copy over and tables that I could just leave empty. I did this by going through the output of "SHOW TABLES" in a MySQL console.
2. In my Makefile, I declared a DB_DATA variable that contained a list of tables I wanted to copy.
3. I wrote a backup_partial target in my Makefile that issued a series of commands: mysqldump -u ${DB_USER} --password=${DB_PASSWORD} ${DB} --no-data > partial.dump # export the schema mysqldump -u ${DB_USER} --password=${DB_PASSWORD} ${DB} --opt --complete-insert ${DB_DATA} >> partial.dump # Dump some of the data mysqldump -u ${DB_USER} --password=${DB_PASSWORD} ${DB} --opt --complete-insert --where='uid< =1' users users_roles >> partial.dump # Dump the admin and anonymous users echo "UPDATE node SET uid=1;" >> partial.dump # Set all the node authors to admin echo "REPLACE INTO search_index_queue (nid, module, timestamp) select nid, type, unix_timestamp(now()) FROM node;" >> partial.dump # Prepare for reindexing
4. I wrote a shell script on an internal server that accepted an argument (the domain to translate to) and performed the following steps:
   1. ssh into the production server and run make with my backup_partial target, compressing the resulting partial.dump
   2. scp the partial.dump.gz from the production server onto the internal server
   3. unpack partial.dump.gz
   4. figure out what $DOMAIN is supposed to be based on the arguments
   5. run perl -pi -e "s/example.com/$DOMAIN/" partial.dump
   6. load partial.dump into my database
   7. run cron.php if it can
5. I added two buttons to my web-based deploy script: one button to migrate the production database to the QA server, one button to make a copy of the production database for the domain "local.example.com". Both buttons call
6. I created multisite settings.php in my Drupal directory (ex: sites/local.example.com and sites/qa.example.com). The production settings go in default/settings.php, and the multisite settings.php override it like this: $conf = array( 'domain_root' => 'local.example.com', ); $cookie_domain = '.' . $conf['domain_root'];

   $conf allows you to override Drupal variables returned by variable_get.

So now, I can click on a button to migrate a sanitized copy of the production database to the QA server or to my local system. Sweet!

Technorati Tags: makefile, mysql

Drupal's coding conventions make it easier to hook into system behavior, but they also result in a lot of repetitive typing. For example, you can run code when upgrading a module by putting the code in a function named modulename_update_N() in your module's install file. I found myself scrolling up and copy-pasting stuff too many times, so I decided to automate it instead.

I've been using yasnippet for my Emacs templates. All I needed to do to automate that little update bit was to write some code that figured out what the next update number should be. Here's the snippet file I've just added (~/elisp/snippets/php-mode/drupal-mode/_update):

function `(sacha/drupal-module-name)`_update_`(sacha/drupal-module-update-number)`() { $ret = array(); $0 $ret[] = array( 'success' => true, 'query' => '$1', ); return $ret; }

The relevant functions from my ~/.emacs:

(defun sacha/drupal-module-update-number () "Return the number of the next module update function. This is one more than the highest number used so far. This function should be called in a module's .install file." (save-excursion (save-restriction (widen) (goto-char (point-min)) (let ((module-name (sacha/drupal-module-name)) (max 0)) (while (re-search-forward (concat "function[ \t\n]+" module-name "_update_\\([0-9]+\\)") nil t) (setq max (max (string-to-number (match-string 1)) max))) (number-to-string (1+ max)))))) (defun sacha/drupal-module-name () "Return the Drupal module name for .module and .install files." (file-name-sans-extension (file-name-nondirectory (buffer-file-name))))

I can't think of how I'd do that in Eclipse. =) Don't get me wrong–I still like Eclipse–but I heart being able to hack my editor on the fly.

Technorati Tags: template, yasnippet

I often find myself needing to variable_set something temporarily, just to try things out. The drush module provides a command-line interface that solves this problem with a little more hacking. To minimize the effect on my source tree, I've unpacked it into the sites directory for my local installation and enabled it in my test database. After I enabled the main drush module and the related modules, I tweaked drush_tools to include an insecure-but-useful eval command. Here it is:

--- sites/local.example.com/modules/drush/drush_tools.module.orig 2008-08-05 17:18:48.000000000 -0400 +++ sites/local.example.com/modules/drush/drush_tools.module 2008-08-05 17:18:55.000000000 -0400 @@ -46,6 +46,10 @@ 'callback' => 'drush_tools_sync', 'description' => 'Rsync the Drupal tree to/from another server using ssh' ); + $items['eval'] = array( + 'callback' => 'drush_tools_eval', + 'description' => 'Evaluate a command', + ); return $items; }   @@ -156,3 +160,6 @@ } }   +function drush_tools_eval($command) { + eval($command); +}

I also added an alias to my ~/.bashrc along the lines of:

alias drush='php ~/drupal/sites/local.example.com/modules/drush/drush.php -r ~/drupal -l http://local.example.com'

where ~/drupal is my multisite Drupal directory root.

After I loaded the alias with "source ~/.bashrc", I can now execute PHP statements in my Drupal context with commands like:

drush eval "variable_set('hello', 'world');"

Good stuff!

Technorati Tags: drush