ActiveRecord Enumerable

Posted on 2007-02-03 by michael

Ever wanted to iterate over a huge number of objects and found
that find(:all) doesn’t cut it? Then this plugin
might be for you. Install it with


$ script/plugin install svn://rubyforge.org/var/svn/ar-enumerable/ar_enumerable/trunk

First, a note of caution. This plugin has not seen much practical
use yet, in its current state it is only slightly above a
proof-of-concept. There are several tests included that cover most
of the functionality. If you intend to use this plugin, please
make sure to independently check that it does what you want.

Enumerating ActiveRecord objects

Let’s start with a boring example, to get the general idea


Person.where(['lastname = ?', 'Smith']).map(&:firstname)

Slightly more interesting


Person.where(['lastname = ?', 'Smith']).inject(0) do |income, person|
  income += person.income
end


siblings = Person.where(
    ['lastname = ?', 'Smith'], 
    :include => :siblings).inject([0, 0]) do |memo, person|
  memo[0] += person.siblings.count
  memo[1] += 1
  memo
end
siblings[0].to_f / siblings[1]

The point here is that your database may contain millions of Smiths,
for what it’s worth, the entire population of the earth could have
turned into Smiths, still, only a limited number of them would be
loaded into memory at the same time.

Internally, enumeration works in two completely different ways.
By default, a chunk of objects is retrieved from the database using
OFFSET and LIMIT clauses. This way is supported by all databases,
however it can be slow.
Therefore, the preferred alternative is to use a standard SQL cursor
for iterating over the result set of a query. It would simply look like
this


Person.where(['lastname = ?', 'Smith'], :use_cursor => true).inject(0) do |income, person|
  income += person.income
end

A further parameter is how many records (or rows, in the case of cursors)
are fetched from the database in one access. The default is 1000. You can
set it explicitly like this


Person.where(['lastname = ?', 'Smith'], :fetch_count => 100).inject(0) do |income, person|
  income += person.income
end

Performance Considerations

Some informal tests suggest that for large result sets without
included associations cursor-based enumeration can be faster than
OFFSET/LIMIT-based enumeration by a factor of 3 to 5.
When associations are included, things become more blurry. In this
case, cursor-based enumeration imposes an ordering on the result
set that can slow down things considerably.
The fetch count has considerable influence on performance, basically,
you can buy speed by spending memory.

Auto-completion for tag lists

Posted on 2007-01-31 by michael

See the end of this post for an important update.

Standard Rails auto-completion is geared toward completing a single value entered in a text field. For most cases this is exactly what’s needed, but there is one prominent case that is different: auto-completion of tags. If something can have a tag at all, it’s part and parcel of the deal that it can have multiple tags.

As far as I can tell, some configuration tweak is not enough to make standard auto-completion do our bidding. But not all is lost. It’s just a small matter of programming to get what we want.

I’m assuming that you’re using Acts As Taggable or Acts As Taggable On Steroids for the model-level tagging, or otherwise that you know what you’re doing.

First, we need client-side support for picking out only the tag we’re currently editing.


Ajax.TagAutocompleter = Class.create();
Object.extend(Object.extend(Ajax.TagAutocompleter.prototype, Ajax.Autocompleter.prototype), {
  getToken: function() {
    var range = this.rangeForCaret();
    var value = this.element.value.substring(range.start, range.end);
    return value;
  },
  updateElement: function(selectedElement) {
    var oldValue = this.element.value;
    var range = this.rangeForCaret();
    var prefix = oldValue.substring(0, range.start);
    var suffix = oldValue.substring(range.end);
    var value = Element.collectTextNodesIgnoreClass(selectedElement, 'informal');
    this.element.value = prefix + value + suffix;
    this.element.focus();
  },
  rangeForCaret: function() {
    var value = this.element.value;
    var start = 0, end = value.length;
    var pos;
    if (typeof this.element.selectionStart != 'undefined') {
      pos = this.element.selectionStart;
    } else if (document.selection) {
      var sel = document.selection.createRange();
      var selLength = sel.text.length;
      sel.moveStart('character', -(value.length));
      pos = sel.text.length - selLength;
    }
    if (pos != undefined) {
      end = pos;
      while (value.charAt(end) != ',' && end < value.length) {
        end++;
      }
      start = end - 1;
      while (value.charAt(start) != ',' && start > 0) {
        start--;
      }
      while (start < end && (value.charAt(start) == ' ' || value.charAt(start) == ',')) {
        start++;
      }
      while (start < end && value.charAt(end) == ' ') {
        end--;
      }
    }
    return {start: start, end: end};
  }
});

Yes, the code of rangeForCaret is a bit convoluted, let’s say traditional JavaScript-style. The general approach for getting the position of the text cursor (“caret”) is explained here. The purpose of all the stuff is to support auto-completion in the middle of the text field, not just at its end.

On the server-side, in the relevant controller, there’s still some work to do as the standard auto_complete_for can’t handle tags. Put the following method in your controller base class, app/controllers/application.rb.


def self.tag_auto_complete_for(object)
  define_method("auto_complete_for_#{object}_tag_list") do
    @items = Tag.find(:all,
      :conditions => [ 'LOWER(name) LIKE ?', '%' + params[object][:tag_list].downcase + '%' ], 
      :\order => 'name ASC',
      :limit => 10)
    render :inline => "<%= auto_complete_result @items, :name %>"
  end
end

Then, in an individual controller, create the tag auto-completion method like this


class PeopleController < ApplicationController
  tag_auto_complete_for :person
end

Almost done!

What’s still missing is a way to activate auto-completion for the tag input field. It would surely be possible to copy & change text_field_with_auto_complete or to monkey patch it appropriately. I prefer another way, in fact, I’m not using text_field_with_auto_complete at all, not even where it would work. Instead, I prefer more unobtrusive way.

The way starts with the class autocomplete to text fields that are amenable to auto-completion — including, of course, the tag list field.


  <%= f.text_field :tag_list, :class => 'autocomplete' %>

As this in itself is completely inert, we need to nudge it a bit in application.js. Here’s a function that looks for all text fields with class autocomplete and actually makes them do what they say.


function installAutocompletion() {
  $$('input.autocomplete[type=text]').each(function(element) {
    var fieldId = element.id;
    var completions = document.createElement('div');
    completions.id = fieldId + '_auto_complete';
    completions.className = 'auto_complete';
    completions.style.display = 'none';
    element.parentNode.insertBefore(completions, element.nextSibling);

    var url = 'auto_complete_for_' + fieldId;
    if (/_tag_list$/.test(fieldId)) {
      new Ajax.TagAutocompleter(element, completions, url);
    } else {
      new Ajax.Autocompleter(element, completions, url);
    }
  });
}

Be sure to call this method some time after the page is loaded. If you don’t have a function from where to call it, add this to application.js


  Event.observe(window, 'load', installAutocompletion);

Update, 2007-02-05

I may not have seen the wood for the trees when looking at the original code of the Rails/Script.aculo.us autocompleter. Of course it is fully capable of auto-completing more than one token in the same text field — with a very little configuration tweak, no less.

What it can’t do is auto-completion in the middle of the line; it only works at the end. That’s an acceptable restriction, to my mind. So, forget about Ajax.TagAutocompleter from above and instead install your autocompleters like this


function installAutocompletion() {
  $$('input.autocomplete[type=text]').each(function(element) {
    var fieldId = element.id;
    var completions = document.createElement('div');
    completions.id = fieldId + '_auto_complete';
    completions.className = 'auto_complete';
    completions.style.display = 'none';
    element.parentNode.insertBefore(completions, element.nextSibling);

    var url = 'auto_complete_for_' + fieldId;
    var options = {};
    if (/_tag_list$/.test(fieldId)) {
      options.tokens = ',';
    }
    new Ajax.Autocompleter(element, completions, url, options);
  });
}

Eclipse: Printing Editor Templates using XSLT, CSS, and Firefox

Posted on 2007-01-01 by michael

I’m using Eclipse with RadRails to edit my Ruby and Rails code (and, yes, I’ve used it for Java, way back when).

Inside Eclipse the various text editors offer so-called templates. These are snippets of common code with placeholders for variable bits. Type in the abbreviation for a template, then type Ctrl-Space and the abbreviation is expanded to the template code. For editing Ruby/Rails templates are available from the RadRails Templates site.

I’ve only recently started to use templates and I’m still early on the learning curve, that is, I haven’t memorized many of the abbreviations. Unfortunately, Eclipse itself only has a list of templates with their expansions in its modal Preferences window. That’s not very helpful when coding. What Eclipse does have is a function to export templates to XML and that’s what I did.

The exported file looks like this


<?xml version="1.0" encoding="UTF-8"?>
<templates>
  <template autoinsert="true" context="ruby" deleted="false" description="tm - all? { |e| .. }" enabled="true" name="all">all? { |${e}| ${cursor} }
  </template>
  <template autoinsert="true" context="ruby" deleted="false" description="tm - alias_method .." enabled="true" name="am">alias_method :${new_name}, :${old_name}
  </template>
  ...
</templates>

The easiest approach I could think of to make this into something printable is this. First, transform it into HTML using an XSL stylesheet, then make the HTML pretty by styling it with an CSS stylesheet. Luckily, most of this can be done inside a web browser such as Firefox that understands XSLT.

Here’s the XSLT


<?xml version="1.0" encoding="iso-8859-1"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:template match="/">
  <html>
    <head>
      <xsl:apply-templates mode="head" />
      <link rel="stylesheet" type="text/css" href="templates.css" />
    </head>
    <body>
      <xsl:apply-templates mode="body" />
    </body>
  </html>
</xsl:template>

<xsl:template match="title" mode="head">
  <title>
    <xsl:value-of select="." />
  </title>
</xsl:template>

<xsl:template match="title" mode="body">
  <h1>
    <xsl:value-of select="." />
  </h1>
</xsl:template>

<xsl:template match="templates" mode="body">
  <dl>
    <xsl:apply-templates mode="body" />
  </dl>
</xsl:template>


<xsl:template match="template" mode="body">
  <dt>
    <xsl:value-of select="@name" />
  </dt>
  <dd>
    <xsl:value-of select="." />
  </dd>
</xsl:template>

</xsl:stylesheet>

And the CSS stylesheet


html {
  font-family: sans-serif;
}

dt {
  border-top: 1px solid #aaa;
  padding-top: 0.2em;
  font-weight: bold;
}

dd {
  margin-top: 0.3em;
  margin-bottom: 1em;
  white-space: pre;
  font-family: monospace;
}

Now, these parts need to be connected. For this, it is necessary to slightly edit the original XML template file. For good measure, I throw in a title.


<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="templates.xsl" ?>
<templates>
  <title>RadRails Ruby Templates</title>
  <template ...>
    ...
  </template>

You may find that the indentation for the template code looks wrong. The reason is that the code contains tabs for indentation. Ruby convention is to indent by 2 spaces for each level, by contrast, Firefox apparently expands tabs to 8 spaces. A small glitch that can be rectified easily


$ sed -i 's/\t/  /g' template.xml

Rails: Almost Automatic Client-Side Validation

Posted on 2006-12-15 by michael

Ruby on Rails already does a good job of specifying almost declaratively what conditions objects must meet to be considered valid–that is, how they are validated.

Thus, in order to make sure that a Person’s last_name attribute does not exceed 100 characters, you would write something like this


class Person < ActiveRecord::Base
  validates_length_of :last_name, :maximum => 100
end

Somewhere in a view you’ll probably refer to the last_name attribute like this


<% form_for :person, @person, :url => { :action => "update" } do |f| -%>
  Last name : <%= f.text_field :last_name %>
  <%= submit_tag %>
<% end -%>

There you get a nice form where users can enter last names as long as they may wish–only to be shown an error message complaining about the length in cases where they have unwittingly overshot the limit. How should they have known?

Well, as a nice and caring person you could add an annotation to each input field saying how long its contents are allowed to be. That would be slightly better, but still leaves more work for the users than necessary.

How about this: Users are immediately notified in an unobtrusive way that some of the content they have (or have not yet) filled into a form does not constitute valid data. Furthermore, users are kept from submitting a form that contains obviously invalid data.

Should be easy, shouldn’t it? Yes, it should and actually it is just two plugins away.

Raising awareness of validations

Out of the box, Rails handles validations in such a way that it adds callbacks for checking them to model classes, but otherwise immediately loses awareness of them. When later on in the lifecycle of a model instance validations are checked, Rails does so on autopilot. A model class just does it’s thing, it won’t tell you what it’s doing, but you can be sure it’ll complain when something is wrong. Let’s add some reflection to make model classes a bit more loquacious. The details are irrelevant, just install the Validation Reflection plugin into your Rails application


$ script/plugin install svn://rubyforge.org//var/svn/valirefl/validation_reflection/trunk

Tell the browser

So far, the newfound awareness of validations idles away on the server-side. Somehow it has to be transported to the user’s browser. That is one of the things the Client-Side Validation plugin does. Install with


$ script/plugin install svn://rubyforge.org//var/svn/clientsidevali/client_side_validation/trunk

The first thing this plugin does is that it enhances tags generated by Rails’s helper methods in such a way that they contain encoded information about the validation constraints that apply to their values.

Engage the client

Still, even though validation information has been inserted into the HTML sent to the browser, where it lays dormant. On the client-side, JavaScript is where the action is. Therefore the task of checking validations there is handled by a validator written in JavaScript included with the Client-Side Validation plugin.

The generic script for the validator is included in your views or layout as part of the default scripts, therefore it is probably just there already.


<%= javascript_include_tag :defaults %>

Here we come to a point where you, the programmer, have to make some decisions and do some work. First, you need to decide (or have your code decide) what locale your users are in. Remember, dates don’t look the same all over the world.


 <%= javascript_include_tag 'validators-en' %>

Currently English (en) and German (de) are supported.

Then, you have to tell the validator that there’s work for it to do. For this, all forms that ought to be validated have to be marked with the class validated. Thus, the above form becomes


<% form_for :person, @person, :url => { :action => "update" }, :html => { :class => 'validated' } do |f| -%>
  Last name : <%= f.text_field :last_name %>
  <%= submit_tag %>
<% end -%>

Also, you need to nudge the validator to start looking at the forms by adding a line to public/javascripts/application.js


Form.Validator.installForAllValidatedForms();

You ain’t see nothing yet

If you’ve followed all these steps, the validator is probably doing its work, but hardly see anything of it. You may notice that the submit button is enabled when the form is valid and disabled when it is invalid, but that is all there is.

A further, but still invisible, thing that the validator does is that it adds (removes) the class “invalid” to input elements when they are invalid. Making this change in attribute visible is a small matter of CSS


 .invalid {
   border: 1px solid #f00;
 }

The result is not pretty, but you can push your stylesheeting abilities to the limit to make them look however you like. Also, the validator offers hooks for customizing most aspects. But that is left for a future installment.

Art Zoyd: Le champ des larmes

Posted on 2006-12-04 by michael

But is it music? Acoustic art may be more descriptive. To get an idea, imagine what it might sound like in the engine bay of battered space freighter. Yes, I’m assuming there’s air in the bay and you better hope that you notice the quietly approaching Alien early enough to run for it.

Listening to this Art Zoyd album won’t kill you, but you better be sure to check your ears afterwards.

RSnapshotting on your own schedule

Posted on 2006-11-26 by michael

I’m using rsnapshot for making regular backups of my Linux box. Neither the computer, nor the external backup disk is running all the time, therefore it would not work to simply have cron run rsnapshot at fixed times.

I want a little more flexibility than that. When the computer is off, there’s no point in making a snapshot. Fine, there’s no work left to handle that case. But in cases where the computer is running, but the backup disk is not, I want it to be recorded that something should have been done–if only the disk had been on. Then, next time the disk is running again, the recorded (queued) tasks are executed in order.

As there are multiple processes involved, for enqueueing tasks and for dequeueing them, and as the later can take some time, there’s a bit of mutual exclusing involved. Luckily, Debian GNU/Linux provides very useful helper programs in the lockfile-progs package.

/usr/local/lib/snapshot/dequeue


#! /bin/sh

QUEUEFILE=/var/lib/snapshot/queue
LOCKFILE=/var/lib/snapshot/LCK..queue

lockfile-create $LOCKFILE
if ! test -f $QUEUEFILE || ! grep -Fqs -- "$*" $QUEUEFILE ; then
  echo $* >> $QUEUEFILE
fi
lockfile-remove $LOCKFILE