Meta Thought

PDF Forms Editing in OS X

2011-03-21T13:15:00.000-07:00

I needed a simple PDF forms editor so that I can fill in and send forms via email and did not want to use the stone age process of printing the PDF out, filling it in, scanning it back in and then sending off the digitised copy.

FormulatePro fits the bill perfectly

Bundling jruby app with rawr:bundle:exe dies with 'private method `split' called for nil:NilClass' in

2011-02-03T19:23:00.000-08:00

I have a little jruby SWT app that I'm trying to bundle up in its own little cocoon with rawr but I keep getting the following:


C:\nms-helper4.git>rake rawr:bundle:exe
(in C:/nms-helper4.git)
mkdir -p package/classes/java
javac -target 1.6 -cp lib/java/jruby-complete.jar;lib/java/swt_linux.jar;lib/jav a/swt_linux64.jar;lib/java/swt_osx.jar;lib/java/swt_osx64.jar;lib/java/swt_win32.jar;src -sourcepath src -d package/classes/java src/org/rubyforge/rawr/Main.java
mkdir -p package/classes/ruby
Compile src/main.rb into package/classes/ruby/main.class
   compile_dirs has src_dirs = ["src"]
   glob_ruby_files has directory 'src' glob ["src/main.rb", "src/swt_wrapper.rb"]
files for src: 2
   ruby_globs.each has glob_data = #
    Go compile ["src/main.rb", "src/swt_wrapper.rb"]
Compiling src/main.rb to class main
Compiling src/swt_wrapper.rb to class swt_wrapper
mkdir -p package/classes/META-INF
mkdir -p package/jar
=== Creating jar file: package/jar/nms-helper.jar
cp lib/java/jruby-complete.jar package/jar/lib/java/jruby-complete.jar
cp lib/java/swt_linux.jar package/jar/lib/java/swt_linux.jar
cp lib/java/swt_linux64.jar package/jar/lib/java/swt_linux64.jar
cp lib/java/swt_osx.jar package/jar/lib/java/swt_osx.jar
cp lib/java/swt_osx64.jar package/jar/lib/java/swt_osx64.jar
cp lib/java/swt_win32.jar package/jar/lib/java/swt_win32.jar
mkdir -p package/windows
Creating Windows application in package/jar/nms-helper.exe
rake aborted!
private method `split' called for nil:NilClass

(See full trace by running task with --trace)

Taking the advice presented at the end there I run it again with:


C:\nms-helper4.git>rake rawr:bundle:exe --trace
(in C:/nms-helper4.git)
** Invoke rawr:bundle:exe (first_time)
** Invoke rawr:jar (first_time)
** Invoke package/jar/nms-helper.jar (first_time, not_needed)
** Invoke package/classes/java/org/rubyforge/rawr/Main.class (first_time, not_needed)
** Invoke src/org/rubyforge/rawr/Main.java (first_time, not_needed)
** Invoke package/classes/java (first_time, not_needed)
** Invoke package/classes/ruby/main.class (first_time, not_needed)
** Invoke src/main.rb (first_time, not_needed)
** Invoke package/classes/ruby (first_time, not_needed)
** Invoke package/classes/ruby/swt_wrapper.class (first_time, not_needed)
** Invoke src/swt_wrapper.rb (first_time, not_needed)
** Invoke package/classes/ruby (not_needed)
** Invoke package/classes/META-INF (first_time, not_needed)
** Invoke package/jar (first_time, not_needed)
** Execute rawr:jar
cp lib/java/jruby-complete.jar package/jar/lib/java/jruby-complete.jar
cp lib/java/swt_linux.jar package/jar/lib/java/swt_linux.jar
cp lib/java/swt_linux64.jar package/jar/lib/java/swt_linux64.jar
cp lib/java/swt_osx.jar package/jar/lib/java/swt_osx.jar
cp lib/java/swt_osx64.jar package/jar/lib/java/swt_osx64.jar
cp lib/java/swt_win32.jar package/jar/lib/java/swt_win32.jar
** Invoke package/windows (first_time, not_needed)
** Execute rawr:bundle:exe
Creating Windows application in package/jar/nms-helper.exe
rake aborted!
private method `split' called for nil:NilClass
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rawr-1.4.5/lib/exe_bundler.rb:92:in `deploy'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rawr-1.4.5/lib/rawr.rb:225:in `(root)'

org/jruby/RubyProc.java:276:in `call'
org/jruby/RubyProc.java:236:in `call'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `execute'
org/jruby/RubyArray.java:1671:in `each'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `execute'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:597:in `invoke_with_call_chain'
c:/jruby-1.6.0.RC1/lib/ruby/1.8/monitor.rb:191:in `mon_synchronize'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:583:in `invoke'

c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2051:in `invoke_task'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
org/jruby/RubyArray.java:1671:in `each'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2023:in `top_level'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2001:in `run'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run'
c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31:in `(root)'
org/jruby/RubyKernel.java:1066:in `load'

Aah, so things are coming undone in exe_bundler.rb:92:


 80       if Platform.instance.using_windows?
 81         # Check for FAT32 vs NTFS, the cacls command doesn't work on FAT32 nor is it required
 82         output = `fsutil fsinfo volumeinfo #{file_dir_name.split(':')[0]}:\\`
 83         # fsutil can only work with admin priviledges
 84         raise output if output =~ /requires that you have administrative privileges/
 85         # ===== Sample output of 'fsutil fsinfo volumeinfo c:\'
 86         # Volume Name :
 87         # Volume Serial Number : 0x80a5650a
 88         # Max Component Length : 255
 89         # File System Name : FAT32
 90         # Preserves Case of filenames
 91         # Supports Unicode in filenames
 92         if 'NTFS' == output.split("\n")[3].split(':')[1].strip
 93           sh "echo y | cacls \"#{file_dir_name}/launch4j/bin-win/windres.exe\" /G \"#{ENV['USERNAME']}\":F"
 94           sh "echo y | cacls \"#{file_dir_name}/launch4j/bin-win/ld.exe\" /G \"#{ENV['USERNAME']}\":F"
 95         end
 96         link_launch4j_bin('win', file_dir_name)

Line 92 seems to rely on what happens in line 82 where we try and get some filesystem info based on the drive name. The bit that tries to determine the drive letter uses '\\' at the end which is the issue. If you change that to '\\\\' you get things to work as they should again:


 82         output = `fsutil fsinfo volumeinfo #{file_dir_name.split(':')[0]}:\\\\`

Somehow the double backslash is not making it to the shell correctly from the backticks being called. Adding the extra set of backslashes saves the day:


C:\nms-helper4.git>rake rawr:bundle:exe
(in C:/nms-helper4.git)
cp lib/java/jruby-complete.jar package/jar/lib/java/jruby-complete.jar
cp lib/java/swt_linux.jar package/jar/lib/java/swt_linux.jar
cp lib/java/swt_linux64.jar package/jar/lib/java/swt_linux64.jar
cp lib/java/swt_osx.jar package/jar/lib/java/swt_osx.jar
cp lib/java/swt_osx64.jar package/jar/lib/java/swt_osx64.jar
cp lib/java/swt_win32.jar package/jar/lib/java/swt_win32.jar
Creating Windows application in package/jar/nms-helper.exe
echo y | cacls "c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rawr-1.4.5/lib/launch4j/bin-win/windres.exe" /G "cmatthee":F
Are you sure (Y/N)?echo y | cacls "c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rawr-1.4.5/lib/launch4j/bin-win/ld.exe" /G "cmatthee":F
Are you sure (Y/N)?call: java -jar "c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rawr-1.4.5/lib/launch4j/launch4j.jar" "package/windows/configuration.xml"
java -jar "c:/jruby-1.6.0.RC1/lib/ruby/gems/1.8/gems/rawr-1.4.5/lib/launch4j/launch4j.jar" "package/windows/configuration.xml"
launch4j: Compiling resources
launch4j: Linking
launch4j: Successfully created C:\nms-helper4.git\package\windows\nms-helper.exe

Reality restored.

From what I can see I am running the latest rawr gem:


C:\nms-helper4.git>jruby -S gem search -r rawr

*** REMOTE GEMS ***

drawr (1.0.1)
rawr (1.4.5)

C:\nms-helper4.git>jruby -S gem list rawr

*** LOCAL GEMS ***

rawr (1.4.5)

Checking the master branch on github though shows this file is no longer the same and the issue has been fixed:


81      if Platform.instance.using_windows?
82        # Check for FAT32 vs NTFS, the cacls command doesn't work on FAT32 nor is it required
83        volume = file_dir_name.split(":").first.upcase + ':'
84        output = `fsutil fsinfo ntfsinfo #{volume}`
85        # fsutil can only work with admin priviledges
86        raise output if output =~ /requires that you have administrative privileges/
87        raise output if output =~ /Error:/
88        if 'NTFS' == output.split("\n")[0][0..3]
89          sh "echo y | cacls \"#{file_dir_name}/launch4j/bin-win/windres.exe\" /G \"#{ENV['USERNAME']}\":F"
90          sh "echo y | cacls \"#{file_dir_name}/launch4j/bin-win/ld.exe\" /G \"#{ENV['USERNAME']}\":F"
91        end
92        link_launch4j_bin('win', file_dir_name)

The Rakefile refers us to 'rawr/rawr_version' that is:


1   module Rawr
2     VERSION = "1.5.0"
3   end

Looks like the latest version of the project on github has not made it into the wild as yet.

Resizing your OS X VMWare Fusion Virtual Disk

2011-02-03T13:39:00.000-08:00

Assumptions:

New disk size is 40GB.
Current virtual disk is foo.vdmk
Current VM image is at /Users/foo/Documents/Foo.vmwarevm

Quick recipe:

Open up a teminal window
Run: cd /Users/foo/Documents/Foo.vmwarevm && /Library/Application\ Support/VMware\ Fusion/vmware-vdiskmanager -x 40GB foo.vmdk
Resize the host OS with a third party tool. In my case I was resizing a NTFS partition with XP installed on it. Easeus Partition Manager home Edition did the trick for me.

TACACS+, packet loss and 'Authorization failed' errors

2010-10-21T20:40:00.000-07:00

I've been hunting a TACACS+ issue that a customer reported whereby they would be able to log into a network device but somewhere in the duration of the session they would try and run a command and get the dreaded '% Authorization failed' error.

After a couple of seconds they would be able to continue as if nothing was amiss.

The tac_plus (I am using the Event-Driven version) logs were not showing anything out of the ordinary so this was a rather perplexing issue. The error seemed to be popping up rather randomly (and always when I did not have a tcpdump running).

While observing several devices over a larger time period I noticed that they were experiencing intermittent packet loss and this got me to wonder of the packet loss events I was noticing were not coinciding with the authorization failures.

Lucky for me tac_plus is running on a Linux host and so I thought I'd simulate packet loss between one of my own devices and the tac_plus service to see what that would result in. To implement the simulated packet loss I simply put the following rule in place:

iptables -A INPUT -m static -mode random -probability 0.5 -s /32 -j DROP

With a probability of 0.5 (this can be between 0 and 1) I was dropping around 40-50% of the packets.

Lo and behold, '% Authorization failure', scientifically reproducible.

Sharing is caring: Ruby, Perl, Memcached and MsgPack

2010-09-01T16:42:00.000-07:00

Do you need to share data structures between ruby and perl ... FAST?

I recently saw MsgPack bubble through my RSS feeds and tagged it to go have another look. It provides very fast multi-language bindings for serialisation/de-serialisation.

A few quick experiments with the data I share (via memcached) between ruby and perl showed a write (serialisation + write to memcached) speed increase from 20s to 1.8s. Read (read from memcached + de-serialisation) performance showed similar performance increases.

All initial testing was done ruby -> memcached -> ruby but as soon as I switched to reading from memcached via perl I started getting 'extra bytes' errors from the perl side. I then tried perl -> memcached -> perl and everything was fine.

Weird.

A closer look at the data written to memcached and then read from perl showed that the data serialised with MsgPack on the ruby end was not the same as the data read by perl from memcached (validating the 'extra bytes' error).

Testing the write -> read process from perl to ruby yielded the following error:

/opt/local/lib/ruby/gems/1.8/gems/memcached-0.19.5/lib/memcached/memcached.rb:514:in `load': incompatible marshal file format (can't be read) (TypeError)

format version 4.8 required; 147.1 given

from /opt/local/lib/ruby/gems/1.8/gems/memcached-0.19.5/lib/memcached/memcached.rb:514:in `get'

from ./t_msgpack.rb:35:in `read_test'

from ./t_msgpack.rb:49

Now why on earth would I be getting a 'incompatible marshal file format' error as I am not using the ruby marshalling lib at all?

Turns out the memcached lib I use turns marshalling of ruby data on by default when you write to/read from memcached. This is most likely the best option for most cases where you don't want to use some other form of serialisation/de-serialisation but was really biting me here.

The solution is to simply stop the default behaviour of the memcached lib by using the following forms of get and set that turns on the 'raw' data handling switch for the memcached lib:

get KEY, false

set KEY, VALUE, TTL, false

The 'false' parameter at the end of those overrides the default behaviour turning default serialisation/de-serialisation via Marshall off.

Reality restored.

How do you make your Clojure REPL suck less?

2010-07-01T19:50:00.000-07:00

Simple, rely on the venerable rlwrap that provides you with a readline wrapper around your existing REPL.

My sucky REPL looked like this:

#!/bin/sh
java -cp PATH_TO/clojure.jar clojure.main $1

Simply install the rlwrap package using your favorite package manager and change your REPL script (clj) to the following:

#!/bin/sh
rlwrap java -cp PATH_TO/clojure.jar clojure.main $1

Major suckiness averted. The added boon of this approach is that you now get all the readline goodness (history traversal, inline editing, etc.) you've come to depend on in other REPLs.

B0rked ports on Snow Leopard

2009-10-12T22:20:00.000-07:00

If you are using MacPort to manage OSS on your box and you recently upgraded to Snow Leopard you will find error messages like this when trying to use the ports system:

$ port selfupdate
dlopen(/Library/Tcl/macports1.0/MacPorts.dylib, 10): no suitable image
found.  Did find:
/Library/Tcl/macports1.0/MacPorts.dylib: mach-o, but wrong architecture
while executing
"load /Library/Tcl/macports1.0/MacPorts.dylib"
("package ifneeded macports 1.0" script)
invoked from within
"package require macports"
(file "/opt/local/bin/port" line 39)

The issue is simply that you have all your libs and binaries managed via MacPorts compiled for the i386 architecture and not x86-64 (as required for Snow Leopard).

The solution is simple, but, arduous. You need to install the new version of MacPorts for Snow Leopard, make a backup list of the installed ports, delete them and reinstall the ones you still want.

You may want to pay close attention to the variants of the ports that you had previously installed when reinstalling them.

Also, you need the latest version of Xcode (a version greater than v3.0 will do) installed.

Fun.

See the following two URLs for more info:

Happy recompiling!

OS X Snow Leopard and broken scrolling in MacVim

2009-09-22T22:02:00.001-07:00

My upgrade to Snow Leopard has been pretty smooth sailing bar one annoying hitch. After the upgrade MacVim stopped scrolling properly.

When you scroll down only the last few lines on the screen update and scrolling up only the first few lines. This requires you CTRL-L to redraw the window every time you've finished scrolling.

Yuck!

Looks like this is caused by MacVim's support for ATSUI which has been deprecated in favour of Core Text in Snow Leopard.

If you are experiencing this you can simply turn the ATSUI renderer off by unchecking MacVim -> Preferences -> Advanced -> Use ATSUI renderer.

This is switched off by default so most people won't be affected by this. At the time of this post there was no real indication from the MacVim project if they'd be switching to Core Text in the future in addition to using the ATSUI renderer.

iTerm, X11.app and the case of the missing DISPLAY

2009-09-13T16:03:00.000-07:00

If you're running X11.app as your X server on OS X you generally need to use the xterm window it launches for you as part of the system or your specific xinitrc to connect to remote hosts via SSH when you want to enable X11 forwarding (and the propagation of your DISPLAY variable).

I prefer iTerm and luckily the solutions to switch to using it instead of the xterm launched by X11 is pretty easy to implement.

The first option is to make a user specific copy of xinitrc and modify the line that launches xterm to rather launch iTerm. If you don't have a user specific xinitrc (~/.xinitrc) you can grab a copy from /etc/X11/xinit/xinitrc or /usr/X11R6/lib/X11/xinit/xinitrc.

Near the bottom of the xinitrc you'll see a few lines that reference xterm that you can comment our and substitute with:


/usr/bin/open /Applications/iTerm.app

If you access remote X applications very infrequently it may simply be easier to ensure that X11 is running and then export your DISPLAY variable manually in iTerm:


$ export DISPLAY=:0

The final solution is to make all terms aware of your DISPLAY variable by editing your local environment plist (~/.MacOSX/environment.plist):


$ open ~/.MacOSX/environment.plist

This will open the plist in the Property List Editor where you can add a new child (DISPLAY) under the root node with a value of ':0'. For this to take effect you need to log out/in. You could also have dropped 'export DISPLAY=:0' into your local ~/.bash_profile to achieve the same results.

Untrusted SSH clients or Gdk-WARNING **: Connection to display localhost:10.0 appears to be untrusted

2009-09-13T15:48:00.000-07:00

While connecting to a local RH system via 'ssh -X' I was seeing errors like:


Connection to display localhost:10.0 appears to be untrusted. Pointer and

keyboard grabs and inter-client communication may not work as expected.

OpenSSH implements the X11 SECURITY extension that provides two clients types: trusted & untrusted.

Trusted clients can do anything with the display while untrusted clients cannot inject synthetic events (mouse movement, keypresses, etc.) or read data from other windows (e.g. take screenshots).

In theory you should be able to run most applications via the untrusted category but it seems that a fair amount of client apps still do not implement this correctly so YMMV.

If you want to initialize a SSH connection with X11 forwarding support and using the trusted security type you simply replace -X with -Y:


$ ssh -Y USER@HOST

TIP: Add a '-C' to your ssh connections to speed things up. Nothings for free but I am sure the CPU/speed trade-off will suit most users.

Installing ghc on OS X and getting "./configure: line 11082: syntax error: unexpected end of file"

2009-09-11T21:46:00.000-07:00

While trying to install ghc from MacPorts I kept getting this:

[...]
--->  Fetching ghc
--->  Attempting to fetch ghc-6.10.3-src.tar.bz2 from

http://distfiles.macports.org/ghc
--->  Attempting to fetch ghc-6.10.3-src-extralibs.tar.bz2 from

http://distfiles.macports.org/ghc
--->  Attempting to fetch testsuite-6.10.3.tar.bz2 from

http://distfiles.macports.org/ghc
--->  Attempting to fetch ghc-6.8.2-darwin-i386-leopard-bootstrap.tar.bz2 from

http://distfiles.macports.org/ghc
--->  Verifying checksum(s) for ghc
--->  Extracting ghc
--->  Applying patches to ghc
--->  Configuring ghc
Error: Target org.macports.configure returned: configure failure: shell command

" cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.

macports.org_release_ports_lang_ghc/work/ghc-6.10.3" && ./configure

--prefix=/opt/local

--prefix=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.

macports.org_release_ports_lang_ghc/work/destroot/opt/local

--datadir=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.

macports.org_release_ports_lang_ghc/work/destroot/opt/local/share/ghc-6.10.3

--with-gmp-includes=/opt/local/include --with-gmp-libraries=/opt/local/lib

--with-ghc='/opt/local/var/macports/build/

_opt_local_var_macports_sources_rsync.macports.org_release_ports_lang_ghc/

work/ghc-bootstrap/bin/ghc' --with-gcc=/usr/bin/gcc-4.0 " returned error 2

Command output: checking build system type... i386-apple-darwin9.7.0
checking host system type... i386-apple-darwin9.7.0
checking target system type... i386-apple-darwin9.7.0
Canonicalised to: i386-apple-darwin
checking version of ghc... 6.8.2
checking for nhc... no
checking for nhc98... no
checking for hbc... no
checking for ld... /usr/bin/ld
./configure: line 11082: syntax error: unexpected end of file

Turns out there's some old cruft in the ghc autoconf macros that were causing the break. The fix was a simple:


$ sudo port selfupdate && sudo port clean ghc && sudo port install ghc

You can see the filed bug and changeset at MacPorts for more information.

Update: Home made nginx + phusion passenger OS X port

2009-09-01T15:56:00.000-07:00

The new v2.2.5 version of Phusion Passenger has been released.

If you want to roll this into OS X without having to use their installer you can simply follow the previous article to get this done.

Important things to note is that wherever you used /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4* you would now use /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.5* (the Portfile and the nginx.conf come to mind).

Mounting a disk image with Windows tools only

2009-07-24T16:03:00.000-07:00

Coming from the *NIX/Mac world I frequently find myself mounting ISO images as local disks for software installation, etc.

The same requirement recently came up for me on a Windows platform and I was a little stumped as I did not want to spend any money on this (as I see this as an intrinsic part of the the OS) by buying Alcohol or Daemon Tools.

To my surprise Microsoft actually has a (unsupported) tool (Virtual CD-ROM Control Panel) and driver that provides virtual CD-ROM services.

Unfortunately the solution is XP-only. Download the executable above and install the service. Its readme is quite short and to the point so getting this going is pretty easy.

Have fun!

Auto-sizing jQuery-UI Dialog

2009-07-08T17:59:00.000-07:00

I recently upgraded all my jquery and jquery-ui (and some supporting) libs. Part of this upgrade was also to get the latest version of nyroModal working.

After several hours of chasing issues with the lib I decided to rip it out of my app and rather use the jquery-ui dialog plugin that ships with the native lib.

A large bugbear with nyroModal has always been that I could never get the dialogs to flow around the different content lengths that I was populating them with properly. After retrofitting the jquery-ui dialog plugin I ran into the similar issues I had with nyroModal.

Whenever the content of my dialog was greater than the distance from the top of the dialog window to the bottom of the viewport, the dialog would simply disappear off the viewport (regardless of whether it had a scrollbar or not).

What I wanted was for the dialog to automagically adjust its vertical height based on the size of the content it contains relative to the top of the dialog position and the bottom of the viewport.

The fact that I am using FBBorderLayout (yep, I am aware this project has been superseded by the jQuery UI plugin but I am taking baby steps here) to set up a nice layout that stops the page from scrolling down and contributes to the dialog sizing off the viewport.

The scene
Based on the documentation on the jquery-ui dialog plugin page I created something like this.

As you can see from this page the dialog stretched off the screen with no scrollbar or way to scroll down to see the content of the dialog.

Styling
Ok, not quite what we wanted so I added the following style code to the head to try and get the dialog container to automatically flow and add a scrollbar if the content was too big.

And? It had no effect whatsoever. I tried all possible styling that I could find to try and get the dialog container to play nice but to no avail.

Unlikely rescuer
I scoured the web for a possible solution. All I found were a lot of questions about this phenomenon but no real usable solutions or suggestions (beyond the styling I'd already tried).

I then thought of trying to calculate the height programatically and set the height of the dialog contents prior to displaying it.

You'll notice I stopped the dialog from displaying automatically by adding the 'autoOpen: false' option to the dialog initialization. This is required so that I am able to calculate the height before displaying the dialog.

Finally
Here's the complete source file for posterity.

Enjoy!

Installing the MySQL gem on OS X

2009-07-05T18:58:00.001-07:00

If you manage your system packages via the MacPorts system you may run into some problems when trying to install the native MySQL driver gem on OS X.

Typically, you'd see output like this when trying to install the gem:


$ sudo gem install mysql
Building native extensions.  This could take a while...
ERROR:  Error installing mysql:
        ERROR: Failed to build gem native extension.

/opt/local/bin/ruby extconf.rb
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... no
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/opt/local/bin/ruby
        --with-mysql-config
        --without-mysql-config
        --with-mysql-dir
        --without-mysql-dir
        --with-mysql-include
        --without-mysql-include=${mysql-dir}/include
        --with-mysql-lib
        --without-mysql-lib=${mysql-dir}/lib
        --with-mysqlclientlib
        --without-mysqlclientlib
        --with-mlib
        --without-mlib
        --with-mysqlclientlib
        --without-mysqlclientlib
        --with-zlib
        --without-zlib
        --with-mysqlclientlib
        --without-mysqlclientlib
        --with-socketlib
        --without-socketlib
        --with-mysqlclientlib
        --without-mysqlclientlib
        --with-nsllib
        --without-nsllib
        --with-mysqlclientlib
        --without-mysqlclientlib


Gem files will remain installed in /opt/local/lib/ruby/gems/1.8/gems/mysql-2.7 for inspection.
Results logged to /opt/local/lib/ruby/gems/1.8/gems/mysql-2.7/gem_make.out

Based on those errors the configure process seems to be failing when looking for MySQL libs. The default path to the libs based on extconf.rb seems to be /usr/local which is not where the ports system installs MySQL.

All you need to do is point to the correct mysql_config and let the gem install command line know of this to get things going:


$ sudo gem install mysql -- --with-mysql-config=/opt/local/bin/mysql_config5  
Building native extensions.  This could take a while...
Successfully installed mysql-2.7
1 gem installed

All's well that ends well.

nginx/passenger: Exception RangeError in PhusionPassenger::Railz::ApplicationSpawner (bignum too big to convert into `long')

2009-07-01T20:41:00.000-07:00

Nasty.

I am running nginx+passenger on my OS X laptop and after getting the passenger module installed I got the following stacktrace in /opt/local/var/log/nginx/error.log:


*** Exception RangeError in PhusionPassenger::Railz::ApplicationSpawner (bignum too big to convert into `long') (process
 56051):
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:363:in `setgid'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:363:in `switch_to_user'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:328:in `lower_privilege'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:250:in
 `initialize_server'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:230:in `report_app_init_st
atus'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:237:in
 `initialize_server'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:193:in `start_sy
nchronously'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:162:in `start'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:213:in
 `start'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:261:in `spawn_rail
s_application'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server_collection.rb:126:i
n `lookup_or_add'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:255:in `spawn_rail
s_application'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server_collection.rb:80:in
 `synchronize'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server_collection.rb:79:in
 `synchronize'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:254:in `spawn_rail
s_application'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:153:in `spawn_appl
ication'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:286:in `handle_spa
wn_application'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:351:in `__send__
'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:351:in `main_loo
p'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:195:in `start_sy
nchronously'
        from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/bin/passenger-spawn-server:61

So, what's up?

The old switcheroo
Passenger implements User Switching which works around the age old problem that plagues most other scripting languages where they run in the same context as the web server they are accessed through.

The default behaviour is for rails apps to be started as the owner of config/environment.rb. There are however some points that should be noted:

The owner of environment.rb must have read access to the rails application folder, and read/write access to the rails application logs folder.

This feature is only available if nginx is started by root. This is the case on most nginx installations.

Under no circumstances will applications be run as root. If environment.rb is owned as root or by an unknown user, then the rails application will run as the user specified by passenger_default_user.

In my case the project was owned by root so changing that to a local user fixed the problem.

Home made nginx + phusion passenger OS X port

2009-07-01T19:06:00.000-07:00

The instructions to do this are similar to those in my previous article.

Install passenger gem
The first step is to install the Phusion Passenger gem:

$ sudo gem install -V -r passenger
[...]
Installing RDoc documentation for passenger-2.2.4...

Take note of where the passenger gem installed the nginx module code (ext/nginx/*) that will be required further down. In this case the install directory will be /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/.

Compile nginx passenger module
You also need to compile the passenger nginx module if it was not already compiled:


$ cd /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/
$ sudo rake nginx
[...]
g++ ext/nginx/HelperServer.cpp -o ext/nginx/HelperServer -Iext -Iext/common -D_REENTRANT -I/usr/local/include -Wall -g -DPASSENGER_DEBUG -DBOOST_DISABLE_ASSERTS  ext/nginx/libpassenger_common.a ext/nginx/libboost_oxt.a -lpthread

Modify the nginx Portfile
Ensure you have nginx port installed. Your Portfile will most likely be in /opt/local/var/macports/sources/rsync.macports.org/release/ports/www/nginx/Portfile.

Add the following variant block to the end of the file:


variant passenger description {Enable Phusion Passenger (mod_rails) support} {
    configure.args-append   --add-module=/opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/nginx
}

Rebuild/install nginx
When you try and reinstall nginx now it will use the updated Portfile and add the passenger module to the mix:


$ sudo port uninstall nginx && sudo port clean nginx
$ sudo port -vvvvv install nginx +passenger

Nginx passenger configuration
To enable and take advantage of the passenger nginx module you need to edit the /opt/local/etc/nginx/nginx.conf config. Here's an example:


#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
  include       mime.types;
  default_type  application/octet-stream;

  #log_format  main  '$remote_addr - $remote_user [$time_local] $request '
  #                  '"$status" $body_bytes_sent "$http_referer" '
  #                  '"$http_user_agent" "$http_x_forwarded_for"';

  #access_log  logs/access.log  main;

  sendfile        on;
  #tcp_nopush     on;

  #keepalive_timeout  0;
  keepalive_timeout  65;

  gzip  on;

  # Enable Phusion Passenger nginx module
  passenger_root /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4;
  passenger_ruby /opt/local/bin/ruby;

  server {
    listen 80;
    server_name foo.example.com;
    root /var/www/foo/public;   # <--- be sure to point to 'public'!
    passenger_enabled on;
  }
}

Ensure that the port, server_name and root variables have valid values for your environment.

If you want to use a non-production rails environment you can add the following to the block after passenger_enabled:


      rails_env development;

Once done you can restart nginx and you should see the following processes that confirm things are running:


$ ps axw | egrep "nginx|passenger"
55935   ??  Ss     0:00.00 nginx: master process nginx
55936   ??  S      0:00.00 nginx: worker process
55925 s001  S      0:00.01 PassengerNginxHelperServer /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4 /opt/local/bin/ruby 3 4 0 6 0 300 1 nobody 4294967294 4294967294 /tmp/passenger.55923
55938 s001  R+     0:00.00 egrep nginx|passenger

The final test to see if everything is working is to open a browser and to browse to the server_name and port defined above. Errors will be reported in /opt/local/var/log/nginx/error.log as well as the development/production logs of the rails project you're running.

Home made nginx + phusion passenger Debian/Ubuntu dep

2009-07-01T17:37:00.000-07:00

If you are using Ubuntu Hardy then you could try using Brightbox's repo instead of building your own deb.

If you are however running some other distro then the following (non-definitive) instructions are for you.

Warning: it may be safer to install nginx + passenger via the passenger-install-nginx-module script as it will match passenger with a tested version of nginx.

The instructions below simply add passenger support to the latest nginx deb in your distro of choice.

Install passenger gem
The first step is to install the Phusion Passenger gem:

$ sudo gem install -V -r passenger
[...]
Installing RDoc documentation for passenger-2.2.4...

Take note of where the passenger gem installed the nginx module code that will be required further down:

$ gem contents passenger | egrep '/ext/nginx/' | head -n 1
/var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx/Configuration.c

In this case the install directory will be /var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx.

Compile nginx passenger module
You also need to compile the passenger nginx module if it was not already compiled:


$ cd /var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx
$ sudo rake nginx

(in /var/lib/gems/1.8/gems/passenger-2.2.4)
mkdir -p ext/nginx/libboost_oxt/boost
g++ -Iext  -D_REENTRANT -I/usr/local/include -Wall -g -DPASSENGER_DEBUG -DBOOST_DISABLE_ASSERTS -o ext/nginx/libboost_oxt/boost/exceptions.o -c ext/boost/src/pthread/exceptions.cpp
[...]
g++ ext/nginx/HelperServer.cpp -o ext/nginx/HelperServer -Iext -Iext/common -D_REENTRANT -I/usr/local/include -Wall -g -DPASSENGER_DEBUG -DBOOST_DISABLE_ASSERTS  ext/nginx/libpassenger_common.a ext/nginx/libboost_oxt.a -lpthread

Install build packages


$ sudo aptitude install fakeroot debhelper dpkg-dev autotools-dev libpcre3-dev libssl-dev

Modify APT sources.list
Open /etc/apt/sources.list and ensure you add deb-src lines for each of the universe repos. The easiest way to do this is to simply copy the universe repo lines that exist and change the starting deb to deb-src:


deb http://au.archive.ubuntu.com/ubuntu/ jaunty universe
deb-src http://au.archive.ubuntu.com/ubuntu/ jaunty universe
deb http://au.archive.ubuntu.com/ubuntu/ jaunty-updates universe
deb-src http://au.archive.ubuntu.com/ubuntu/ jaunty-updates universe

Replace 'au.archive.ubuntu.com' with the repo closest to you and also replace 'jaunty' with your distro. Ensure you do a 'sudo aptitude update' to load the latest package indexes.

Set up a packaging environment


$ mkdir -p /var/tmp/nginx-passenger && cd /var/tmp/nginx-passenger
$ apt-get source nginx
Reading package lists... Done
Building dependency tree      
Reading state information... Done
NOTICE: 'nginx' packaging is maintained in the 'Svn' version control system at:
svn://svn.debian.org/svn/collab-maint/deb-maint/nginx/trunk
Need to get 537kB of source archives.
Get:1 http://au.archive.ubuntu.com jaunty/universe nginx 0.6.35-0ubuntu1 (dsc) [1305B]
Get:2 http://au.archive.ubuntu.com jaunty/universe nginx 0.6.35-0ubuntu1 (tar) [525kB]
Get:3 http://au.archive.ubuntu.com jaunty/universe nginx 0.6.35-0ubuntu1 (diff) [11.0kB]
Fetched 537kB in 0s (1504kB/s)
gpg: Signature made Sat 21 Feb 2009 07:02:34 AM EST using DSA key ID 3FE63E00
gpg: Can't check signature: public key not found
dpkg-source: extracting nginx in nginx-0.6.35
dpkg-source: info: unpacking nginx_0.6.35.orig.tar.gz
dpkg-source: info: applying nginx_0.6.35-0ubuntu1.diff.gz
$ ls -la
total 548
drwxr-xr-x 3 root root   4096 2009-06-29 08:55 .
drwxrwxrwt 3 root root   4096 2009-06-29 08:39 ..
drwxr-xr-x 8 root root   4096 2009-06-29 08:55 nginx-0.6.35
-rw-r--r-- 1 root root  11018 2009-02-21 08:04 nginx_0.6.35-0ubuntu1.diff.gz
-rw-r--r-- 1 root root   1305 2009-02-21 08:04 nginx_0.6.35-0ubuntu1.dsc
-rw-r--r-- 1 root root 524987 2009-02-21 08:04 nginx_0.6.35.orig.tar.gz
$ rm -f nginx_*
$ mv nginx-0.6.35 nginx-passenger-0.6.35

Package customizations
In order to build our nginx-passenger custom package we need to modify the following packaging resources:

control file

changelog file

rules file

nginx.install file

nginx.logrotate file

passenger.conf file

postinst file

Control file


$ cd nginx-passenger-0.6.35/debian

Edit the control file and change the Source, Maintainer, Uploaders, Package, Provides, Conflicts and Description fields:


Source: nginx-passenger
Maintainer: Charl Matthee
Uploaders: Charl Matthee
Package: nginx-passenger
Provides: httpd, nginx
Conflicts: nginx
Description: small, but very powerful and efficient web server
 Nginx (engine x) is a web server created by Igor Sysoev and kindly provided
 to the open-source community. This server can be used as standalone HTTP
 server and as a reverse proxy server before some Apache or another big
 server to reduce load to backend servers by many concurrent HTTP-sessions.
 .
 This is a custom nginx package that includes Phusion Passenger
 support for nginx.

Note: spacing in the control file is significant.

More information on the control file internals can be found here.

Changelog
Add a new changelog entry in the changelog file:


nginx-passenger (0.6.35-0custom1) jaunty; urgency=low

  * New custom Passenger Phusion nginx module addition

 -- Charl Matthee   Mon, 29 Jun 2009 09:08:36 +0100

nginx (0.6.35-0ubuntu1) jaunty; urgency=low

  * New upstream bugfix release

 -- Daniel Hahler   Fri, 20 Feb 2009 21:01:23 +0100
[...]

More information on the changelog file internals can be found here.

Rules
Add the passenger module inclusion to the configure script options when nginx is built by modifying the './configure $(CONFIGURE_OPTS)' block in the rules file like this:


        ./configure $(CONFIGURE_OPTS) --conf-path=/etc/nginx/nginx.conf \
        --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid \
        --lock-path=/var/lock/nginx.lock   --http-log-path=/var/log/nginx/access.log \
        --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy \
        --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-http_stub_status_module \
        --with-http_flv_module --with-http_ssl_module --with-http_dav_module \
        --with-http_realip_module --add-module=PASSENGER_LIB_DIR

Where PASSENGER_LIB_DIR is the path you noted above when you installed the passenger gem. So, the updated block looks like this:


        ./configure $(CONFIGURE_OPTS) --conf-path=/etc/nginx/nginx.conf \
        --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid \
        --lock-path=/var/lock/nginx.lock   --http-log-path=/var/log/nginx/access.log \
        --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy \
        --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-http_stub_status_module \
        --with-http_flv_module --with-http_ssl_module --with-http_dav_module \
        --with-http_realip_module --add-module=/var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx/

More infomration on the changelog file internals can be found here.

nginx.install
Move this file to the correct name based on the package name and add the installation instructions for the default nginx configuration that is required to turn passenger support on:


$ mv nginx.install nginx-passenger.install
$ cat nginx-passenger.install
objs/nginx usr/sbin
html/* var/www/nginx-default
debian/conf/* etc/nginx
debian/passenger.conf etc/nginx/conf.d/
debian/NEWS.Debian usr/share/doc/nginx

nginx.logrotate
Move this file to the correct name based on the package name:


$ mv nginx.logrotate nginx-passenger.logrotate

passenger.conf
Create the following file in the debian directory:


    passenger_root /var/lib/gems/1.8/gems/passenger-2.2.4;
    passenger_ruby /usr/bin/ruby1.8;

Ensure those paths point to the correct passenger isntall and ruby directories.

postinst
Add the following steps to the postinst script after the '#DEBHELPER#' token:


# Phusion Passenger module settings for nginx
echo
echo "***********************************************************************"
echo "For each virtual host you want to add Phusion Passenger support for you"
echo "need to create a config like this in /etc/nginx/site-available:"
echo
echo "    server {
      listen 80;
      server_name DOMAIN_NAME;
      root /var/www/RAILS_PROJECT/public; #<--- be sure to point to 'public'!
      passenger_enabled on;
    }"
echo
echo "Don't forget to add a symlink from this file(s) to"
echo "/etc/nginx/site-enabled to enable the configuration in nginx."
echo "***********************************************************************"
echo

This will warn administrators that install this package that they still need to do some manual work to get a virtual host set up to take advantage of the passenger module.

Package rebuild
The package can then be rebuilt just like any other custom deb:


$ cd /var/tmp/nginx-passenger/nginx-passenger-0.6.35
$ dpkg-buildpackage -rfakeroot
dpkg-buildpackage: set CFLAGS to default value: -g -O2
[...]
dpkg-deb: building package `nginx-passenger' in `../nginx-passenger_0.6.35-0custom1_i386.deb'.
dpkg-deb: ignoring 1 warnings about the control file(s)
 signfile nginx-passenger_0.6.35-0custom1.dsc
gpg: skipped "Charl Matthee ": secret key not available
gpg: [stdin]: clearsign failed: secret key not available

dpkg-genchanges  >../nginx-passenger_0.6.35-0custom1_i386.changes
dpkg-genchanges: warning: the current version (0.6.35-0custom1) is smaller than the previous one (0.6.35-0ubuntu1)
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload; Debian-native package (full source is included)
dpkg-buildpackage: warning: Failed to sign .dsc and .changes file

Those signing errors were triggered because I have did not have the required key infrastructure installed. They can be ignored for the moment.

Install the new package:


$ sudo dpkg -i ../nginx-passenger_0.6.35-0custom1_i386.deb
[...]
Setting up nginx-passenger (0.6.35-0custom1) ...

***********************************************************************
Create a config like this for each virtual host you want to add Phusion
Passenger support to. They should be created in
/etc/nginx/site-available:

    server {
      listen 80;
      server_name DOMAIN_NAME;
      root /var/www/RAILS/public;     # <--- be sure to point to 'public'!
      passenger_enabled on;
    }

Don't forget to add a symlink from this file(s) to
/etc/nginx/site-enabled to enable the configuration in nginx.
***********************************************************************
[...]

Nginx passenger configuration
To enable and take advantage of the passenger nginx module you need to create a /etc/nginx/conf.d/passenger.conf config:


    passenger_root /var/lib/gems/1.8/gems/passenger-2.2.4;
    passenger_ruby /usr/bin/ruby1.8;

This will of course have been done when we installed our custom package.

Next, add a virtual host for each of the rails applications you want to support by dropping a file like 'foo' into /etc/nginx/sites-available/:


    server {
      listen 80;
      server_name foo.example.com;
      root /var/www/foo/public;   # <--- be sure to point to 'public'!
      passenger_enabled on;
    }


      rails_env development;

To enable the virtual host(s) you need to ensure that there's a symlink linking the virtual host in sites-available to sites-enabled:


$ sudo ln -s /etc/nginx/sites-available/foo /etc/nginx/sites-enabled/foo
$ ls -la /etc/nginx/sites-enabled
total 8
drwxr-xr-x 2 root root 4096 2009-06-29 12:33 .
drwxr-xr-x 5 root root 4096 2009-06-29 10:51 ..
lrwxrwxrwx 1 root root   34 2009-06-02 07:25 default -> /etc/nginx/sites-available/default
lrwxrwxrwx 1 root root   30 2009-06-29 12:33 foo -> /etc/nginx/sites-available/foo

Once done you can restart nginx and you should see the following processes that confirm things are running:


$ ps axf | egrep "nginx|passenger"
28608 pts/1    S+     0:00                      \_ egrep nginx|passenger
28560 pts/1    Sl     0:00 PassengerNginxHelperServer /var/lib/gems/1.8/gems/passenger-2.2.4 /usr/bin/ruby1.8 3 4 0 6 0 300 1 nobody 33 33 /tmp/passenger.28557
28580 ?        Ss     0:00 nginx: master process /usr/sbin/nginx
28581 ?        S      0:00  \_ nginx: worker process

The final test to see if everything is working is to open a browser and to browse to the server_name and port defined above. Errors will be reported in /var/log/nginx/error.log as well as the development/production logs of the rails project you're running.

Commandline PDF Manipulation

2009-06-11T18:07:00.001-07:00

Want to do any of the following from a *NIX based system commandline with PDF documents:

Merge PDF Documents
Split PDF Pages into a New Document
Rotate PDF Pages or Documents
Decrypt Input as Necessary (Password Required)
Encrypt Output as Desired
Fill PDF Forms with FDF Data or XFDF Data and/or Flatten Forms
Apply a Background Watermark or a Foreground Stamp
Report on PDF Metrics such as Metadata, Bookmarks, and Page Labels
Update PDF Metadata
Attach Files to PDF Pages or the PDF Document
Unpack PDF Attachments
Burst a PDF Document into Single Pages
Uncompress and Re-Compress Page Streams
Repair Corrupted PDF

One way to accomplish any of these would be to use pdftk (PDF Toolkit).

Something I need to do quite often is concatenate (join) several PDF documents into one document. This can quite trivially be achieved with the following from the commandline:

$ pdftk 01.pdf 02.pdf cat output all_together_now.pdf

Above we concatenated 01.pdf and 02.pdf to form the new combined all_together_now.pdf document.

Broken memcached gem/libmemchached deb on Ubuntu

2009-04-22T00:19:00.000-07:00

This post follows on a previous post that was OS X centric.

To fix this issue follow the same steps (starting with the download of libmemcached-0.25.14.tar.gz) detailed here.

Because the source installation of libmemcached dropped the libs in a non-standard place (from a Debian/Ubuntu viewpoint) we need a little more magic before things will work.

As root, create /etc/ld.so.conf.d/libmemcached.conf:

# Manual installation of libmemcached dropped the libs in a non-standard place
/usr/local/lib

Run ldconfig to load the new config and test to ensure your script can now access libmemcached properly through the memcached gem.

Unpacking a RPM on Ubuntu (or Debian)

2009-03-22T18:35:00.000-07:00


# alien --scripts PACKAGE.rpm
# ar -x PACKAGE.deb

You now have two tarballs control.tar.gz and data.tar.gz.

The first is the information dpkg needs to do a proper installation and configuration of the package, the second is contains the binaries and data files.

Broken memcached gem/libmemchached port on OS X

2009-02-23T15:13:00.000-08:00

Update (2009/04/22):
This post was originally only targeted at OS X but I have since noticed Ubuntu also having this same issue. Jump to my additional instructions for Ubuntu.

Recipe for disaster:

Use MacPorts for your package management
Use the ruby memcached gem
Use the libmemcached port as a dependency on the gem above
Upgrade your ports

Synopses
I upgraded all my ports on my MBP recently and one of the libs that was upgraded in the process was libmemcached (upgraded from v0.25 to v0.26). Unfortunately this has broken my access to memcached on my system (the script dies when trying to connect).

Not knowing what the problem may be I also upgraded my gem to v0.14 which then could not build. A quick search for the issue found this entry on the memcached discussion forum.

How now brown cow?
Evan Weaver, the maintainer of the memcached project, mentioned that libmemcached v0.26 lacks some critical patch and suggested people may have to wait for v0.27 to rectify things.

For those who cannot wait and have inconsistent, broken systems, Evan goes on to mention that you should currently use the libmemcached lib and memcached gem pair from here.

Here he also provides a convenient compatibility matrix so you can see how to align the different versions of libmemcached and the memcached gem.

Fix
Unfortunately this messes with things because libmemcached now needs to be installed as a source package in a system managed with MacPorts.

Yuck!

Unfortunately the choices are limited as the versions of libmemcached that ships as a port are either v0.26 (the latest, but broken for our purposes) or v0.25 (which aligns with the older v0.13 of the memcached gem).

First off remove any libmemcached ports you may have installed:

$ sudo port uninstall libmemcached

Download the libmemcached source archive, extract, build and install it:

ibm-99tvvxc:msp charl$ cd /var/tmp/
$ wget http://blog.evanweaver.com/files/libmemcached-0.25.14.tar.gz
--2009-02-24 10:50:40--  http://blog.evanweaver.com/files/libmemcached-0.25.14.tar.gz
Resolving blog.evanweaver.com... 208.78.102.192
Connecting to blog.evanweaver.com|208.78.102.192|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 428047 (418K) [application/x-gzip]
Saving to: `libmemcached-0.25.14.tar.gz'

100%[==========================================================>] 428,047      135K/s   in 3.1s 

2009-02-24 10:50:49 (135 KB/s) - `libmemcached-0.25.14.tar.gz' saved [428047/428047]

$ tar zxvf libmemcached-0.25.14.tar.gz
libmemcached-0.25.14/
[...]
$ cd libmemcached-0.25.14
$ ./configure
checking build system type... i386-apple-darwin9.6.0
checking host system type... i386-apple-darwin9.6.0
checking target system type... i386-apple-darwin9.6.0
checking for a BSD-compatible install... /usr/bin/install -c
[...]
$ make
Making all in docs
/usr/bin/pod2man -c "libmemcached" -r "" -s 3 libmemcached.pod > libmemcached.3
/usr/bin/pod2man -c "libmemcached" -r "" -s 3 libmemcached_examples.pod > libmemcached_examples.3
[...]
$ sudo make install
Password:
Making install in docs
make[2]: Nothing to be done for `install-exec-am'.
test -z "/usr/local/share/man/man1" || ../config/install-sh -c -d "/usr/local/share/man/man1"
/usr/bin/install -c -m 644 './memcat.1' '/usr/local/share/man/man1/memcat.1'
[...]

You may want to keep the source directory around so that you can easily uninstall (run sudo make uninstall in the source directory) the source installation later when the relevant aligned packaged versions are available for installation.

The final step is to install the gem:

$ sudo env ARCHFLAGS="-arch i386" gem install -r -V memcached --no-rdoc --no-ri
GET 200 OK: http://gems.rubyforge.org/latest_specs.4.8.gz
GET 200 OK: http://gems.github.com/latest_specs.4.8.gz
connection reset after 2 requests, retrying
GET 200 OK: http://gems.rubyforge.org/quick/Marshal.4.8/memcached-0.14.gemspec.rz
Installing gem memcached-0.14
/opt/local/lib/ruby/gems/1.8/gems/memcached-0.14/BENCHMARKS
/opt/local/lib/ruby/gems/1.8/gems/memcached-0.14/CHANGELOG
[...]
Successfully installed memcached-0.14
1 gem installed

Success!

We now have libmemcached v0.25.14 and the memcached v0.14 gem installed and we can go on with our lives.

Silly Rails Environment Optimisation

2009-02-22T12:18:00.000-08:00

Do you find supplying the ./script base directory for any of the commands you want to run in it annoying?

Why not practice some Convention over Configuration and assume these commands will always be run from within a valid rails project directory.

Based on this assumption you can drop the following into your .bash_profile (OS X) or some system-wide profile to facilitate some laziness:

# Rails aliases
alias about="./script/about"
alias console="./script/console"
alias dbconsole="./script/dbconsole"
alias destroy="./script/destroy"
alias generate="./script/generate"
alias performance="./script/performance"
alias plugin="./script/plugin"
alias process="./script/process"
alias runner="./script/runner"
alias server="./script/server"

Don't forget to load it into you current shell with:

$ . ~/.bash_profile

Now you can simply run 'console', 'dbconsole', 'server', etc. when you're in a valid rails project directory without having to append that annoying base directory.

Caveat: If similar commands exist on your system or are introduced by a non-rails package a conflict will arise and the alias version of the command will win. Also, TAB-completion now no longer works for these commands and you'll have to type them out in their entirety.

What's next with acts_as_state_machine?

2009-02-20T18:04:00.000-08:00

I've been using acts_as_state_machine and there seems to be at least two convenience methods missing that I find quite useful:

next_state - what state will we be transitioning to
next_states - list of states that we still need to transition to

Drop the following in a suitable place (your model using acts_as_state_machine will do) and abuse it to your hearts content:

# Monkey patch acts_as_state_machine to allow us to see what the next states are
# for the current state
module ScottBarron
 module Acts
   module StateMachine
     module InstanceMethods
         def next_state(state=nil)
           state ||= current_state()
           self.class.read_inheritable_attribute(:transition_table).each_value do |event|
             event.each do |transition|
               return transition.to if transition.from == state 
             end
           end
           nil
         end
        
         def next_states(state=nil)
           state ||= current_state()
           states = [] 
           while state = next_state(state)
             states << state            
           end            
           states          
        end       
      end    
    end  
  end
end

Thanks to Lourens Naude's' post on acts_as_state_machine for providing me with some ideas and to Scott Barron for the great acts_as_state_machine plugin.

Signing your own gems

2009-02-11T17:18:00.000-08:00

Chapter 9 (Signing Your Gems) of the RubyGems User Guide contains everything you need to know to do this.

Here's a quick overview of the steps required.

Build a certificate and private key pair

$ cd /to/where/you/wish/to/keep/your/cert_and_key
$ gem cert --build gemmaster@example.com

Two files are generated from this: gem-private_key.pem & gem-public_cert.pem.

Ensure you keep gem-private_key.pem somewhere safe that only you have access to.

Modify your build specification
Add the following spec to your existing gem's gemspec or to the relevant Rakefile ("s" is your Gem::Specification instance):

$ s.signing_key = 'gem-private_key.pem'
$ s.cert_chain  = ['gem-public_cert.pem']

Rebuild your package and you're done!

Your own gem repository

2009-02-11T17:05:00.000-08:00

Setting up your own repository is quite simple. All you need is the following:

Web server
Builder gem
A repository
Client setup

Web server
I used nginx but you should be able to tailor this to your needs without too much effort.

Create a directory to contain your local gem repository:

$ sudo mkdir -p /var/www/gems-repo/gems

Note: to get everything working correctly you need to ensure you have that /gems subdirectory off your base directory configured in nginx. You can find more info on this at the RubyGems FAQ.

Add a virtual host config in /etc/nginx/sites-available for your gems repository (I'm going to assume gems.example.com):

$ cat /etc/nginx/sites-available/gems.example.com
server {
 listen   80;
 server_name  gems.example.com;
 access_log  /var/log/nginx/gems.example.com.access.log;
 root /var/www/gems-repo;
}

Ensure you drop a symlink in /etc/nginx/sites-enabled/ for your virtual host and restart nginx.

Install the builder gem

$ sudo gem install builder

Build your repository
Jump into the base directory defined above (/var/www/gems-repo). Copy all your own gems into /var/www/gems-repo/gems/. Then generate the relevant resource files for rubygems:

$ sudo gem generate_index -d /var/www/gems-repo
Loading 1 gems from /var/www/gems-repo
.
Loaded all gems
Generating quick index gemspecs for 1 gems
.
Complete
Generating specs index
Generating latest specs index
Generating quick index
Generating latest index
Generating Marshal master index
Generating YAML master index for 1 gems (this may take a while)
.
Complete
Compressing indicies

You would re-run this every time you add to or make changes to the repository.

Client setup
Finally run the following on all local clients that need access to the repository:

$ sudo gem sources -a http://gems.example.com/

You can ensure the extra source was added by running 'gem sources'. If you now do a search for you gem that you added above you should see it listed as a target.

Stunneling (SSL Tunnels) MySQL on Ubuntu

2009-01-27T15:00:00.000-08:00

We have really restrictive IT policies at my current employer which dictates that no client/server communication can happen in the clear. This has some interesting implications as most software servers do not support encryption or encrypted tunnels in any way.

What are the options?

Write our own software that supports client/server encryption
Only use existing applications that support client/server encryption
Treat the encryption layer as an abstraction that the client/server is not aware of

The first two options have obvious drawbacks in terms of time and choice. The third seems to be the best way to transparently insert encryption into an existing architecture without requiring massive changes.

This approach can be done by:

SSH tunnels
OpenVPN tunnels
Stunnel

There may be other options but these are the main ones I have had experience with. SSH tunnels are very convenient but tend to be unstable requiring restarts every now and again.

OpenVPN has also provided very little stability requiring frequent restarts.

Lets explore the setup of the third option then, stunnel.

Stunneling
The stunnel home page describes stunnel as:

Stunnel is a program that allows you to encrypt arbitrary TCP connections inside
SSL (Secure Sockets Layer) available on both Unix and Windows. Stunnel can allow
you to secure non-SSL aware daemons and protocols (like POP, IMAP, LDAP, etc) by
having Stunnel provide the encryption, requiring no changes to the daemon's code.

Sounds promising. They have a really comprehensive examples section on their site which I highly recommend you have a look at.

In my case I needed to get a MySQL master/slave pair to replicate via a secure channel. MySQL seems to support SSL but the package requires a rebuild with the required options which would mean I need a non-standard MySQL package which makes me itchy. I also needed a similar solution for several other applications that need secure communications channels so I've decided on this solution throughout my architecture.

See this Ubuntu thread for more info on rebuilding the MySQL package with SSL support if you're so inclined.

Our installation targets are an Ubuntu servers and will therefore be flavored accordingly but you should be able to easily adapt anything mentioned here for your platform of choice.

Preamble
If you are going to be using this for the same purposes I am ensure that you have a working MySQL master/slave setup. You don't want to troubleshoot multiple applications if something is not playing nice.

Installation
A simple aptitude install stunnel4 on the master and slave should do the trick.

Now, keep in mind that if one of the nodes you have stunnel installed on is acting as both a stunnel client and stunnel server you need to have at least one stunnel server running for the client entries in a config file and one for the server entries in a config file.

Not heading this warning will result in you seeing errors like this on the client:

2009.01.28 07:02:16 LOG7[11508:3082910608]: SSL state (connect): before/connect initialization
2009.01.28 07:02:16 LOG7[11508:3082910608]: SSL state (connect): SSLv3 write client hello A
2009.01.28 07:02:16 LOG7[11508:3082910608]: SSL alert (write): fatal: handshake failure
2009.01.28 07:02:16 LOG3[11508:3082910608]: SSL_connect: 1408F10B: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
2009.01.28 07:02:16 LOG5[11508:3082910608]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket

And this on the server:

2009.01.28 07:10:06 LOG7[13426:3082861456]: SSL state (connect): before/connect initialization
2009.01.28 07:10:06 LOG7[13426:3082861456]: SSL state (connect): SSLv3 write client hello A
2009.01.28 07:10:06 LOG7[13426:3082861456]: SSL alert (write): fatal: handshake failure
2009.01.28 07:10:06 LOG3[13426:3082861456]: SSL_connect: 1408F10B: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
2009.01.28 07:10:06 LOG5[13426:3082861456]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket

My convention is to use the default config file (/etc/stunnel/stunnel.conf) that ships with the stunnel4 package for the client configuration and a modified copy of that as a config (/etc/stunnel/mysql-master.conf, in this specific case) for the server.

If you will not be using the MySQL master server as a stunnel client then you can simply ensure that your /etc/stunnel/stunnel.conf is the same as my /etc/stunnel/mysql-master.conf.

Keep in mind that multiple config files in /etc/stunnel/ will cause multiple stunnel servers to be run with each of the files found there.

Server
First off, let's generate our own key (/etc/ssl/master.pem) that we'll be using for encryption. This key must be treated as a secure file that should have the relevant access controls placed on it.

root@master:~# cd /etc/ssl/
root@master:~# openssl req -new -x509 -nodes -days 730 -out master.pem -keyout master.pem
root@master:~# chmod 600 master.pem
root@master:~# dd if=/dev/urandom of=temp_file count=2
root@master:~# openssl dhparam -rand temp_file 512 >> master.pem
root@master:~# ln -sf master.pem `openssl x509 -noout -hash < coretools01.pem`.0

Now, change the following lines in /etc/stunnel/mysql-master.conf:

cert = /etc/ssl/master.pem
;client = yes
debug = 7
output = /var/log/stunnel4/stunnel.log

[mysql-master]
accept  = 443
connect = 127.0.0.1:3306
TIMEOUTclose = 0

This config will ensure we're running as a stunnel server, listening for incoming SSL connections on 0.0.0.0:443 and shunting the unencrypted connection to 127.0.0.1:3306 (ensure your MySQl master is listening for connections here).

Fire stunnel up on the master with /etc/init.d/stunnel4 start && tail -f /var/log/stunnel4/stunnel.log.

Client
Just modify the default /etc/stunnel/stunnel.conf that was installed via the package by changing the lines below:

;cert = /etc/stunnel/mail.pem
;key = /etc/stunnel/mail.pem
debug = 7
output = /var/log/stunnel4/stunnel.log
client = yes

[mysql-master]
accept  = 127.0.0.1:3307
connect = master.example.com:443
TIMEOUTclose = 0

This sets up the MySQL slave machine as a stunnel client, forwarding all unencrypted connections on 127.0.0.1:3307 via SSL to master.example.com:443. Just replace master.example.com with the correct FQDN for your master server. You also need to ensure that your MySQL slave's config points to 127.0.0.1:3307 for its replication requirements.

Fire stunnel up on the slave with /etc/init.d/stunnel4 start && tail -f /var/log/stunnel4/stunnel.log.

Testing
An initial test can be done from the slave with the mysql command line utility in a separate terminal session by connecting to a suitable user/db configured on the master with access from 'localhost' or '127.0.0.1':

root@slave:~# mysql -u replication -h 127.0.0.1 -P 3307 -p test
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 21
Server version: 5.0.51a-3ubuntu5.4-log (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

Yay!

Now, apply your general method for determining if your master and slave is in sync. You can use the logs at /var/log/stunnel4/stunnel.log if you hit any snags. Once you're happy with everything ensure you disable the debugging in the stunnel config files.

drb.rb:852:in `initialize': getaddrinfo: nodename nor servname provided, or not known (SocketError) (aka DRb TCPServer.open(0) failure on OS X)

2008-12-15T17:07:00.000-08:00

Update (2009/01/01): Problem TT moved to redmine.
Update (2009/05/13): TT resolved.

I am currently busy building an priority queue server in ruby and I have chosen to use DRb as my communications platform.

While experimenting the simple examples from the Net (see here and here) I was consistently getting the same error from inside drb.rb (/opt/local/lib/ruby/1.8/drb/drb.rb:852):


/opt/local/lib/ruby/1.8/drb/drb.rb:852:in `initialize': getaddrinfo: nodename nor servname provided, or not known (SocketError)
     from /opt/local/lib/ruby/1.8/drb/drb.rb:852:in `open'
     from /opt/local/lib/ruby/1.8/drb/drb.rb:852:in `open_server_inaddr_any'
     from /opt/local/lib/ruby/1.8/drb/drb.rb:864:in `open_server'
     from /opt/local/lib/ruby/1.8/drb/drb.rb:759:in `open_server'
     from /opt/local/lib/ruby/1.8/drb/drb.rb:757:in `each'
     from /opt/local/lib/ruby/1.8/drb/drb.rb:757:in `open_server'
     from /opt/local/lib/ruby/1.8/drb/drb.rb:1346:in `initialize'
     from /opt/local/lib/ruby/1.8/drb/drb.rb:1634:in `new'
     from /opt/local/lib/ruby/1.8/drb/drb.rb:1634:in `start_service'
     from ./queue-provider.rb:32

What's up?
After poking drb.rb::self.open_server_inaddr_any(host, port) with a stick a few times two issues came to light:

Multiple network address families are not catered for properly in the the code.

TCPServer.open(port) where port == 0 fails under OS X but not Linux

Multiple Address Families
The code in question looks like this:


def self.open_server_inaddr_any(host, port)
infos = Socket::getaddrinfo(host, nil,
                           Socket::AF_UNSPEC,
                           Socket::SOCK_STREAM,                                  0,
                           Socket::AI_PASSIVE)
family = infos.collect { |af, *_| af }.uniq
case family
when ['AF_INET']
 return TCPServer.open('0.0.0.0', port)
when ['AF_INET6']
 return TCPServer.open('::', port)
else
 return TCPServer.open(port)
end
end

From that we can see that we only seem to expect one network address family which is a little naive. Socket::getaddrinfo() on my MacBook Pro has the following to say (where host == 'localhost'):

$ irb
irb(main):006:0> require "socket"
=> true
irb(main):007:0> host = 'localhost'
=> "localhost"
irb(main):008:0> Socket::getaddrinfo(host, nil,
irb(main):009:1* Socket::AF_UNSPEC,
irb(main):010:1* Socket::SOCK_STREAM,                                  0,
irb(main):011:1* Socket::AI_PASSIVE)
=> [["AF_INET6", 0, "localhost", "::1", 30, 1, 6], ["AF_INET6", 0, "localhost", "fe80::1%lo0", 30, 1, 6], ["AF_INET", 0, "localhost", "127.0.0.1", 2, 1, 6]]

When you take this as your input you'll see that we don't end up matching either 'AF_INET' or 'AF_INET6' and we fall through to return TCPServer.open(port) because the case block expects a match against an array with one element.

TCPServer.open(0) OS X Weirdness
I have used DRb on both Linux and Windblowns in the past without a hitch so I was rather surprised to run into something like this which is a show stopper on OS X. I though I'd see if I was having the same problems on Linux to have something to compare with:

$ irb
irb(main):001:0> require "socket"
=> true
irb(main):002:0> port = 0
=> 0
irb(main):003:0> TCPServer.open(port)
=> #

Works a treat! Let's try that on OS X:

$ irb
irb(main):001:0> require "socket"
=> true
irb(main):002:0> port = 0
=> 0
irb(main):003:0> TCPServer.open(port)
SocketError: getaddrinfo: nodename nor servname provided, or not known
     from (irb):3:in `initialize'
     from (irb):3:in `open'
     from (irb):3
     from :0

CRASH! BOOM! BANG!

DRb Quilt
The first issue is rather trivial to fix:

def self.open_server_inaddr_any(host, port)
infos = Socket::getaddrinfo(host, nil,
                            Socket::AF_UNSPEC,
                            Socket::SOCK_STREAM,
                            0,
                            Socket::AI_PASSIVE)
families = Hash[*infos.collect { |af, *_| af }.uniq.zip([]).flatten]
return TCPServer.open('0.0.0.0', port) if families.has_key?('AF_INET')
return TCPServer.open('::', port) if families.has_key?('AF_INET6')
return TCPServer.open(port)
end

The code now rather assumes we're dealing with an array of one or more network address families and tries the IPv4 and IPv6 families first and then falls though to TCPServer.open(port).

I have opened a TT on RubyForge for this that contains a patch from me to fix the first issue.

What is required to fix the second issue? Dunno just yet, I'll keep looking and see if anything interesting pops up in the TT.

Massaging Rails Models (with a happy ending)

2008-11-26T02:29:00.000-08:00

How do you alter data in a model so that the data which is stored to and gathered from the database is first filtered/transformed?

Two ways come to mind:

Insert the required behavior into your model's before_save, before_create and after_initialize callbacks.
Manually modify your attribute accessors for the attributes in question to do the magic for you.

We'll use the following contrived Model as our example:


class Gogga < ActiveRecord::Base
end

CREATE TABLE  `foo`.`goggas` (
`id` int(11) NOT NULL auto_increment,
`secret` varchar(255) default NULL,
PRIMARY KEY  (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1

A Gogga has an id and a secret field. Let's pretend we need to keep Gogga.secret encrypted (using the super secret ROT13 algorithm) in our db and we would like the fact that it is encrypted to be transparent to our rails app. We need to therefore handle en/decryption of the secret data transparently from the rest of the app within the model.

Callbacks
The before_save, before_create and after_initialize callbacks are well documented in the Callbacks API documentation.

The strategy behind using the callbacks is to simply insert the behavior we want at the relevant stage of the object's life cycle. Here's one way to accomplish this using the mentioned callbacks:

class Gogga < ActiveRecord::Base
  def before_save
    self.secret = rot13(self.secret)
  end

  def before_create
    self.secret = rot13(self.secret)
  end

  def after_initialize
    self.secret = rot13(self.secret)
  end

protected
  def rot13(corpus)
    return corpus.tr!("A-Za-z", "N-ZA-Mn-za-m")
  end
end

If everything works as advertised our secret attribute should now be encoded when you call create or save on its model and decoded when you call a new on its model. The major drawback of this strategy is that if you are manipulating a large list of Goggas you will be post/pre-processing each of those instances.

The Lazy Way
An alternative would be to override the default behavior of the model to auto-generate attribute accessors via method_missing in the mystical black guts of ActiveRecord::Base. There's some info on this in the API docs as well in the Overwriting default accessors section.

This would look something like this:

class Gogga < ActiveRecord::Base
  def secret
    return rot13(read_attribute("secret"))
  end

  def secret=(value)
    write_attribute("secret", rot13(value))
  end

protected
  def rot13(corpus)
    return corpus.tr!("A-Za-z", "N-ZA-Mn-za-m")
  end
end

This technique has the advantage that you never really mess with the internals of the model (as the view you have of it from outside is tinted by the accessor transformations) and of course work only gets done when you need to read/write the specific attribute.

Now, when you have one or two attributes that need to be protected writing out two accessors for each is not the end of the world. However, when you have several things become messy, tedious and downright boring.

Maybe we can look at some meta-magic to DRY things up a bit in another article.

Oh ExtJS TreePanel Click, wherefore art thou?

2008-08-21T13:56:00.000-07:00

ExtJS is a high quality JS UI library that really eases the cross-platform blues when it comes to designing online apps. Unfortunately I find the documentation can be a be a little obtuse from time to time.

Even the trusty Google librarian cannot answer some of the questions that pop up right off the bat and a fair amount of searching is required to get some info.

TreePanel Click
I have a TreePanel that I populate through lazy loading to save the amount of data I need to send off to the client in any one request. Unfortunately for the life of me I could not work out how to determine which node in the tree was clicked (so that I could fire off an Ajax request to populate another panel with data related to the clicked node).

Attaching Events
The first thing to do would obviously be to attach a click event listener somewhere so that we can pick up when were being prodded. If you're not familiar to JavaScript and ExtJS specifically you'd be inclined to attach an event to each of the nodes.

This is wasteful in the browser context as you should be taking advantage of event bubbling. So we simply attach a listener to the whole tree component:

  Ext.onReady(function(){
   // Create initial root node
   root = new Ext.tree.AsyncTreeNode({
     text: 'Invisible Root',
     id:'0'
   });
  
   // Create the tree
   tree = new Ext.tree.TreePanel({
     loader: new Ext.tree.TreeLoader({
       url:'/companies',
       requestMethod:'GET',
       baseParams:{format:'json'}
     }),
     renderTo:'company-tree',
     root: root,
     rootVisible:false
   });
  
   // Expand invisible root node to trigger load of the first level of actual data
   root.expand();
  
   // Listen for mouse clicks
   Ext.get('company-tree').on("click", function(){
     // Code to determine which node was clicked ...
   });
 });

Where did that click go?
Never having used the TreePanel control before I had no real inkling how I was going to reap the relevant id from the node that was clicked. I tried several approaches but nothing was bearing any fruit till I found this article on the ExtJS forums discussing something similar.

Armed with a new weapon I was able to refactor the listener:

  // Listen for mouse clicks
 Ext.get('company-tree').on("click", function(){
   node = tree.getSelectionModel().getSelectedNode();
   console.log("You clicked on", node.id);
 });

That'll do.

Installing PECL/PEAR PHP modules on a RHEL box

2008-06-17T16:08:00.000-07:00

The default RHEL 5.2 installation does not come with xdebug as part of any of the php RPMs. A quick look around the Net also provided no real RPM candidates that I could use on this system so I had to fall back to using the package management tools (pecl and pear) provided by php.

PECL's Odyssey
RHEL is a general PITA for me already so I just sigh and get on with it:

$ sudo pecl install xdebug
downloading xdebug-2.0.3.tgz ...
Starting to download xdebug-2.0.3.tgz (286,325 bytes)
...........................................................done: 286,325 bytes
66 source files, building
running: phpize
Configuring for:
PHP Api Version:         20041225
Zend Module Api No:      20060613
Zend Extension Api No:   220060519
/usr/bin/phpize: /tmp/pear/download/xdebug-2.0.3/build/shtool: /bin/sh: bad interpreter: Permission denied
Cannot find autoconf. Please check your autoconf installation and the $PHP_AUTOCONF
environment variable is set correctly and then rerun this script.

ERROR: `phpize' failed

Yep, loving it already!

Why on earth am I not able to invoke /bin/sh (as can be seen by the '/bin/sh: bad interpreter: Permission denied' error above)? Let's see if root can actually run the shell interpreter:

$ sudo /bin/sh
sh-3.2# exit

OK, everything looks good. Why is it breaking when we're trying to run the interpreter from /tmp/pear/download/xdebug-2.0.3/build/shtool?

Back to basics
Perhaps this has something to do with where we're trying to run it from and the user we're doing the installation as (root) seems to be capable of running the interpreter but not from the shtool script for some reason.

$ ls -ld /tmp/
drwxrwxrwt 17 root root 4096 Jun 18 07:41 /tmp/

Obviously _not_ a permissions issue.

$ grep tmp /etc/fstab
/dev/sda2     /tmp    ext3    defaults,nosuid,nodev,noexec    1 2

Ah, there you are! /tmp is mounted with a 'noexec' flag so that's what's causing the execution to fail when we try to install xdebug via pecl. No problem, I'll just set pecl to use /var/tmp instead ... oh, wait, on RHEL systems /var/tmp is just a symlink to /tmp.

*sigh*

Hand me half a brick
Time to work around the issue. Let's go find those directories pear expects to be somehow related to /tmp or /var/tmp:

$ pear config-show | grep tmp
PEAR Installer download        download_dir     /tmp/pear/download
PEAR Installer temp directory  temp_dir         /var/tmp

I updated these to temporarily point elsewhere:

$ pecl config-show| grep tmp
PEAR Installer download        download_dir     /root/tmp/pear/download
PEAR Installer temp directory  temp_dir         /root/tmp
$ sudo mkdir -p /root/tmp/pear/download

Second verse, same as the first
Let's give that installation another whirl and see where we are:

$ sudo pecl install xdebug
downloading xdebug-2.0.3.tgz ...
Starting to download xdebug-2.0.3.tgz (286,325 bytes)
.....................................done: 286,325 bytes
66 source files, building
running: phpize
Configuring for:
PHP Api Version:         20041225
Zend Module Api No:      20060613
Zend Extension Api No:   220060519
building in /var/tmp/pear-build-root/xdebug-2.0.3
running: /root/tmp/pear/download/xdebug-2.0.3/configure
checking for egrep... grep -E
checking for a sed that does not truncate output... /bin/sed
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details.
ERROR: `/root/tmp/pear/download/xdebug-2.0.3/configure' failed

A quick look at /root/tmp/pear/download/xdebug-2.0.3/configure shows no obvious reasons why we're failing so out comes the cluebat :

$ sudo strace pecl insstall xdebug 2>&1
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
execve("/usr/bin/pecl", ["pecl", "install", "xdebug"], [/* 16 vars */]) = 0
brk(0)                                  = 0x8c3d000

[... truncated ...]

flock(3, LOCK_UN)                       = 0
close(3)                                = 0
write(1, "ERROR: `/root/tmp/pear/download/"..., 63ERROR: `/root/tmp/pear/download/xdebug-2.0.3/configure' failed
) = 63
stat64("/var/tmp/pear-build-root/xdebug-2.0.3", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/var/tmp/pear-build-root/xdebug-2.0.3", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/var/tmp/pear-build-root/xdebug-2.0.3", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
time(NULL)                              = 1213740767
lstat64("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/var/tmp", {st_mode=S_IFLNK|0777, st_size=4, ...}) = 0
readlink("/var/tmp", "/tmp", 4096)      = 4
lstat64("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0
lstat64("/tmp/pear-build-root", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/tmp/pear-build-root/xdebug-2.0.3", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents(3, /* 4 entries */, 4096)      = 80
getdents(3, /* 0 entries */, 4096)      = 0
close(3)                                = 0
stat64("/tmp/pear-build-root/xdebug-2.0.3/config.log", {st_mode=S_IFREG|0644, st_size=4948, ...}) = 0
stat64("/tmp/pear-build-root/xdebug-2.0.3/config.nice", {st_mode=S_IFREG|0755, st_size=93, ...}) = 0
unlink("/tmp/pear-build-root/xdebug-2.0.3/config.log") = 0
unlink("/tmp/pear-build-root/xdebug-2.0.3/config.nice") = 0
rmdir("/tmp/pear-build-root/xdebug-2.0.3") = 0
stat64("/var/tmp/pear-build-root/install-xdebug-2.0.3", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/var/tmp/pear-build-root/install-xdebug-2.0.3", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/var/tmp/pear-build-root/install-xdebug-2.0.3", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
time(NULL)                              = 1213740767
lstat64("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/var/tmp", {st_mode=S_IFLNK|0777, st_size=4, ...}) = 0
readlink("/var/tmp", "/tmp", 4096)      = 4
lstat64("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0
lstat64("/tmp/pear-build-root", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/tmp/pear-build-root/install-xdebug-2.0.3", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents(3, /* 2 entries */, 4096)      = 32
getdents(3, /* 0 entries */, 4096)      = 0
close(3)                                = 0
rmdir("/tmp/pear-build-root/install-xdebug-2.0.3") = 0
stat64("/tmp/glibctestUP8dWe", 0xbfa97ca0) = -1 ENOENT (No such file or directory)
unlink("/tmp/glibctestUP8dWe")          = -1 ENOENT (No such file or directory)
umask(022)                              = 022
close(2)                                = 0
close(1)                                = 0
close(0)                                = 0
munmap(0xb7fd6000, 4096)                = 0
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
brk(0xa3ce000)                          = 0xa3ce000
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
brk(0xa18a000)                          = 0xa18a000
exit_group(0)

After all that pecl (aka pear) still tried to muck with /tmp by trying to run things in /var/tmp/pear-build-root/xdebug-2.0.3.

Fine. Be that way.

$ sudo cd /tmp && rm -fr pear pear-build-root
$ sudo ln -s /root/tmp/pear-build-root .

One more time please Sam:

$ sudo pecl install xdebug
downloading xdebug-2.0.3.tgz ...
Starting to download xdebug-2.0.3.tgz (286,325 bytes)
.....................................................done: 286,325 bytes
66 source files, building
running: phpize
Configuring for:
PHP Api Version:         20041225
Zend Module Api No:      20060613
Zend Extension Api No:   220060519
building in /var/tmp/pear-build-root/xdebug-2.0.3
running: /root/tmp/pear/download/xdebug-2.0.3/configure
checking for egrep... grep -E
checking for a sed that does not truncate output... /bin/sed
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes

[... truncated ...]

Build complete.
Don't forget to run 'make test'.

running: make INSTALL_ROOT="/var/tmp/pear-build-root/install-xdebug-2.0.3" install
Installing shared extensions:     /var/tmp/pear-build-root/install-xdebug-2.0.3/usr/lib/php/modules/
running: find "/var/tmp/pear-build-root/install-xdebug-2.0.3" -ls
36962402    4 drwxr-xr-x   3 root     root         4096 Jun 18 08:20 /var/tmp/pear-build-root/install-xdebug-2.0.3
36962465    4 drwxr-xr-x   3 root     root         4096 Jun 18 08:20 /var/tmp/pear-build-root/install-xdebug-2.0.3/usr
36962466    4 drwxr-xr-x   3 root     root         4096 Jun 18 08:20 /var/tmp/pear-build-root/install-xdebug-2.0.3/usr/lib
36962467    4 drwxr-xr-x   3 root     root         4096 Jun 18 08:20 /var/tmp/pear-build-root/install-xdebug-2.0.3/usr/lib/php
36962468    4 drwxr-xr-x   2 root     root         4096 Jun 18 08:20 /var/tmp/pear-build-root/install-xdebug-2.0.3/usr/lib/php/modules
36962464  608 -rwxr-xr-x   1 root     root       618210 Jun 18 08:20 /var/tmp/pear-build-root/install-xdebug-2.0.3/usr/lib/php/modules/xdebug.so

Build process completed successfully
Installing '/usr/lib/php/modules/xdebug.so'
install ok: channel://pecl.php.net/xdebug-2.0.3
configuration option "php_ini" is not set to php.ini location
You should add "extension=xdebug.so" to php.ini

Smell that? It's sweet success!

No more crawling, it's time to walk!
Let's finish the install off for brevity's sake:

$ sudo echo 'zend_extension="/usr/lib/php/modules/xdebug.so"' > /etc/php.d/xdebug.ini
$ php -i | grep xdebug
/etc/php.d/xdebug.ini,
xdebug
xdebug support => enabled
xdebug.auto_trace => Off => Off
xdebug.collect_includes => On => On
xdebug.collect_params => 0 => 0
xdebug.collect_return => Off => Off
xdebug.collect_vars => Off => Off
xdebug.default_enable => On => On
xdebug.dump.COOKIE => no value => no value
xdebug.dump.ENV => no value => no value
xdebug.dump.FILES => no value => no value
xdebug.dump.GET => no value => no value
xdebug.dump.POST => no value => no value
xdebug.dump.REQUEST => no value => no value
xdebug.dump.SERVER => no value => no value
xdebug.dump.SESSION => no value => no value
xdebug.dump_globals => On => On
xdebug.dump_once => On => On
xdebug.dump_undefined => Off => Off
xdebug.extended_info => On => On
xdebug.idekey => root => no value
xdebug.manual_url => http://www.php.net => http://www.php.net
xdebug.max_nesting_level => 100 => 100
xdebug.profiler_aggregate => Off => Off
xdebug.profiler_append => Off => Off
xdebug.profiler_enable => Off => Off
xdebug.profiler_enable_trigger => Off => Off
xdebug.profiler_output_dir => /tmp => /tmp
xdebug.profiler_output_name => cachegrind.out.%p => cachegrind.out.%p
xdebug.remote_autostart => Off => Off
xdebug.remote_enable => Off => Off
xdebug.remote_handler => dbgp => dbgp
xdebug.remote_host => localhost => localhost
xdebug.remote_log => no value => no value
xdebug.remote_mode => req => req
xdebug.remote_port => 9000 => 9000
xdebug.show_exception_trace => Off => Off
xdebug.show_local_vars => Off => Off
xdebug.show_mem_delta => Off => Off
xdebug.trace_format => 0 => 0
xdebug.trace_options => 0 => 0
xdebug.trace_output_dir => /tmp => /tmp
xdebug.trace_output_name => trace.%c => trace.%c
xdebug.var_display_max_children => 128 => 128
xdebug.var_display_max_data => 512 => 512
xdebug.var_display_max_depth => 3 => 3

I can now delete those temporary directories I created in /root/tmp and get back to twitching in the corner, or ...

Easy alternative - just add remount
Instead of the hoops I jumped through to get the pecl bits to stop addressing /tmp I could simply have remounted the /tmp filesystem to allow execution:

$ sudo mount -o remount,exec /tmp
$ sudo pecl install xdebug
$ sudo mount -o remount,defaults,nosuid,nodev,noexec /tmp

This however would have compromised the security that was put in place to stop the possible malicious execution of bits (especially root kits) in /tmp (or /var/tmp).

You have the power (and the knowledge now) so wield it to your benefit.

warning: SASL authentication failure: cannot connect to saslauthd server: No such file or directory

2008-05-30T13:36:00.000-07:00

There's nothing quite like being in a complete coding frenzy, communicating with your customers to get feedback on critical bugs, and your mail server going to SMTP heaven on you.

The Setup
My MTA (postfix) is set up to do secure SMTP-AUTH and TLS via Cyrus SASL library (a.k.a. saslauthd via the sasl2-bin package) on an Ubuntu box.

Postfix SASL support (RFC 4954, formerly RFC 2554) is used to authenticate remote SMTP clients to the MTA and the Postfix SMTP client to a remote SMTP server.

The Error
I originally set things up via the Postfix-SMTP-AUTH-TLS-Howto and everything was working fine until earlier today when I started seeing the following log entries when trying to send mail vi the MTA:

May 30 03:03:36 pyxidis postfix/smtpd[2840]: connect from unknown[x.x.x.x]
May 30 03:03:37 pyxidis postfix/smtpd[2840]: setting up TLS connection from unknown[x.x.x.x]
May 30 03:03:40 pyxidis postfix/smtpd[2840]: Anonymous TLS connection established from unknown[x.x.x.x]: TLSv1 with cipher AES128-SHA (128/128 bits)
May 30 03:03:40 pyxidis postfix/smtpd[2840]: warning: SASL authentication failure: cannot connect to saslauthd server: No such file or directory
May 30 03:03:40 pyxidis postfix/smtpd[2840]: warning: SASL authentication failure: Password verification failed
May 30 03:03:40 pyxidis postfix/smtpd[2840]: warning: unknown[x.x.x.x]: SASL PLAIN authentication failed: generic failure
May 30 03:03:46 pyxidis postfix/smtpd[2840]: lost connection after AUTH from unknown[x.x.x.x]
May 30 03:03:46 pyxidis postfix/smtpd[2840]: disconnect from unknown[x.x.x.x]

The Solution
I checked and the saslauthd process was happily running. Next up I had a peek in /var/spool/postfix/var/run/saslauthd/ (which I had previously created as per the HOWTO above) but there were no *mux* files to be seen as there should have been.

I then dawned on me that postfix runs in a chrooted jail and that saslauthd for some reason had stopped writing the required info to the chrooted jail where postfix was running. A quick look at the saslauthd rc script and its default file showed that it no longer had the required config to do this properly.

Why? Dunno. I'll have to go do some snooping a little later.

For now thought the fix was as simple as modifying the OPTIONS variable in the /etc/defaults/saslauthd config file to be something like this:

OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"

Restart saslauthd and things start appearing where they should and mail is back in business.

Cacti Segfaults

2008-05-30T13:31:00.000-07:00

I recently did a cacti installation for a customer and ended up skipping step #3 from the Install and Configure Cacti guide.

Step in question:

shell> mysql cacti

Why would I have skipped this very crucial step you might say?

Well, the installation was done in two sessions with enough time having elapsed between the initial and final sessions that I had gotten hazy on what was and wasn't done. I had created the database but just never took the next step.

After completing the rest of the configuration I fired Cacti but via my browser and php promptly did a segfault and lay there on the ground haemorrhaging. From my recent experience php has a propensity to do this in two specific cases:

Something went awry with a database connection or using a database resource
You're pushing the php boundaries with recursive regexps in a pre_match*() function

Once I sat down and went through each of the steps required to set the beastie up I realized I simply needed to import the db schema and initial data to get things going.

Presto!

Tales from the (PHP and Perl) Crypt - AES Encryption in MySQL

2008-05-24T18:12:00.000-07:00

I was looking for a way to share encrypted information between two systems where a table in MySQL was the integration point.

The one system is based on php while the other component is a perl daemon.

Let's get cryptic

My first stab at this was a perl based solution using the Crypt::CBC and Crypt::Blowfish libraries plus a shared secret/key. This meant I had to develop a perl script which I called from php to do the encryption which is a rather inelegant solution.

At first I could not find the right libs in php to get this done but later stumbled upon the Mcrypt suit off php and MCrypt perl functions that allow you to do encryption between the two different subsystems.

Unfortunately this means you have double the amount of hassle when it comes to updates and ensuring things Just Work™.

Move it back to the source
Some more checking brought me to the MySQL AES encryption functions that are built into MySQL. They provide the best cryptographic algorithms MySQL currently has to offer and are pretty respectable from a academic encryption perspective.

This means en/decryption is dealt with at one integration point across all languages involved which is much more elegant.

Tales from the Crypt
The MySQL AES encryption functions allow you to en/decrypt data quite easily. To encrypt a string you simply issue the following, assuming your shared secret is lesser-spotted-mountain-squid:
mysql> INSERT INTO test_table (test_column) VALUES(AES_ENCRYPT('this is a super-secret message', 'lesser-spotted-mountain-squid'));
Query OK, 1 row affected (0.09 sec)

mysql> SELECT * FROM test_table;
+----------------------------------+
| test_column |
+----------------------------------+
| Aÿ„1
ý#ôärO™é=:Žï ¼Ñ†kWA |
+----------------------------------+
1 row in set (0.00 sec)
Et voila!

One thing you need to keep in mind is that the field you want to store your encrypted data in must be a MySQL BLOB data type.

Sucking our super secret string back out into a usable form is as simple as:
mysql> SELECT AES_DECRYPT(test_column, 'lesser-spotted-mountain-squid') AS top_secret FROM test_table;
+--------------------------------+
| top_secret |
+--------------------------------+
| this is a super-secret message |
+--------------------------------+
1 row in set (0.00 sec)
The security lesson
This is rather obvious but your security is only as strong as the weakest link in the chain. In this specific case I did not want to have clear text data in the db and achieved that amicably.

Because my secret is in clear text in two different systems I am rather exposed if those systems are not as secure as they could be. Lucky for my they are pretty much locked away from daylight so I'm not too concerned.

Ubuntu 8.04 and PAM SMB Password

2008-05-06T16:39:00.000-07:00

For those of you who have taken the plunge to Ubuntu v8.04 (Hardy Heron) you may have noticed that your auth.log is being filled with the following:

May  4 03:17:01 example CRON[10796]: PAM adding faulty module: /lib/security/pam_smbpass.so
May  4 04:17:01 example CRON[10799]: PAM unable to dlopen(/lib/security/pam_smbpass.so)
May  4 04:17:01 example CRON[10799]: PAM [error: /lib/security/pam_smbpass.so: cannot open shared object file: No such file or directory]

What's up?
For some reason the Ubuntu gods have decided by default to include PAM configuration for the PAM SMB password module without actually installing the PAM SMB password module.

Hence the complaints in your logs.

Make it go away!
Sure, simply install the libpam-smbpass package or edit two config files on your system like this:

$ perl -p -i -e 's/(password\s+optional\s+pam_smbpass.so nullok use_authtok use_first_pass)/#$1/' /etc/pam.d/common-password
$ perl -p -i -e 's/(auth\s+optional\s+pam_smbpass.so migrate)/#$1/' /etc/pam.d/common-auth

You can find some more info on this boog here.

Baking CakePHP with nginx

2008-04-19T16:23:00.000-07:00

I've switched all my web server related infrastructure to nginx which is a high performance HTTP server (amongst other things) which blows apache out of the water (IMNSHO).

If you're a CakePHP user you'll know that it has some special rewriting requirements which can be found in $ROOT/.htaccess:


$ cat .htaccess

RewriteEngine on
RewriteRule    ^$ app/webroot/    [L]
RewriteRule    (.*) app/webroot/$1 [L]

Unfortunately this .htaccess file is an apache mechanism to achieve the URL rewriting that is required to get CakePHP to play nice. The apache rewrite rules do not translate directly to something nginx can use though so some work is required to get things going.

Being lazy (and believing that no problem I discover is unique to me) I had a look a round and found this article by Chris Hartjes which needed a some mods for it to work for my setup:


# CakePHP rewrite rules
location / {
  root /opt/local/html/live_site;
  index index.php;

  # Serve static page immediately
    if (-f $request_filename) {
    break;
  }

  if (!-f $request_filename) {
    rewrite ^/(.+)$ /index.php?url=$1 last;
    break;
  }
}

Here's my complete nginx.conf to give you an idea of how everything fits together:



user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
  worker_connections  1024;
}

http {
  include       etc/nginx/mime.types;
  default_type  application/octet-stream;
  server_names_hash_bucket_size  128;

  sendfile           on;
  keepalive_timeout  20;
  tcp_nodelay        on;

  server {
    listen       80;
    server_name  localhost;
    rewrite_log on;

    #error_page  404              /404.html;
    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
    root   share/nginx/html;
    }

    # Serve static content directly with some caching goodness
    location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico)$ {
      root /opt/local/html/live_site/app/webroot;
      access_log        off;
      expires           1d;
    }

    # CakePHP rewrite rules
    location / {
      root /opt/local/html/live_site;
      index index.php;

      # Serve static pages immediately
      if (-f $request_filename) {
        break;
      }
      
      # Rewrite all other URLs
      if (!-f $request_filename) {
        rewrite ^/(.+)$ /index.php?url=$1 last;
        break;
      }
    }

    # Pass PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    location ~ \.php$ {
    root /opt/local/html/live_site;
    fastcgi_pass   127.0.0.1:9000;
    fastcgi_index  index.php;
    include        /opt/local/etc/nginx/fastcgi_params;
    }
  }
}

PHP5 SimpleXML and CDATA

2008-04-18T19:41:00.000-07:00

I am in the process of porting a rather large PHP4 application to PHP5 (just in time for PHP6, yes, yes, I know) for one of my customers. Most of the application is pure imperative programming so the switch has been rather painless.

Unfortunately (for me) they have made rather judicious use of the PHP4 domxml extension libraries which no longer exist in PHP5 (where they have been replaced with the dom extension).

Moving from domxml to dom was straight forward (looping over tags, tags, attributes and text inside tags are addresses differently) so I chose to simply re-write the code to use the new dom extensions instead of opting for a translation library like the one provided by Alexandre Alapetite.

One exception to this was the use of < !--[CDATA[...]]--> blocks which SimpleXML simply seemed to discard when creating a new object.

A quick look around Google (here, here and here) and I found what needed to be done to address SimpleXML's ignorant behaviour.

The SimpleXML constructor allows you to pass in extra libxml2 parameters which allow you to get further functionality out of the library. The one I was interested was of course:


LIBXML_NOCDATA  (integer)
Merge CDATA as text nodes

So, simply changing my constructor from:

$xml = new SimpleXMLElement($text)

to:

$xml = new SimpleXMLElement($text, LIBXML_NOCDATA)

was all that was required for me to gain access to those < !--[CDATA[...]]--> structures.

Aspell, PSPELL, PHP and OS X

2008-04-01T19:46:00.000-07:00

Until quite recently there was no way to use Aspell (via the PSPELL PHP libs) on an OS X host that was using the MacPorts system for package management.

This was simply because of the lack of a pspell variant for the php{45} packages.

Port of call
The ports system has the notion of a variant for packages that are conditional modifications of port installation behaviour.

There are two types of variants: user-selected variants and platform variants.

User-selected variants are options selected by a user when a port is installed while platform variants are selected automatically by MacPorts' base according to the OS or hardware platform (Darwin, FreeBSD, Linux, i386, PPC, etc.).

More stuff and less fluff
There are several ways of working this. The first would be to log a ticket at the MacPorts trac with a request to add the variant.

Depending on the package maintainer's load you may be a response pretty quickly. I've generally gotten something back within days of logging the ticket.

In the meantime your universe cannot come to a halt waiting for someone else to add the next greatest thing as a variant to your favourite package. So here's some manual steps to get things up and running in the meantime:

Install Aspell
Recompile and install PHP with the required PSPELL support
Check php to ensure the new PSPELL libs are active
Do a simple test to see if everything is working as it should

Install Aspell
I'll assume you're using MacPorts for your your package management on your OS X host.

Grab the aspell application, libs and whichever dictionaries catch your fancy:


$ sudo port install aspell aspell-dict-en

Be sure to install at least one dictionary or your spell checking days will be rather deflated.

Recompile and install PHP with the required PSPELL support
To manually add a compilation flag you need to edit the Portfile that comes with your installed PHP version. Mine is located at:


/opt/local/var/macports/sources/rsync.macports.org/release/ports/www/php5/Portfile

Edit the Portfile and add “–with-pspell=${prefix}” to configure.args. You can then re-ininstall PHP and it should now use the modified Portfile to compile PHP.

Check php to ensure the new PSPELL libs are active
Use the following script from the command line to determine if your PHP was re-installed with the required PSPELL bits enabled:


$ php -r 'phpinfo();' | grep PSpell
PSpell Support => enabled

Do a simple test to see if everything is working as it should
You should be able to just use the example from the PHP documentation page to ensure everything is fine:


$ cat /tmp/t_pspell.php

$ php -f /tmp/t_pspell.php
Sorry, wrong spelling

The packaging gods exist!
A few days after creating this ticket on the MacPorts trac I got a response from the package maintainer informing me that the variant had been added to all the relevant PHP packages.

Cool!

If your ports distribution files are up-to-date you should now be able to do the following to see which variants are available for your PHP version of choice:


 $ port info php5
 php5 5.2.5, Revision 2, www/php5 (Variants: universal, darwin_6, darwin_7, macosx, apache, apache2, fastcgi, gmp, imap, pspell, tidy, mssql, snmp, macports_snmp, mysql3, mysql4, mysql5, oracle, postgresql, sqlite, ipc, pcntl, pear, readline, sockets)
 http://www.php.net/

 PHP is a widely-used general-purpose scripting language that is especially suited for developing web sites, but can also be used for command-line scripting.

 Library Dependencies: libxml2, libxslt, openssl, zlib, bzip2, libiconv, expat, gettext, tiff, mhash, libmcrypt, curl, pcre, jpeg, libpng, freetype
 Platforms: darwin freebsd
 Maintainers: ryandesign@macports.org jwa@macports.org

Now that pspell is listed as a variant you can install PHP with this variant by doing the following, after removing previous PHP version:


$ sudo port install php5 +pspell

Pre-queue content-filter connection overload

2008-02-22T14:48:00.000-08:00

In the last while I've been seeing the following error pop up in my logwatch report for postfix:


 *Warning: Pre-queue content-filter connection overload

At first I was concerned that the pre-queue content-filtering subsystem of postfix was somehow being overwhelmed and I was possibly loosing mail. Digging around a bit more lead to these types of log entries that seemed like they were the subject of the logwatch report:


 Feb 21 13:01:45 pyxidis postfix/smtpd[5994]: connect from unknown[unknown]
 Feb 21 13:01:45 pyxidis postfix/smtpd[5994]: lost connection after CONNECT from unknown[unknown]
 Feb 21 13:01:45 pyxidis postfix/smtpd[5994]: disconnect from unknown[unknown]

Huh?!
From what I can glean here the log entries above indicate that a SMTP connection was established with the kernel but the connecting host hot potatoed it before postfix was able to process the connection.

When postfix tries to process the connection there's nobody home because the kernel had already removed the connection and it dumps something like the lines above to the mail log.

Wherefor?
Here's the scoop from the logwatch docs:


This sometimes occurs in reaction to a portscan or broken bots, or when postfix is overloaded, due to excessive header_checks / body_checks content filtering, or even too few smtpd processes to service the demand. One could reduce the number of header_checks and body_checks, and possibly set smtpd_timeout to 60 (seconds). The key is that existing clients are overloading the number of smtpd daemons. The postfix-logwatch section configuration variable is postfix_ConnectionLostOverload, and the command line option is --connectionlostoverload. If you consider this sub-section to be meaningless, set the level limiter value to 0 and the sub-section will be suppressed.

I was not going to change my header or body checks (because they keep the unwashed spammers at bay) so I opted for tuning my smtpd_timeout down to 30 seconds in main.cf.

Because this looks like it is a possible resource issue you could also up the amount of smtpd processes allowed to service the pre-queue content-filtering subsystem.

Firefox 3 (beta x) and Firebug

2008-02-20T19:08:00.000-08:00

18 February 2008

Firefox 3 (beta x) and Firebug

My favourite extension, by far, for Firefox is Firebug. It is a combination of strace and tcpdump for web applications. It allows you to drill down into all aspects of the send-response loop between your browser and a web app.

If you're doing any JavaScript/AJAX development this tool will be invaluable to you!

Living on the edge though, as you do, I upgraded the version of Firefox I was running to v3.0b3 and too my annoyance Firebug now no longer worked.

Damn.

Lucky for me the crew at Fireclipse have enhanced Firebug 1.05 by Joe Hewitt with enhancements and bug fixes. I simply grabbed the XPI from here and installed it from within Firefox.

Et Voilà!

Another Reality Dysfunction is averted and life continues unimpeded.

Dovecot time Machine

2008-02-20T18:45:00.000-08:00

05 January 2008

Dovecot time Machine

I have a XEN virtual machine which was complaining about time. Dovecot seemed to be the most vocal with errors like this in the logs:


 dovecot: IMAP(charl): Time just moved backwards by 1 seconds. I'll sleep now until we're back in present. http://wiki.dovecot.org/TimeMovedBackwards

I run openntpd (OpenBSD NTP daemon) on the box but that was not seemingly keeping the date, well, up-to-date. Manually running ntpdate was also not providing the sync I sought and because this is a XEN box I have no access to hwclock.

Travelling back in time
A bit of digging on my provider's forums though showed that they were controlling the time syncing and had forgotten to turn the time sync back on after some troubleshooting they were doing.

That's all good and well but my box was still not syncing.

The Dovecot linked article in the log error message simply states:


 With Xen you should run ntpd only in dom0. Other domains should synchronize time automatically (see this Xen FAQ).

XEN wisdom
The XEN FAQ has the following relevant things to say about time on a XEN box:


 Q:  My xen machines doesn't accept setting its time. Basically, it's stuck to RTC time. ntpdate, date, hwclock all "seem" to work, but they don't actually change the system time. The only way I have to change it right now is to change it in the BIOS.
 A: Only affects 1.0. Fixed in newer versions.

 Q: Where does a domain get its time from?
 A: Briefly, Xen reads the RTC at start of day and by default will track that with the precision of the periodic timer crystal. Xen's estimate of the wall-clock time can only be updated by domain 0. If domain 0 runs ntpdate, ntpd, etc. then the synchronised time will automatically be pushed down to Xen every minute (and written to the RTC every 11 minutes, just as normal x86 Linux does).  All other domains always track Xen's wall-clock time: setting the date, or running ntpd, on these domains will not affect their wall-clock time. Note that the wall-clock time exported by Xen is UTC --- all domains must have appropriate timezone handling (i.e. a correct /etc/localtime file). 

 Q: Is there is some cross-domain time synchronization : are they always in perfect sync, or should I run some kind of ntp in each subdomain ? Or only domain 0 would be enough ?
 A:If you want each domain to keep its own time, there are two ways to cause a domain to run its wallclock independently from Xen:
   1. Specify 'independent_wallclock' on the command line.
   2. 'echo 1 >/proc/sys/xen/independent_wallclock'

  To reenable tracking of Xen wallclock:
   1. 'echo 0 >/proc/sys/xen/independent_wallclock'

"Shut it down, Shut it down forever!"
Following the FAQ I modified /proc/sys/xen/independent_wallclock and added it to /etc/sysctl.conf ("xen.independent_wallclock = 1") so that the change would survive a reboot.

I also changed my timezone to the same local as where I am working currently by doing the following:

$ sudo cp /usr/share/zoneinfo/Australia/Sydney /etc/localtime

Time marches on
openntpd now keeps time synced and Dovecot no longer complains about running in the future.

Ubuntu's little upstart

2007-12-27T16:01:00.000-08:00

One of my customers has a mix of Debian and Ubuntu servers installed in their network with a treasure trove of distribution versions from Debian GNU/Linux 3.0 (Woody) to Ubuntu 7.10 (Gutsy Gibbon). They have a fair amount of custom debs which need to coexist on these servers.

While installing a new Gutsy box I noticed that several of the in-house debs were failing while trying to modify /etc/inittab (our preferred way to keep things running that should never die). This was quite confusing to me as the last Ubuntu server I worked with was Ubuntu 6.06.1 LTS (Dapper Drake) which did not exhibit this weirdness.

Down the rabbit hole
Imagine my amazement when I tried to find /etc/inittab and it was completely missing! My reality reset, I checked again and it was still missing.

At first I thought some critical package (sysvinit in the versions of Ubuntu I know) was somehow missing after the base install. I jumped on the Ubuntu packages site and did a search for 'sysvinit'. Sure enough, there it was but the file list showed no inittab.

Suddenly I felt like I a four year old whose mommy had lost them at the Mall.

Further down the spiral
The file list did however list a bunch of items that referred to something called 'upstart-compat-sysv'. Browsing to this package listed the follwoing description for the package:

This package contains compatibility tasks and utilities that emulate the behaviour of the original sysvinit package, including runlevels, and ensures that the initscripts in /etc/rc*.d are still run.

OK. I faintly hear someone announcing that my mother is looking for me at the Mall's security office.

So if upstart-compat-sysv _emulates_ the original sysvinit, then what has _replaced_ it?

Upstart, show thy face
Google provided me with this link that points to the Ubuntu upstart page which went a long way towards uniting me and my mommy:

Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.

It was originally developed for the Ubuntu distribution, but is intended to be suitable for deployment in all Linux distributions as a replacement for the venerable System-V init.

Sounds even more pervasive than a simple missing iniitab!

Features:

Tasks and Services are started and stopped by events
Events are generated as tasks and services are started and stopped
Events may be received from any other process on the system
Services may be respawned if they die unexpectedly
Bi-directional communication with init daemon to discover which jobs are running, why jobs failed, etc.

To my amazement upstart has been turned on by default since Ubuntu 6.10 (Edgy Eft) which explains why I was totally in the dark (but upstart was seemingly nothing new to everyone that had been following the normal upgrade path).

So, upstart was not only superseding everything we were trying to do with inittab but also changed the way one interacts with scripts via /etc/event.d.

Why reinvent the wheel?
With the Ubuntu decision to move to the 2.6 kernel, and all the hotplug facilities it provides, they were left with several problems in Dapper. The kernel could now completely cope with hardware coming and going but that had a knock-on effect in that there was now no way to guarantee that particular devices were available at particular point in the boot process.

For example: Dapper cannot mount USB disks in /etc/fstab because it is not guaranteed that the block device exists at the point in the mount process where that happens.

Several other reasons are also provided on the Ubuntu site.

At first, the team decided to look at the available alternatives in third party projects such as Solaris SMF, Apple's launchd, the LSB initserv/chkconfig tools and initNG. None of these met the design criteria that the team had set out for themselves and in true OSS style they decided to roll their own.

The design and implementation documentation is pretty clear so I won't repeat it here.

Epilogue
In the end I simply had to add some logic to the custom debs to check if the system was running upstart and do The Right Thing(TM) (using upstart if it was available or falling back to using inittab) based on that. That amounted to dropping a relevant file in /etc/event.d/ for every server that we previously ran from inittab for systems that were using upstart.

Further Reading

ReplacementInit

Upstart Wiki

Func it up with JavaScript

2007-10-05T07:10:00.000-07:00

UPDATE: David Pollak has a great introduction to FP via JavaScript and Ruby.

Using functional programming paradigms in JavaScript are non-existent, clumsy, verbose and difficult to read in most cases. "Oliver Steel":http://osteele.com/ has built an excellent little library that does the grunt work for you when trying to get your func on.

To Func || ! Func
Most people go about their imperial programming days without a thought of functional programming techniques and how they can be applied to their daily problems. In most part this is due to the inherent difficulties with _doing_ functional programming in their tool of choice.

I urge you to do some further investigation (read Why Functional Programming Matters) into functional programming techniques even if you think you'll never use them anywhere. This is not intended to be an exercise in (academic) pointlessness but to place you outside of your comfort zone and expand your thinking across different domains. The depth of knowledge gained from this will enable you to solve problems from a larger pool of tools (sometimes allowing you to bring functional programming paradigms to bare on a problem or simply augmenting your existing tools for a more efficient or elegant solution to problems).

Learn to get your Func on!

From a pure functional programming language perspective this approach to problem solving offers the following advantages over the imperative and OOP approaches:

No (re-)assignment

No side effects

No flow of control

Functional calls can therefore have no other effect that to compute its result. In a pure functional language there are no assignments statements. Once you assign a value to a variable the variable never changes. In this sense variables in a functional language have more in common with algebraic variables that the normal programming stock we're used to.

At first this seems like a debilitating restriction but after giving your brain some time to expand you'll find that this simple restriction eliminates one of the largest sources of bugs in programming and makes the order of execution irrelevant as no side-effect can change the value of an expression and it can be evaluated at any time.

Gone are the days of worrying about orchestrating the flow control of your program. Your programs are now referentially transparent because expressions, variables and their values can be freely evaluated and replaced at any time.

Elements of Func
From a strict academic sense functional programming refers to programs that has a main body which is a function that receives it's input as its arguments and delivers the output/transformation as it's result.

So far this definition should not seem too foreign to most people that have worked with c/c++. Where this departs from the general imperative meme is that the main function is generally defined in terms of other functions, which in
turn are defined in terms of still more functions, until at the lowest level the functions are first-class citizens (language primitives).

These functions are much like ordinary mathematical functions (in that the same input will always deliver the same output).

Higher-order programming (HOP), function level programming (FLP) and partial function application (PFA) are all styles used in functional programming.

Programming Transcendence
HOP is the ability to use functions as values, in other words you can pass functions as arguments to other functions and functions can be returned as a value of other functions. An example of HOP in JavaScript would be something like the simple sort() method that you can apply to an array.

In its simplest form the sort() function takes an unordered/ordered array and sorts the array:


var a = [2,3,1,4]
document.write(a.sort())
// prints "1,2,3,4"

The sort() method however allows you to use a comparison function as an optional argument, allowing you to pass it a function as a parameter, ergo implementing HOP. Let's assume we've got an array of date objects that we want to sort in a chronological order:


array_of_dates.sort{ function (x, y) { return x.date - y.date; } }

Here we pass in an anonymous function as our comparison function to sort(). The anonymous function is called for each object in the array of dates and it must return a negative value when x < x ="="> y.

This technique is best used when you have at least two functions that perform the same take with a slight variance. Here you would then combine the functions by replacing the part(s) that are different with a function call to a separate function which is passed in to the more general function as a function parameter.

The Functional library implements string lambdas that allow you to express some of the functional programming tools more succinctly. The traditional JavaScript way of doing say a map or filter would be something like this:


map(function(x){return x+1}, [1,2,3])         // returns [2,3,4]
filter(function(x){return x>2}, [1,2,3,4]]    // returns [3,4]
some(function(w){return w.length < 3}, 'are there any short words?'.split(' '))  // returns false

Instead, string lambdas allow you to write this in the following way:


map('x+1', [1,2,3])
select('x>2', [1,2,3,4])
some('_.length < 3', 'are there any short words?'.split(' '))

Here are some other way to bend a program to your functional will using simply map, reduce and filter:


// Double the items in a list:
map('*2', [1,2,3])              // [2, 4, 6]

// Find just the odd numbers:
filter('%2', [1,2,3,4])         // [1, 3]

// Find just the evens:
filter(not('%2'), [1,2,3,4])    // [2, 4]

// Find the length of the longest word:
reduce(Math.max, 0, map('_.length', 'how long is the longest word?'.split(' ')))  // 7

// Parse a binary array:
reduce('2*x+y', 0, [1,0,1,0])   // 10

// Parse a (non-negative) decimal string:
reduce('x*10+y', 0, map('.charCodeAt(0)-48', '123'.split(/(?=.)/)))               // 123

Much more succinct, clear to read and easier to understand.

Func Levels
Value-level programming manipulates values, transforming a sequence of inputs into an output. Function-level programming manipulates functions, applying operations to functions to construct a new function. This new function transforms the inputs into outputs.

How can we make JavaScript dance to a functional-level programming paradigm using the Functional library as meter? Here's some example's:


// Find the reciprocal only of values that test true:
map(guard('1/'), [1,2,null,4])              // [1, 0.5, null, 0.25]

// Apply '10+' only to even values, leaving the odd ones alone:
map(guard('10+', not('%2')), [1,2,3,4])     // [1, 12, 3, 14]

// Write a version of map that only applies to the evens:
var even = not('%2');
var mapEvens = map.prefilterAt(0, guard.rcurry(even));
mapEvens('10+', [1,2,3,4])

// Find the first power of two that's greater than 100:
until('>100', '2*')(1)                      // 128

// Or, the first three-digit power of two (these are equivalent):
until('String(_).length>2', '2*')(1)
until(compose('>2', pluck('length'), String), '2*')(1)
until(sequence(String, pluck('length'), '>2'), '2*')(1)

Hot/Medium/Mild Curry?
Partial function application (aka currying) transforms a function that takes n arguments into a function that takes only one argument and returns a curried function of n - 1 arguments.

In English please! OK, let's try:

Currying is the process of partially, or incrementally, supplying arguments to a function. Curried functions are delayed functions expecting the remainder of the arguments to be supplied. Once all the arguments are supplied, the function evaluates normally. So, curried functions lead to lazy execution of the complete function.

From the definitions above it is clear that partial function application, or specialisation, creates a new function out of an old one. To illustrate how we apply this with Functional we'll implement between(x, y, z) which determines whether y is bounded by x and z. We then curry the first and last arguments to produce a function that tests whether a number is positive:


// Function that needs to be curried
function increasing(a, b, c)
{
return a < b && b < c;
}

// Define the set of positive numbers via lazy evaluation
var positive = increasing.partial(0, _, Infinity);

// Determine if each of the values -1, 0 and 1 fall in our range
map(positive, [-1, 0, 1])    // [false, false, true]

// Define the set of negative numbers via lazy evaluation
var negative = increasing.partial(-Infinity, _, 0);

// Determine if each of -1, 0 and 1 fall in our range
map(negative, [-1, 0, 1])    // [true, false, false]

Currying leads to lazy evaluation which allows you to work with structures like the infinite sets we created above. Cool eh?!

Epilogue
Functional does a great job at making your life easier if you want to experiment with functional programming in JavaScript without getting yourself tangled up in the verbose, standard syntax. The creator does however offer a word of warning with regards to performance if you use this lib in production. Functional is also confirmed to work in Firefox 2.0, Safari 3.0, and MSIE 6.0.

Links

2007-08-14T13:13:00.000-07:00

Dynamic Attribute-based Finder Extensions
szeryf has written a nice little exposé on how you can extend ActiveRecord::Base to add _or_ and _not_ operators to the dynamic finders that rails provides you with.

You can write the following types of finders out of the box with rails:


User.find_by_login_and_status(some_login, 1)
User.find_by_login_and_status_and_role_(some_login, 1, role)

The additional extensions as described in the article adds the following to the repertoire above:


User.find_by_login_and_status_or_role(some_login, 1, role)
User.find_by_login_and_status_not_role_(some_login, 1)

The two statements above would then result in the following SQL, respectively:


login = ? and (status = ? or (role = ?))
login = ? and (status = ? not (role = ?))

rparsec (the union of ActiveRecord :select and :include)
As you most probably already know you use :select in a find to modify the fields that are in your result set and :include to specify that related tables are loaded via joins to provide improved performance. Unfortunately you cannot use either of these two together due to the limitations imposed by the ActiveRecord implementation.

:select meets :include (or a pitch for rparsec) is an interesting article by Charlie Savage which suggests using a SQL SELECT parser to provide the required functionality.

He goes on to suggest doing this with the rparsec parser combinator framework.

The Controller Formula
Nick Kallen provides a lucid look at how one can produce poetic controller code.

Creating Multiple Models in One Action
This article is a followup on the previous one elaborating on the method that can be used to create multiple models from one action.

It covers the simplistic case where one model is simply created based on the creation of the other (when crating a group model, the creating uses needs to be the first member of the group) and the more complex case where there is a dependancy relationship between two models that needs to be enforced (a cyclops creation cannot succeed if the creation of the eye does not succeed).

Prototype, IE and Edge Rails Failures

2007-08-12T23:20:00.000-07:00

I recently wrote a little conceptual file upload application with scaffold_resource that used the iframe remoting pattern with some baked-in AJAX goodness to minimise the amount of data returned from the server as well as making the UI a lot more snappy.

Everything went well and tested a-OK in Firefox. Unfortunately my default development platform does not support IE so I only tested the app in IE a little later. To my horror IE rendered the page differently as well as spewing the following JS error:


Line: 1629
Char: 9
Error: Invalid target element for this operation.
Code: 0
URL: <REMOVED>

Prototype goodness wherefor art thou?
A few concise questions to the Oracle of Google and I found a thread started by Rob Sanheim which detailed the same problem I was seeing.

A fix, that worked for many of the thread readers, was proposed by Andy (12 December 2005 @ 1pm) in the same thread to deal with the way in which prototype inserts content in a tbody or tr tag.

So, off I went and upgraded my app to the latest Edge Rails using:


rake rails:freeze:edge

Edge Rails will you be the end of me?

Starting my app up I got a similar nasty to this:


/Applications/Locomotive2/Bundles/standardRailsFeb2007.locobundle/i386/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require': no such file to load -- active_resource (MissingSourceFile) from /Applications/Locomotive2/Bundles/standardRailsFeb2007.locobundle/i386/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require' from /Users/joelmeyer/src/FotoDir/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:495:in `require' from /Users/joelmeyer/src/FotoDir/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:342:in `new_constants_in' from /Users/joelmeyer/src/FotoDir/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:495:in `require' from ./config/../vendor/rails/railties/lib/initializer.rb:160:in `require_frameworks' from ./config/../vendor/rails/railties/lib/initializer.rb:160:in `each' from ./config/../vendor/rails/railties/lib/initializer.rb:160:in `require_frameworks' from ./config/../vendor/rails/railties/lib/initializer.rb:88:in `process' ... 8 levels... from /Applications/Locomotive2/Bundles/standardRailsFeb2007.locobundle/i386/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:12:in `execute!' from /Applications/Locomotive2/Bundles/standardRailsFeb2007.locobundle/i386/lib/ruby/gems/1.8/gems/capistrano-1.4.1/bin/cap:11 from /Applications/Locomotive2/Bundles/standardRailsFeb2007.locobundle/i386/bin/cap:16:in `load' from /Applications/Locomotive2/Bundles/standardRailsFeb2007.locobundle/i386/bin/cap:16

Seems that when you're using Edge Rails you need to use the active resource gem in tandem or you world will turn pear-shaped. Installing the gem, with dependancies did the trick:


gem install -y activeresource --source http://gems.rubyonrails.org

IE: Dark side of the moon
So, I have now upgraded to the latest prototype lib (via the Edge Rails upgrade) and got Edge Rails to run with the active_scaffold and active resources I was using in my app.

Was everything fine then and did I get to live a more fulfilling existence? NAY!

Back to the entry by Rob Sanheim and I noticed that Chris Nolan reported that the prototype lib had been fixed but that he was still experiencing the same issue.

There is a boat and I feel that Chris and I are both in it together, brothers in misery.

While debugging the issue with trusty old JS alert() he found that he was trying to insert content at an id that belonged to a table tag which was not supported. The tags supported for this were of course tbody and/or tr.

Even though the tbody tag seems to be optional according to the W3C IE still does not allow you to insert the content based on a table id.

The simple addition of the required tbody tags cured all.

Turning responsibility inside-out via delegation

2007-08-05T14:56:00.000-07:00

5 August 2007

Turning responsibility inside-out via delegation

What is the delegation pattern? You find this where you have an object that expresses a certain behaviour externally but internally defers, or, delegates the responsibility for implementation to another object in an inversion of responsibility.

Turning responsibility inside-out
Ruby provides five ways to accomplish this: three (SimpleDelegator, DelegateClass and Delegator) encapsulated in the delegate library and the remaining two (Forwardable and SingleForwardable) via the forwardable library.

Let's use a queue data structure that delegates to an array to illustrate the various ways of accomplishing delegation.

SimpleDelegator
This is the simplest way to accomplish delegation. You simply pass an object to the constructor and all methods supported by the object will be delegated.

_This object can be changed later._


require 'delegate'

class Queue
 def initialize
  @sd = SimpleDelegator.new([]) # we delegate to an array object
 end

 def enqueue(element) 
  @sd.push(element)
 end

 def dequeue 
  @sd.shift
 end
end

q = Queue.new
q.enqueue(10)    # [10]
q.enqueue(20)    # [10, 20]
q.dequeue        # [20]

If you want to change the object you're delegating to you just use __setobj__(obj). You should just keep in mind that this does *not* cause SimpleDelegator’s methods to change which means that you should only be delegating to objects of the same type as the original delegate to avoid nastiness.

DelegateClass
If SimpleDelegator does not spin your propeller then the next step would be to look at DelegateClass. Using the top level DelegateClass method to setup delegation through class inheritance is considered more flexible and is seemingly the most common use for this library.


require 'delegate'

class Queue < DelegateClass(Array) # we delegate to an array object
  def initialize(arg=[])
    super(arg)
  end

  alias_method :enqueue, :push     # alias_method sets up the method aliasing for us
  alias_method :dequeue, :shift
end

q = Queue.new
q.enqueue(10)    # [10]
q.enqueue(20)    # [10, 20]
q.dequeue        # [20]

Delegator
The final tool from the delegator library is Delegator which provides you with full control over the delegation scheme. The contrived example below is derived from the SimpleDelegator’s implementation.


require 'delegate'

class QueueDelegator < Delegator # inherit from the Delegator class
 def initialize(obj)
  super             # pass obj to Delegator constructor
  @_sd_obj = obj    # store obj for future use
 end

 def __getobj__
  @_sd_obj          # return the object we are delegating to
 end

 def __setobj__(obj)
  @_sd_obj = obj    # change delegation object, a feature we're providing
 end
end

The conventional wisdom here however is that you should most likely be using the forwardable library instead of Delegator.

Fowardable
If you need class-level delegation this is your beast of burden.


require 'forwardable'

class Queue
 extend Forwardable

 def initialize(obj=[])
  @queue = obj    # delegate to this object
 end

 def_delegator :@queue, :push,  :enqueue
 def_delegator :@queue, :shift, :dequeue
 def_delegators :@queue, :clear, :empty?, :length, :size, :<<
end

There are a few things to take note of here. First, def_delegator is used to set up the delegation relationship between the method call, the delegated object and the method to call on the delegated object.

Second, notice the syntax (:@queue, instead of @queue or :queue) to specify the delegated object we're defining methods for. This is simply an artefact of the way that Forwardable is implemented.

SingleForwardable
Where Forwardable provides class-level delegation, SingleForwardable provides object level delegation. For this example I'll simply copy the example provided in the library documentation.


require 'forwardable'

printer = String.new
printer.extend SingleForwardable        # prepare object for delegation
printer.def_delegator "STDOUT", "puts"  # add delegation for STDOUT.puts()
printer.puts "Howdy!"

Epilogue
Using DelegateClass and Forwardable for your delegation needs will most likely cover most of the cases you may end up needing to implement the delegator pattern.

Did we default or not?

2007-08-05T08:10:00.000-07:00

You may sometimes find yourself having to distinguish whether a method attribute was supplied externally or taken from the default specified in the method definition.

Let's say, for example, that you want to warn a user when they have neglected to set an attribute but still continue with execution. Here is a snippet that would accomplish this:


irb(main):060:0> def some_method(first, second=(flag=true; '2nd'))
irb(main):061:1> p "Default value #{second} used for unspecified parameter 'second'" if flag.inspect == "true"
irb(main):062:1> end
=> nil
irb(main):063:0> some_method(1,2)
=> nil
irb(main):064:0> some_method(1,'2nd')
=> nil
irb(main):065:0> some_method(1)
"Default value 2nd used for unspecified parameter 'second'"
=> nil
irb(main):066:0>

Can you work out what is going on in the method parameter declaration?

All that's happening is that the code in the round brackets after the equals sign defines a local variable _flag_, sets its value and returns the default value we want to set it to.

Ruby rocks!

Tracking fast-paced packages on debian based systems (aka debian-volatile project)

2007-07-29T07:18:00.000-07:00

debian-volatile
If you run some ISP services (your own mail server with virus and/or spam scanning tools) you will have run into the age old problem that the scanning tools in the stable distribution do not evolve as fast as they should to keep up with their fast-paced projects.

Even continual updates of the software in your distribution are not enough to stay up to date as the release cycle of the stable distribution is out of sync with the speed at which things change in the wild.

According to the debian-volatile project page:


The main goal of volatile is allowing system administrators to update their systems
in a nice, consistent way, without getting the drawbacks of using unstable, even
without getting the drawbacks for the selected packages. So debian-volatile will
only contain changes to stable programs that are necessary to keep them functional.

volatile-sloppy
Great effort goes into ensuring that no functional changes are made to packages in debian-volatile (so that configuration file changes, etc. are not required) for painless upgrades. Unfortunately painful upgrades are not always avoidable so a volatile-sloppy section was created to contain packages that are fast-paced but also require some functional change to how it runs, is installed or configured.

Security
You should note that the debian-volatile project is not supported by the _official_ security team. This responsibility falls to the debian-volatile team who currently has at least one member that is shared with the official debian testing security team.

How do I use it?
Add the relevant repository (volatile and/or volatile-sloppy) to your /etc/apt/sources.list file:

Sarge

deb http://volatile.debian.org/debian-volatile sarge/volatile main contrib non-free
deb http://volatile.debian.org/debian-volatile sarge/volatile-sloppy main contrib non-free

Etch

deb http://volatile.debian.org/debian-volatile etch/volatile main contrib non-free
deb http://volatile.debian.org/debian-volatile etch/volatile-sloppy main contrib non-free

Save sources.list and run _atp-get update_ which should generate something like this (your listing will vary depending on the repositories you have listed in your sources file):


# apt-get update
Get:1 http://archive.ubuntu.com dapper Release.gpg [189B]
Get:2 http://us.archive.ubuntu.com dapper Release.gpg [189B]                                                 
Get:3 http://us.archive.ubuntu.com dapper-backports Release.gpg [191B]                                       
Get:4 http://archive.ubuntu.com dapper-updates Release.gpg [191B]                                            
Get:5 http://volatile.debian.org etch/volatile Release.gpg [189B]                                            
Hit http://us.archive.ubuntu.com dapper Release                                 
Hit http://archive.ubuntu.com dapper Release                                    
Get:6 http://volatile.debian.org etch/volatile Release [40.7kB]                                                     
Hit http://archive.ubuntu.com dapper-updates Release                                                                
Hit http://us.archive.ubuntu.com dapper-backports Release                                                           
Hit http://archive.ubuntu.com dapper/main Packages                                                                  
Hit http://archive.ubuntu.com dapper/restricted Packages                                                    
Hit http://us.archive.ubuntu.com dapper/universe Packages                                                   
Hit http://us.archive.ubuntu.com dapper/universe Sources                                                    
Hit http://archive.ubuntu.com dapper/main Sources                                                           
Hit http://archive.ubuntu.com dapper/restricted Sources                               
Hit http://archive.ubuntu.com dapper-updates/main Packages                            
Hit http://archive.ubuntu.com dapper-updates/restricted Packages                                             
Hit http://us.archive.ubuntu.com dapper-backports/main Packages                                              
Hit http://us.archive.ubuntu.com dapper-backports/restricted Packages                                        
Hit http://us.archive.ubuntu.com dapper-backports/universe Packages                                          
Hit http://archive.ubuntu.com dapper-updates/main Sources                                                    
Hit http://archive.ubuntu.com dapper-updates/restricted Sources                                              
Hit http://us.archive.ubuntu.com dapper-backports/multiverse Packages                                        
Hit http://us.archive.ubuntu.com dapper-backports/main Sources                         
Hit http://us.archive.ubuntu.com dapper-backports/restricted Sources                   
Hit http://us.archive.ubuntu.com dapper-backports/universe Sources                     
Hit http://us.archive.ubuntu.com dapper-backports/multiverse Sources                   
Ign http://volatile.debian.org etch/volatile Release                                   
Get:7 http://volatile.debian.org etch/volatile/main Packages [3953B]
Hit http://volatile.debian.org etch/volatile/contrib Packages                  
Hit http://volatile.debian.org etch/volatile/non-free Packages
Get:8 http://security.ubuntu.com dapper-security Release.gpg [191B]
Hit http://security.ubuntu.com dapper-security Release
Hit http://security.ubuntu.com dapper-security/main Packages
Hit http://security.ubuntu.com dapper-security/restricted Packages
Hit http://security.ubuntu.com dapper-security/main Sources
Hit http://security.ubuntu.com dapper-security/restricted Sources
Hit http://security.ubuntu.com dapper-security/universe Packages
Hit http://security.ubuntu.com dapper-security/universe Sources
Fetched 44.8kB in 5s (7554B/s)
Reading package lists... Done
W: GPG error: http://volatile.debian.org etch/volatile Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EC61E0B0BBE55AB3
W: You may want to run apt-get update to correct these problems
#

The inclusion of the debian-volatile release fails because we do not have a key to authenticate the repository. Adding the following will import their key (mentioned as EC61E0B0BBE55AB3 above) into your key ring:


# gpg --keyserver subkeys.pgp.net --recv-keys EC61E0B0BBE55AB3
gpg: directory `/root/.gnupg' created
gpg: new configuration file `/root/.gnupg/gpg.conf' created
gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/root/.gnupg/secring.gpg' created
gpg: keyring `/root/.gnupg/pubring.gpg' created
gpg: requesting key BBE55AB3 from hkp server subkeys.pgp.net
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key BBE55AB3: public key "Debian-Volatile Archive Automatic Signing Key (4.0/etch)" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
# gpg --armor --export EC61E0B0BBE55AB3 | apt-key add -
gpg: no ultimately trusted keys found
OK
#

Another spin of _apt-get update_ (and possibly _apt-get upgrade_ if their are any outdated packages) should then do the trick.

Links

2007-07-24T11:00:00.000-07:00

IRB
Dr Nic has a great little article describing his favourite additions to his .irbrc. He covers some common productivity wins such as:

TABed auto-completion
Map by method which allows you to get rid of constructs like articles.columns.map {|p| p.name} or articles.columns.map &:name and simply replace it with a plural: articles.columns.names or articles.columns.name
MethodFinder/Object.what?
pp
Auto-tabbing

GuessMethod
This little gem is the best bad idea ever! Gone are those frustrating typos that waste extra cycles finding and fixing them.

Wirble
Continuing on the irb enhancements theme do yourself a favour and have a look at Wirble. It offers you tab-completion, history, and a built-in ri command as well as colorised results and a couple other goodies.

Links

2007-07-17T13:49:00.000-07:00

Sequel
Sequel is a light-weight ORM that fills in the gaps where using ActiveRecord without rails doesn't fit your needs.

Gregory Brown has a nice little expose on it as the winner of the June 2007 Ruby Project Spotlight.

Introduction to .NET 3.0 for Architects
Keep your friends close and competition^H^H^H^H^H^H^H^H^H^H alternative platforms closer. InfoQ has a great 50000 foot look at .NET 3.0.

Profiling your ails app with ruby-prof
Charlie Savage wrote a great article on doing some profiling on your rails app using ruby-prof. He discusses using both flat and graph (with associated call tree information) profiles to nuke performance hogs.

ruby-prof is a fast code profiler for Ruby. Its features include:

Speed - it is a C extension and therefore many times faster than the standard Ruby profiler.
Flat Profiles - similar to the reports generated by the standard Ruby profiler
Graph profiles - similar to GProf, these show how long a method runs, which methods call it and which methods it calls
Threads - supports profiling multiple threads simultaneously
Recursive calls - supports profiling recursive method calls
Reports - can generate both text and cross-referenced html reports
Output - can output to standard out or to a file

AR-Delegation
This plugin extends ActiveRecord::Base to add useful delegation features. For example: has_columns :from => :source, :only => ["title", "name"] has_column "title", :from => :source, :as => "source_title".

It really improves the conciseness of your code but in so doing hides your implementation adding a layer of indirection that may make your code a little more difficult to understand if the person reading your code does not know that AR-Delegation was used.

Exception Notifier
I use this with most of my projects that reside on remote customer networks where the only way for the application to give me a heads up is if it sends me an email with an attached problem report.

The Journey from Lambda to Unlambda

2007-05-19T08:24:00.000-07:00

While scrounging around the Net looking at functional programming languages I came across Unlambda. The name by itself piqued my interest so I though I'd go have a look-see.

Here follows the troubled tale of the journey from Lambda to Unlambda ...
Nature of the Beast
It was created by David Madore and is a minimal functional programming language that was specifically built to make programming in it obtuse (or as the author's site refers to it as: "... fun and challenging"). It is based on combinatory logic but omits the lambda abstraction forcing you to rely on K and S combinators.

Beside hailing itself as a functional language it also succeeds in being an obfuscated programming language through severely restricting the set of allowed operations in the language and making it generally alien to programmers from more conventional languages.

It strictly only manipulates functions. A function is the only function parameter that you can pass in to a function and a function is the only construct that can be returned from a function. It relies heavily on its built-in k and s functions (K and S combinators) to get anything done.

The source is built to be intentionally incomprehensible for a human making it next to impossible to deduce what the intention of the program is by simply reading the source.

You are welcome to create your own functions, but be warned, you cannot name or save the custom function(s) because Unlambda does not have support for variables. Besides dispelling variables from its cloth you will also notice the lack of built-in support for data structures or code constructs (e.g. loops, conditionals, etc.). You are however welcome to build your own code constructs.

As an illustration, here is a loop that prints “Hello, world!” repeatedly, followed by an incrementing number of asterisks (an explanation of this severely limited subset of alphabet soup can be found here):


```s``sii`ki
``s``s`ks
   ``s``s`ks``s`k`s`kr
             ``s`k`si``s`k`s`k
                             `d````````````.H.e.l.l.o.,. .w.o.r.l.d.!
                      k
    k
`k``s``s`ksk`k.*

Unlambda is the little functional language that could however and in the face of all its consciously implemented obscurity it still manages to be Turing-complete.

Sure, but can it speak?
Here's the Unlambda equivalent for the trusty old "Hello world" we're used to:


`r```````````.H.e.l.l.o. .w.o.r.l.di

More examples, a tutorial and some HOWTOs can be found at David's site for Unlambda.

Secret Murmurings on Unlambda

“It's disgusting — it's revolting — we love it.” CyberTabloid

“Unlambda, the language in which every program is an IOUCC.” Encyclopædia Internetica

“The worst thing to befall us since Intercal.” Computer Languages Today

“The effect of reading an Unlambda program is like having your brains smashed out by a Lisp sexp wrapped around an ENIAC. You won't find anything like it west of Alpha Centauri.” The Hitch-Hacker's Guide to Programming

Epitaph
Unlambda is inevitably compared to Intercal, but unlike Intercal it has a kind of weird elegance; this is because Intercal gets in your way, but Unlambda simply fails to help you."

OS X File Name Subterfuge

2007-05-15T18:02:00.000-07:00

While saving a file via the well known OS X file Save As dialog I noticed something really queer. The filename I had pasted in was magically altered to adhere to OS X's ideas on which characters are kosher for file/directory names!

Silent Substitution
To see the magic simply open TextEdit, create a new document (if you are not already presented with one) and add the following text to the document:


 watch: this: space:

Now, select the text, copy it and save the file with the selected text as the file name.

Et voilà!

Subterfuge Uncorked
The dialog magically converts the colons to hyphens (-) to ensure that you don't have to go through he whole 'This filename is not allowed' error cycle needlessly. Brill.

The reason for the substitution is that OS X prohibits the use of colon characters in file/directory names because this character is used to represent a directory in the HFS+ file system.

According to the HFS+ spec you can use any Unicode or ASCII (including NUL) characters. OS APIs may limit some of these characters for legacy reasons.

I love subtlety.

Dynamic Arbitrary Depth Hashes In Ruby

2007-05-15T17:23:00.000-07:00

UPDATE: Charles Duan has an interesting article in a similar vein.

Arbitrary array and hash depth constructs cannot be created in Ruby in the way you would in Perl or PHP. The following will simply fail with an error:


irb(main):001:0> a = []
=> []
irb(main):002:0> a[1][2][3][4] = 1
NoMethodError: undefined method `[]' for nil:NilClass
      from (irb):2
irb(main):003:0> h = {}
=> {}
irb(main):004:0> h[1][2][3][4] = 5
NoMethodError: undefined method `[]' for nil:NilClass
      from (irb):4

When dynamically constructing your array or hash (aka Autovivification) this really gets in the way.

Autovivification
This is a dynamic data structure creation feature that can be found in Perl and PHP (those are the ones I know of). It allows you to create dynamic, complex, nested data structures based on the types implied in the syntax of the statement of code accessed through the data structure.

IOW, the act of fetching or storing a value at a leaf through a branch dynamically creates the branch(es) to the leaf.

VivifiedHash
One approach is to do the following:


  irb(main):011:0* VivifiedHash = Hash.new(&(p=lambda{|h,k| h[k] = Hash.new(&p)}))
  => {}
  irb(main):012:0> VivifiedHash[1][2][3][4] = 5
  => 5
  irb(main):013:0> VivifiedHash[1][2][3][4]
  => 5
  irb(main):014:0> VivifiedHash[1][2][3]
  => {4=>5}
  irb(main):015:0> VivifiedHash[1][2]
  => {3=>{4=>5}}
  irb(main):016:0> VivifiedHash[1]
  => {2=>{3=>{4=>5}}}
  irb(main):017:0> VivifiedHash
  => {1=>{2=>{3=>{4=>5}}}}

All this does is recursively assign the default key of the hash a new hash object as value. Each branch you specify in your assignment will recursively trigger the creation of a new hash.

Limitations
The limitation on this are of course that your data structure cannot contain anything but hashes as branches. Leaf nodes can be any data type though.

Sources

Ruby Hashes of Arbitrary Depth

Multidimensional arrays and hashes discussion on the RubyTalk mailing

Auto Vivification

Mechanized Scraping

2007-05-07T18:56:00.000-07:00

Ever needed to interface with a web application without any real APIs? Take one step back from looking for a traditional API and use WWW::Mechanize to bend the application to your will.

WWW:Mechanize (inspired by "Andy Lester's":mailto://andy@petdance.com perl Mechanize module and written by Aaron Patterson) allows you to moonlight as a web User Agent (browser) from the comfort of your ruby scripting environment. It is great for building automated tests of your web applications, creating your favourite mashups and also to treat another web application's UI as the API to the application.

I've been working on some code that needs to gather reporting information from our billing system but I have no real access to the Oracle db in the back to get to the require stored procedures. So, I decided to simply use the UI as my API to the data and dusted my trusty old WWW:Mechanize (which uses Hpricot internally to parse and tokenise pages) off for the challenge.

It provides you with all the required tools to log in to a site (as well as automatic cookie handling), click on URI, submit forms and oh so much more. The only real feature currently lacking is support for JavaScript (they do however provide you with ideas on how to manoeuvre around some of the more mundane corers) which is becoming more and more painful in this Web2.0 world of ours.

WWW:Mechanize is quite easy to use so I am not going to write an exposé on the in's and out's of the lib or share with you its secrets that helped me to sate world hunger and bring peace to all. Instead, I will mention some of the bits that tripped me up while trying to make the web application dance to my flute.

Button Value Attributes
I was getting nowhere while trying to submit a form in the web application with some crafted values. Tinker here, tinker there and still no go. Try a browser and the application itself and things work like swiss ~~cheese~~watches.

Right you mangy ASP application, its time for the big guns! Out comes Wireshark and the debugging starts in earnest. First I dump a session from my script and then one from a browser.

From the diff of of the POST request I notice that the browser has the value attribute for the 'Save' button in the form set whereas I didn't. Because the form was posting back to itself I assume they had some code like (pseudocode):


if $submit == 'Submit'
then
    do your stuff when the form has been submitted
else
    display the normal form
end

Adding something that resembles the following did the trick:


form.buttons.name('some_convoluted_button_name').value = 'Submit'

Out of Buffer Error
A few more form hoops later and I started getting an error like:


hpricot/parse.rb:44:in `scan': ran out of buffer space on element <group>, starting on line 361. (Hpricot::ParseError?)

Hey?!

A quick look on the bug db for WWW:Mechanize on RubyForge listed this closed bug that has some application to our situation. The error messages are not the same (I assume this is the case due to an earlier version of Hpricot that was used when this was reported).

According to this TT it is a Hrpicot issue and refers to this TT.

According to the problem description:


An 'OUT OF BUFFER SPACE' error shuts down my whole app when I try to parse through an aspx page with an abnormally (or normally?) large viewstate stuffed into an input. Here's what it looks like:

    <input type="hidden" name="__VIEWSTATE"
        value="dDw3NzQ0ODQ2ODQ ... 11954 characters in total ... DsXdJfP+k" />

If I remove the large value it works fine. Is there a way hpricot could not exit when trying to parse a page like this?

DING! DING DING!

I am also scraping an ASP application and lo and behold I too have a ginormous __VIEWSTATE input tag in the page in question. I knew ASP was evil, but this?!

The limit on the buffer was of course a protection mechanism to ensure that a parsed page does not cause your computer to become the black hole of memory. The workaround for this is quite simple though, just increase the buffer


Okay, kids. [98] now has a buffer_size method.
    Hpricot.buffer_size = 262144
    doc = Hpricot(open("http://asp.net/big-viewstate-vomit.html"))

Perhaps I will find the wherewithal to fix the parser to read these massive attributes, but on-the-other-hand I don't want to encourage this disastrous behavior by ASP.NET!! You know?

"That's all good and well but we're not really using Hpricot directly, we're using WWW:Mechanize!", you all shout in unison.

True, true. All you do is simply add the buffer_size declaration after instantiating your shiny new WWW:Mechanize object like so:


agent = WWW::Mechanize.new
Hpricot.buffer_size = 204800

The default buffer size is defined in hpricot_scan.rl as:


[...]

#define BUFSIZE 16384

[...]

buffer_size = BUFSIZE;
if (rb_ivar_defined(self, rb_intern("@buffer_size")) == Qtrue) {
  bufsize = rb_ivar_get(self, rb_intern("@buffer_size"));
  if (!NIL_P(bufsize)) {
    buffer_size = NUM2INT(bufsize);
  }
}
buf = ALLOC_N(char, buffer_size);

[...]

That's a buffer of about 16KB for an attribute which under normal circumstances would be more than ample space for an attribute but working with ASP seems to be anything but normal.

In Closing
I have not had as much fun in quite some time. WWW:Mechanize had me clapping my little hands in glee while shouting "Wheeeeeeeeeee!" like a little kid that was given his first bunny rabbit just after having his second double espresso for the hour.

Ruby (Hpricot) Program Guide - III

2007-05-02T22:25:00.000-07:00

As discussed in the previous article our next steps will be to refactor the constructor and provide an example of how we can use objects from the DSTVSchedule class to collect and display channels of our choice.

Let's change the constructor to take the channel ID, time offset (to account for different time zones) and the period ahead in time for which we want to gather schedule information as parameters. This will mean that we get rid of the custom hash class and tidy things up a little bit:


def initialize(channel=219, offset=2, period=30)
  start_date, end_date = get_search_dates(period)
  url = build_url(build_query_string(channel, start_date ,end_date))

  p "Start: #{start_date}  End: #{end_date}  URL: #{url}"

  @hp = Hpricot(open(url))
  @ic = Iconv.new('US-ASCII//TRANSLIT', 'UTF-8')
  @coder = HTMLEntities.new
  @schedule = process_html(@hp, offset)
end

def get_search_dates(period=30)
  [DateTime.now().strftime("%d %b %Y"), (DateTime.now()+period).strftime("%d %b %Y")]
end

def build_query_string(channel, start_date, end_date)
  urlencode({
    'channelid' => channel,
    'startDate' => start_date,
    'EndDate'   => end_date}) +
      '&sType=5&searchstring=&submit=Submit'
end

def build_url(query_string)
  host = 'www.mnet.co.za'
  cgi = '/schedules/default.asp?'
  "http://#{host}#{cgi}#{query_string}"
end

def urlencode(hash)
  hash.map {|k, v| "#{URI::encode(k.to_s)}=#{URI::encode(v.to_s)}"}.join('&')
end

We no longer statically define the query parameters in the constructor and therefore have no real need for the custom hash. We can still use the urlencode() method though and add it as a helper in the class.

The start and end dates for the query are calculated based on today's date and the period provided to the constructor as an argument.

We also dumped all that horrible looking query string and url variable construction code into separate methods.

The next step is to provide some automation to the channel schedule collection code for our example program. Look at the the HTML data in any of the search pages and you'll see the following (excerpt):


<select  name="channelid" class="ScheduleInputSelect">
    <option value="" >CHANNEL</option>
    <option value=246>actionX                                           </option>
    <option value=322>Activate                                          </option>
    <option value=496>Africa Magic</option>
    <option value=487>Africa Magic Channel (C-Band)                     </option>
    <option value=639>Africa Magic W4</option>
    <option value=417>Animal Planet                                     </option>
[...]
    <option value=254>TV Globo                                          </option>
    <option value=493>TV5 Afrique                                       </option>
    <option value=110>TV5 Afrique (Africa)                              </option>
    <option value=65>VH1                                               </option>
    <option value=67>ZEE TV                                            </option>
</select>

These are the channels that we can search for. What we need is to represent this information as an internal data structure that we can use to search for the channels we want. I suggest a hash that has the channel name as a key and the channel ID and offset as a tuple.

I am lazy so I'd prefer to avoid typing all that information up or manually trying to transform it in the editor. Perhaps we can use some good old command line ruby to chew up and spit out the code we need which we can then just cut 'n paste or import (depending on the editor you use).

Copy the HTML and drop it in a file somewhere. Let's call the file in.html and run it through this command line script (output is truncated):


$ ruby -n -e '$_=~/value=(\d+)\>(.+)\s+\</;if $1&&$2 then a=$1;b=$2;print "\# \"#{b.sub(/\s+$/,"")}\" => [#{a}, 120],\n" end' < in.html | head   
# "actionX" => [246, 120],
# "Activate" => [322, 120],
# "Africa Magic Channel (C-Band)" => [487, 120],
# "Animal Planet" => [417, 120],
# "B4U Movies" => [227, 120],
# "BBC Food" => [284, 120],
# "BBC Prime" => [121, 120],
# "BBC World" => [5, 120],
# "Bloomberg Information TV" => [8, 120],
# "Boomerang" => [314, 120],
[...]

Now take the output and place it in your script as a hash (as described above):


channels = {
  # "actionX" => [246, 120],
  # "Activate" => [322, 120],
  # "Africa Magic Channel (C-Band)" => [487, 120],
  # "Animal Planet" => [417, 120],
  # "B4U Movies" => [227, 120],
  "BBC Food" => [284, 120],
  "BBC Prime" => [121, 120],
  # "BBC World" => [5, 120],
  # "Bloomberg Information TV" => [8, 120],
  # "Boomerang" => [314, 120],
  # "BVN" => [270, 120],
  # "Canal+ Horizons" => [237, 120],
  # "Cartoon Network" => [13, 120],
  # "Cartoon Network (Africa)" => [219, 120],
  # "Cartoon Network (W4)" => [182, 120],
  # "Channel O - Sound Television" => [27, 120],
  # "China Central Television 4" => [15, 120],
  # "China Central Television 9 (Africa)" => [226, 120],
  # "CNBC" => [90, 120],
  # "CNBC (Africa)" => [194, 120],
  # "CNBC (W4)" => [187, 120],
  # "CNN International" => [18, 120],
  # "Deukom - 3SAT" => [165, 120],
  # "Deukom - ARD" => [93, 120],
  # "Deukom - DW" => [94, 120],
  # "Deukom - PRO 7" => [164, 120],
  # "Deukom - RTL" => [91, 120],
  # "Deukom - SAT 1" => [92, 120],
  # "Deukom - ZDF" => [95, 120],
  "Discovery Channel" => [21, 120],
  # "E-Entertainment" => [646, 120],
  "ESPN" => [24, 120],
  # "eTV" => [111, 120],
  # "Fashion TV" => [145, 120],
  # "Fashion TV (Africa)" => [196, 120],
  # "Fashion TV (W4)" => [216, 120],
  "GO" => [542, 120],
  # "Go  (K-World Teen)" => [341, 120],
  "Hallmark Entertainment Network" => [32, 120],
  "History Channel" => [484, 120],
  # "History Channel (Africa)" => [485, 120],
  # "K-TV World" => [36, 120],
  # "KTV (Indian Bouquet)" => [501, 120],
  # "kykNET" => [112, 120],
  # "M-Net Domestic" => [39, 120],
  "M-Net East (Africa)" => [40, 120],
  "M-Net Series" => [75, 120],
  # "MK89" => [592, 120],
  # "Movie Magic (Africa)" => [57, 120],
  "Movie Magic 2 (Africa)" => [234, 120],
  # "Movie Magic 2 (W4)" => [233, 120],
  # "MTV" => [42, 120],
  # "MTV Base" => [69, 120],
  "National Geographic" => [102, 120],
  # "NDTV" => [499, 120],
  # "Parliamentary Service" => [45, 120],
  # "Pay Per View" => [109, 120],
  "Reality TV" => [248, 120],
  # "Rhema Network" => [46, 120],
  # "RTPi" => [48, 120],
  # "SABC 1" => [84, 120],
  # "SABC 2" => [85, 120],
  # "SABC 3" => [86, 120],
  # "SABC Africa" => [87, 120],
  # "SIC" => [255, 120],
  # "Sky News" => [120, 120],
  "Sony Entertainment" => [228, 90],
  # "Summit" => [104, 120],
  # "Sun TV" => [500, 120],
  # "SuperSport" => [52, 120],
  # "SuperSport 2" => [54, 120],
  # "SuperSport 3" => [80, 120],
  # "SuperSport 3 (W4)" => [172, 120],
  # "SuperSport 5" => [208, 120],
  # "SuperSport 5 (Africa)" => [252, 120],
  # "SuperSport 5 (W4)" => [251, 120],
  # "SuperSport 6" => [209, 120],
  # "SuperSport 7 (C-Band)" => [580, 120],
  # "SuperSport Zone Mosaic" => [235, 120],
  # "TellyTrack" => [34, 120],
  # "Travel Channel" => [61, 120],
  # "Trinity Broadcasting Network" => [276, 120],
  # "Turner Classic Movies" => [59, 120],
  # "Turner Classic Movies (Africa)" => [60, 120],
  # "Turner Classic Movies (W4)" => [181, 120],
  # "TV Globo" => [254, 120],
  # "TV5 Afrique" => [493, 120],
  # "TV5 Afrique (Africa)" => [110, 120],
  # "VH1" => [65, 120],
  # "ZEE TV" => [67, 120]
}

You'll notice I have removed the comments from any of the channels I want (I recommend you do the same for the channels you may be interested in). I also added a default time offset of 2 hours (120 minutes) for most of the channels to adjust the time for my time zone. You can change this in the command line ruby filter above to suit your needs.

All we need to do now is wrap our object creation and the output from it in a loop and we're off:


channels.keys.each do |channel|
  p "Channel: #{channel}"
  schedule = DSTVSchedule.new(channels[channel][0], channels[channel][1], 30)
  schedule.print_schedule
  print "\n\n"
end

All done. Here is the complete script source listing:


#!/usr/bin/ruby

class DSTVSchedule
  require 'rubygems'
  require 'hpricot'
  require 'open-uri'
  require 'htmlentities'
  require 'iconv'
  require 'collections/sequenced_hash'

  def initialize(channel=219, offset=2, period=30)
    start_date, end_date = get_search_dates(period)
    url = build_url(build_query_string(channel, start_date ,end_date))

    p "Start: #{start_date}  End: #{end_date}  URL: #{url}"

    @hp = Hpricot(open(url))
    @ic = Iconv.new('US-ASCII//TRANSLIT', 'UTF-8')
    @coder = HTMLEntities.new
    @schedule = process_html(@hp, offset)
  end

  def process_html(hp, offset)
    schedule = SequencedHash.new
    date = ""
    time = ""
    (hp/"td").each do |line|
      case line.inner_html
      when /ScheduleChannel/
        @channel = sanitize((line/"[@class='ScheduleChannel']").inner_html)
      when /(ScheduleDate|date)/
        date = utf7((line/"[@class='ScheduleDate']|[@class=date]").inner_html)
        schedule[date] = SequencedHash.new
      when /ScheduleTime/
        time = sanitize((line/"[@class='ScheduleTime']").inner_html)
        time = (Time.parse("#{date} #{time}") + (60 * offset)).strftime("%H:%M")
        schedule[date][time] = []
      when /ScheduleTitle/
        schedule[date][time] << sanitize((line/"[@class='ScheduleTitle']").inner_html)
      when /\<p\>/
        schedule[date][time] << sanitize((line/"p").inner_html)
      end
    end

    schedule
  end

  def to_s
    self.print_schedule("\t")
  end

  alias :to_tdt :to_s

  def to_csv
    ##TODO - Add channel to the output
    self.print_schedule(",")
  end

  def print_schedule(separator="||")
    sep = separator
    @schedule.keys.each do |date|
      @schedule[date].keys.each do |time|
        print [date, time, @schedule[date][time][0], @schedule[date][time][1]].join(sep) + "\n"
      end
    end
  end

  protected

  def sanitize(string)
    string.gsub!(/\<\!\-\-.+$/, '')  # remove HTML comments to the end of the line
    string.gsub!(/^\s+/, '')  # remove leading whitespace
    string.gsub!(/\s+$/, '')  # remove trailing whitespace
    string
  end

  def utf7(string="")
    @ic.iconv(@coder.decode(string))
  end

  def get_search_dates(period=30)
    [DateTime.now().strftime("%d %b %Y"), (DateTime.now()+period).strftime("%d %b %Y")]
  end

  def build_query_string(channel, start_date, end_date)
    urlencode({
      'channelid' => channel,
      'startDate' => start_date,
      'EndDate'   => end_date}) +
        '&sType=5&searchstring=&submit=Submit'
  end

  def build_url(query_string)
    host = 'www.mnet.co.za'
    cgi = '/schedules/default.asp?'
    "http://#{host}#{cgi}#{query_string}"
  end

  def urlencode(hash)
    hash.map {|k, v| "#{URI::encode(k.to_s)}=#{URI::encode(v.to_s)}"}.join('&')
  end
end


#
# Main
#
channels = {
  # "actionX" => [246, 120],
  # "Activate" => [322, 120],
  # "Africa Magic Channel (C-Band)" => [487, 120],
  # "Animal Planet" => [417, 120],
  # "B4U Movies" => [227, 120],
  "BBC Food" => [284, 120],
  "BBC Prime" => [121, 120],
  # "BBC World" => [5, 120],
  # "Bloomberg Information TV" => [8, 120],
  # "Boomerang" => [314, 120],
  # "BVN" => [270, 120],
  # "Canal+ Horizons" => [237, 120],
  # "Cartoon Network" => [13, 120],
  # "Cartoon Network (Africa)" => [219, 120],
  # "Cartoon Network (W4)" => [182, 120],
  # "Channel O - Sound Television" => [27, 120],
  # "China Central Television 4" => [15, 120],
  # "China Central Television 9 (Africa)" => [226, 120],
  # "CNBC" => [90, 120],
  # "CNBC (Africa)" => [194, 120],
  # "CNBC (W4)" => [187, 120],
  # "CNN International" => [18, 120],
  # "Deukom - 3SAT" => [165, 120],
  # "Deukom - ARD" => [93, 120],
  # "Deukom - DW" => [94, 120],
  # "Deukom - PRO 7" => [164, 120],
  # "Deukom - RTL" => [91, 120],
  # "Deukom - SAT 1" => [92, 120],
  # "Deukom - ZDF" => [95, 120],
  "Discovery Channel" => [21, 120],
  # "E-Entertainment" => [646, 120],
  "ESPN" => [24, 120],
  # "eTV" => [111, 120],
  # "Fashion TV" => [145, 120],
  # "Fashion TV (Africa)" => [196, 120],
  # "Fashion TV (W4)" => [216, 120],
  "GO" => [542, 120],
  # "Go  (K-World Teen)" => [341, 120],
  "Hallmark Entertainment Network" => [32, 120],
  "History Channel" => [484, 120],
  # "History Channel (Africa)" => [485, 120],
  # "K-TV World" => [36, 120],
  # "KTV (Indian Bouquet)" => [501, 120],
  # "kykNET" => [112, 120],
  # "M-Net Domestic" => [39, 120],
  "M-Net East (Africa)" => [40, 120],
  "M-Net Series" => [75, 120],
  # "MK89" => [592, 120],
  # "Movie Magic (Africa)" => [57, 120],
  "Movie Magic 2 (Africa)" => [234, 120],
  # "Movie Magic 2 (W4)" => [233, 120],
  # "MTV" => [42, 120],
  # "MTV Base" => [69, 120],
  "National Geographic" => [102, 120],
  # "NDTV" => [499, 120],
  # "Parliamentary Service" => [45, 120],
  # "Pay Per View" => [109, 120],
  "Reality TV" => [248, 120],
  # "Rhema Network" => [46, 120],
  # "RTPi" => [48, 120],
  # "SABC 1" => [84, 120],
  # "SABC 2" => [85, 120],
  # "SABC 3" => [86, 120],
  # "SABC Africa" => [87, 120],
  # "SIC" => [255, 120],
  # "Sky News" => [120, 120],
  "Sony Entertainment" => [228, 90],
  # "Summit" => [104, 120],
  # "Sun TV" => [500, 120],
  # "SuperSport" => [52, 120],
  # "SuperSport 2" => [54, 120],
  # "SuperSport 3" => [80, 120],
  # "SuperSport 3 (W4)" => [172, 120],
  # "SuperSport 5" => [208, 120],
  # "SuperSport 5 (Africa)" => [252, 120],
  # "SuperSport 5 (W4)" => [251, 120],
  # "SuperSport 6" => [209, 120],
  # "SuperSport 7 (C-Band)" => [580, 120],
  # "SuperSport Zone Mosaic" => [235, 120],
  # "TellyTrack" => [34, 120],
  # "Travel Channel" => [61, 120],
  # "Trinity Broadcasting Network" => [276, 120],
  # "Turner Classic Movies" => [59, 120],
  # "Turner Classic Movies (Africa)" => [60, 120],
  # "Turner Classic Movies (W4)" => [181, 120],
  # "TV Globo" => [254, 120],
  # "TV5 Afrique" => [493, 120],
  # "TV5 Afrique (Africa)" => [110, 120],
  # "VH1" => [65, 120],
  # "ZEE TV" => [67, 120]
}

channels.keys.each do |channel|
  p "Channel: #{channel}"
  schedule = DSTVSchedule.new(channels[channel][0], channels[channel][1], 30)
  schedule.print_schedule
  print "\n\n"
end

I hope these articles have tickled your lobes and gets you to go explore Hpricot and the Wonderful World of Web Scraping.

Ruby (Hpricot) Program Guide - II

2007-04-30T23:38:00.000-07:00

For this installment we'll see if we can build on what we learnt last time to provide a less naive solution to get a complete schedule for a channel that spans several days, each having variable amounts of programs per day.

First thing first though. Let's add the code that will retrieve the page for the channel we choose. Let's assume we want the schedule for Cartoon Network (Africa). The channel id for this channels happens to be 219 (as per the select list on the search page).


class Hash
  require 'uri'

  def urlencode
    map {|k, v| "#{URI::encode(k.to_s)}=#{URI::encode(v.to_s)}"}.join('&')
  end
end

class DSTVSchedule
  require 'rubygems'
  require 'hpricot'
  require 'open-uri'
  require 'htmlentities'
  require 'iconv'

  def initialize()
    query_params = {
      'startDate' => '30 Apr 2007',
      'EndDate'   => '01 May 2007',
      'channelid' => 219
    }
    query_string = query_params.urlencode + '&sType=5&searchstring=&submit=Submit'
    host = 'www.mnet.co.za'
    cgi = '/schedules/default.asp?'
    url = "http://#{host}#{cgi}#{query_string}"
    @hp = Hpricot(open(url))
    @ic = Iconv.new('US-ASCII//TRANSLIT', 'UTF-8')
    @coder = HTMLEntities.new
    @channel = channel
    @date = date
    @time = time
    @title = title
    @synopsis = synopsis

    printf "Channel: %s\nDate: %s\nTime: %s\nTitle: %s\nSynopsis: %s\n",
        @channel, @date, @time, @title, @synopsis
  end

  def channel
    sanitize(@hp.at("font[@class='ScheduleChannel']").inner_html)
  end

  def date
    sanitize(@hp.at("font[@class='ScheduleDate']").inner_html)
  end

  def time
    sanitize(@hp.at("font[@class='ScheduleTime']").inner_html)
  end

  def title
    sanitize(@hp.at("font[@class='ScheduleTitle']").inner_html)
  end

  def synopsis
    sanitize((@hp/"td[@colspan=5]/p").first.inner_html)
  end

  def sanitize(string)
    @ic.iconv(@coder.decode(string))
  end
end


#
# Main
#
schedule = DSTVSchedule.new()

So what interesting changes are there from our last try? The first thing you'll notice is that I monkey patched the Hash class and added a nifty urlencode method to encode my URL parameters that are used to construct the query string which we will be sending off to the search application.

Inside the DSTVSchedule class we've added query_params to temporarily hold our variable URL parameters. We then construct the URL we'll use for the query and simply pass that to the open() method from open-uri.

The rest should all seem familiar to you (if you followed the previous article).

Now that we have that behind us do you notice we sit with a little dilemma? If we want multiple days' programs we cannot use the class as it stands because we will religiously only output the first program in the schedule. Let's replace all those methods (channel, time, date, title, synopsis) with one method that initialises an internal data structure which will represent the channel information.


def initialize()
  query_params = {
    'startDate' => '30 Apr 2007',
    'EndDate'   => '01 May 2007',
    'channelid' => 219
  }
  query_string = query_params.urlencode + '&sType=5&searchstring=&submit=Submit'
  host = 'www.mnet.co.za'
  cgi = '/schedules/default.asp?'
  url = "http://#{host}#{cgi}#{query_string}"
  @hp = Hpricot(open(url))
  @ic = Iconv.new('US-ASCII//TRANSLIT', 'UTF-8')
  @coder = HTMLEntities.new
  @schedule = process_html(@hp)

  self.print_schedule
end

def process_html(hp)
  schedule = SequencedHash.new
  date = ""
  time = ""
  (hp/"td").each do |line|
    case line.inner_html
    when /ScheduleChannel/
      @channel = sanitize((line/"[@class='ScheduleChannel']").inner_html)
    when /(date|ScheduleDate)/
      date = utf7((line/"[@class=date]|[@class='ScheduleDate']").inner_html)
      schedule[date] = SequencedHash.new
    when /ScheduleTime/
      time = sanitize((line/"[@class='ScheduleTime']").inner_html)
      schedule[date][time] = []
    when /ScheduleTitle/
      schedule[date][time] << sanitize((line/"[@class='ScheduleTitle']").inner_html)
    when /\<p\>/
      schedule[date][time] << sanitize((line/"p").inner_html)
    end
  end

  schedule
end

The process_html method replaces all the methods we removed. All we've done is use Hpricot to search for all table column tags, and their content, and done some further search refinement in the case statement.

In the case structure I use simple regexps to find the classes I want and then use Hpricot to pull out the information contained in the matched tag. The structure I create is a hash of hashes that has the date and time as keys and the title and synopsis as 2 elements in an array (tuple).

There is one strange case above; when searching for dates. The reason for this is to cope with the inconsistent semantics used in the HTML (as mentioned in the previous article). The first date is listed with a class attribute of 'ScheduleDate' while all the rest have a class attribute of 'date'.

Take note of the use of the specialised hash SequencedHash that is used instead of the vanilla hash that is included in the core of ruby. The SequencedHash is part of the Ruby Collections gem which keeps track in which order we add elements so that we're able to pull them out in the same order.

I suspect storing the order of the keys may be a lot faster than trying to sort through a (potentially) large data set at the end to ensure the data is printed out in ascending date/time order.

The sanitize() method has changed in the following ways from the last article:

Forcing of encoding to UTF7 has been moved to the utf7() method.

Drop any text that is a HTML comment to the end of the string.

Reap any leading and trailing white space.

They are protected so we can only use them in our class.


protected

def sanitize(string)
  string.gsub!(/\<\!\-\-.+$/, '')  # remove HTML comments to the end of the line
  string.gsub!(/^\s+/, '')  # remove leading whitespace
  string.gsub!(/\s+$/, '')  # remove trailing whitespace
  string
end

def utf7(string="")
  @ic.iconv(@coder.decode(string))
end

We can now construct a valid query, execute the search and build an internal data structure that represents our schedule. We now need to find some way to output what we have internally.


def to_s
  self.print_schedule("\t")
end

alias :to_tdt :to_s

def to_csv
  ##TODO - Add channel to the output
  self.print_schedule(",")
end

def print_schedule(separator="||")
  sep = separator
  @schedule.keys.each do |date|
    @schedule[date].keys.each do |time|
      print [date, time, @schedule[date][time][0], @schedule[date][time][1]].join(sep) + "\n"
    end
  end
end

print_schedule() forms the basis of my output strategy. It takes an optional separator character(s) and walks the internal data structure to construct a schedule entry with data concatenated by the separator.

I reuse this method in the to_s() and to_csv() methods to print out TAB delimited and comma separated values, respectively. I also added a to_tdt (TAD Delimited Text) alias which is essentially just another name for to_s().

Running the class as it stands should give you something like this (extract):


30 April 2007||00:20||King Arthur's Disasters||Following the crazy adventures of King Arthur as he tries to find a present for his true love, Princess Guinevere.
30 April 2007||00:45||Spaced Out||'Death Of An Alien!'. George feels guilty when a Russian astronaut who saved his life is evicted from the space station.
30 April 2007||01:10||The Cramp Twins||Follow the fun and adventures of the troublesome twins, Lucien and Wayne Cramp, who are always fighting, arguing and embarrassing each other!
[...]
1 May 2007||00:20||King Arthur's Disasters||'The Ice Palace'. King Arthur and Merlin are sent to Switzerland to find Guinevere an ice palace that she can live inside.
1 May 2007||00:45||Spaced Out||'Invasion'. When cockroaches invade the space station, the Martins are asked by a cockroach prince to solve a conflict between his people and another clan.
1 May 2007||01:10||The Cramp Twins||Follow the fun and adventures of the troublesome twins, Lucien and Wayne Cramp, who are always fighting, arguing and embarrassing each other!
[...]

Feel free to play with the other output options for more fun.

Here is the complete class as it stands now:


class Hash
  require 'uri'

  def urlencode
    map {|k, v| "#{URI::encode(k.to_s)}=#{URI::encode(v.to_s)}"}.join('&')
  end
end

class DSTVSchedule
  require 'rubygems'
  require 'hpricot'
  require 'open-uri'
  require 'htmlentities'
  require 'iconv'
  require 'collections/sequenced_hash'

  def initialize(channel='', period=30, time_offset=2)
    query_params = {
      'startDate' => '30 Apr 2007',
      'EndDate'   => '1 May 2007',
      'channelid' => "219"
    }
    query_string = query_params.urlencode + '&sType=5&searchstring=&submit=Submit'
    host = 'www.mnet.co.za'
    cgi = '/schedules/default.asp?'
    url = "http://#{host}#{cgi}#{query_string}"
    @hp = Hpricot(open(url))
    @ic = Iconv.new('US-ASCII//TRANSLIT', 'UTF-8')
    @coder = HTMLEntities.new
    @schedule = process_html(@hp)
  end

  def process_html(hp)
    schedule = SequencedHash.new
    date = ""
    time = ""
    (hp/"td").each do |line|
      case line.inner_html
      when /ScheduleChannel/
        @channel = sanitize((line/"[@class='ScheduleChannel']").inner_html)
      when /(ScheduleDate|date)/
        date = utf7((line/"[@class='ScheduleDate']|[@class=date]").inner_html)
        schedule[date] = SequencedHash.new
      when /ScheduleTime/
        time = sanitize((line/"[@class='ScheduleTime']").inner_html)
        schedule[date][time] = []
      when /ScheduleTitle/
        schedule[date][time] << sanitize((line/"[@class='ScheduleTitle']").inner_html)
      when /\<p\>/
        schedule[date][time] << sanitize((line/"p").inner_html)
      end
    end

    schedule
  end

  def to_s
    self.print_schedule("\t")
  end

  alias :to_tdt :to_s

  def to_csv
    ##TODO - Add channel to the output
    self.print_schedule(",")
  end

  def print_schedule(separator="||")
    sep = separator
    @schedule.keys.each do |date|
      @schedule[date].keys.each do |time|
        print [date, time, @schedule[date][time][0], @schedule[date][time][1]].join(sep) + "\n"
      end
    end
  end

  protected

  def sanitize(string)
    string.gsub!(/\<\!\-\-.+$/, '')  # remove HTML comments to the end of the line
    string.gsub!(/^\s+/, '')  # remove leading whitespace
    string.gsub!(/\s+$/, '')  # remove trailing whitespace
    string
  end

  def utf7(string="")
    @ic.iconv(@coder.decode(string))
  end
end


#
# Main
#
schedule = DSTVSchedule.new()
schedule.print_schedule

Further refactoring may see us adding some attributes to the constructor (channel name, time offset) and providing an example on how we can use objects from this class to collect and display multiple channels of our choice.

Sounds like there's another article in there somewhere.

Unholy Triumvirate: TextMate, MacPorts and Ruby

2007-04-27T16:22:00.000-07:00

After switching back from a Ubuntu laptop to my MacBook Pro I was once again getting back to using TextMate to do some development and systems scripting. The combination of ruby and RubyGems have been a little bit rocky on OS X.

In part it was due to the default install of ruby on OS X, me using Fink for package management and then later switching from that to MacPorts.

Apple (and I presumably) suck cvyrf.

The problem I ran into was that after installing ruby and rb-rubygem via the ports system, TextMate no longer seems too interested in compiling ruby scripts when I hit CMD-R and provides me with a lovely:

"No such file to load ” rubygems

Checking Google the first listing I get is this.

It did not provide me with an applicable solution but got me thinking ... Either I have some environment variables that are not being set (or set incorrectly) or my library paths are screwy somehow.

An easy way to confirm the former is to check if your shell environment also suffers from the same malady:

$ ruby -r rubygems -e "p 1"
1

Not the problem then. Next step, let's pull out find and off a hunting we go:

$ sudo find / -name ruby -type f
Password:
/opt/local/bin/ruby /opt/local/var/db/dports/software/ruby/1.8.6_0/opt/local/bin/ruby /usr/bin/ruby

Let's see if there is some disparity between the ruby binary in /opt/local/bin and /usr/bin:

$ /usr/bin/ruby -v
ruby 1.8.2 (2004-12-25) [universal-darwin8.0]
$ /opt/local/bin/ruby -v
ruby 1.8.6 (2007-03-13 patchlevel 0) [i686-darwin8.9.1]

Well, what do you know. The version in /usr/bin is older and also looks for its libs in a non /opt location which means that it won't pick up the good work port has done for me. I moved /usr/bin/ruby to /tmp and added a soft link for /opt/local/bin/ruby to /usr/bin.

Running my script in TextMate now works like a charm!

Puffing with SSHKeychain

2007-04-26T13:20:00.000-07:00

In one of my previous articles I showed how you could use ssh-agent to your advantage to maximize lackadaisicalness. I have since then moved from the Ubuntu laptop that I was using at the time to my Mac that became available again.

I was looking for a nice and neat way to integrate ssh-agent into the Mac environment but could not get my shell scripting approach to gel elegantly. While doing the obligatory search on the web I found and fell in love with SSHKeychain.

This little app does all the had work (running ssh-agent from the correct place and exporting your keys into memory with ssh-add) for you, and more ... It not only handles the ssh-agent side of things but also provide support for integrating with the Apple Keychain and forward local ports over a ssh connection to set up ssh tunnels.

Go see the full feature list for more info.

Installation
Here are the step from their site:

Download SSHKeychain.dmg and mount it.
Copy SSHKeychain (SSHKeychain.app) to your Applications folder.
Run SSHKeychain. This should open a dock item and a statusbar item.
Click either the Statusbar Item, the Dock Item, or Main Menu and open the Preferences.
Open the Environment tab.
Enable "Manage global environment variables". This will make SSHKeychain available for other applications.
Open the keys tab and see if any of your keys are missing (~/.ssh/id_dsa and ~/.ssh/identity are default).
Re-login to make the global variables work.
Start up SSHKeychain, and you're set.

I added SSHKeychain to my Login Items in the System Preferences panel to ensure the app was running after a restart or log out/in sequence.

Setup
If you followed the installation instructions above there should be nothing further to do (assuming you had some pre-created keys in the default place like I had).

Excellent!

When I now fire Terminal.app up and log into a box that has my public key on it no password is required and I am logged in without further ado.

ssh-agent for Developers

2007-04-20T17:29:00.000-07:00

Have you ever wanted to automate the ssh pass phrase login procedure when connecting to remote systems that have your public key in their .ssh/authorized_keys?

This is done using ssh-agent (and ssh-add) which will be on your Debian or Ubuntu system if you have the openssh-client dep installed. For other flavours of Linux, OS X or UNIX please refer to your package management documentation (or install from source) to see how you can install the required software.

On an Ubuntu system ssh-agent is started for you by default. Please refer to your system documentation for ssh-agent to find the correct way to run it on your system.

The following approach should work for situations where the client (the computer with the private key) is either a server (access is generally restricted to it via remote shell) or a desktop (includes laptops) with a graphical terminal program.

Add the following to the end of your ~/.bashrc (or other suitable shell setup configuration file):

# Run ssh-add if it has not been run already.
if  ssh-add -l | grep -q 'The agent has no identities.'
then
eval "ssh-add"
fi

Save the addition to your .bashrc (or suitable alternative) and log out and back in.

You will be presented with a request for your pass phrase you chose when creating your public/private keys. Enter it and sigh with relief as your default key(s) are cached in memory.

When you now try and log into the remote system again there will be no passwords or pass phrases required for this session.

Ruby (Hpricot) Program Guide - I

2007-04-18T17:50:00.000-07:00

Do you live outside of South Africa and subscribe to the M-Net Africa service? Ever wanted to avoid the M-Net Africa site and just get the program guide for your region?

Well, look no further. Ruby and Hpricot to the rescue!

The M-Net Schedule site has changed quite often over the last few months so chances are good that by the time you get to this article their site may have devolved again. Doing screen scraping on web sites is generally fraught with pain, suffering and disappointment.

This is generally due to the fact that you're providing a static way to read dynamic (over time) content. Don't get discouraged though, just build notification of changes into your screen scraper and ensure that it can notify you when things have changed so that you can up date it.

To compound the problem, many sites (including the M-Net Schedule site) do not conform to the XHTML standard. This simply means that they have not used semantic tools to layout their site to abstract the structure, content and behaviour from their site. A quick validation via the W3C Markup Validation Service confirms that the parser can't even determine the content encoding.

Embrace change - it is a lot less painful (not to mention more productive ;).

Analysis of Structure
The first step of parsing content from an outside source is to analyse the structure of the content to determine what strategies you are going to employ to read and parse the content. Below is an extract of the type of content we're interested in:

<tr>
<td colspan="5">
  <font class="ScheduleSchedule">Today's Schedule for :</font>
  <font class="ScheduleChannel">Cartoon Network (Africa)</font>
</td>
</tr>
<tr>
<td colspan="5">&nbsp;</td>
</tr>
<tr>
<td colspan="5">
  <font class="ScheduleDate">17&nbsp;April&nbsp;2007</font>
</td>
</tr>
<tr bgcolor="F5F5F5">
<td colspan="5">&nbsp;</td>
</tr>
<tr bgcolor="F5F5F5">
<td width="40">
  <b><font class="ScheduleTime"> 06:25</font></b>
</td>
<!--Time-->
<td width="420">
  <font class="ScheduleTitle">Codename: Kids Next Door
  <!--Title-->
  </font>
</td>
<td width="17"></td>
<td width="188"></td>
<!--SMS Reminder-->
<td width="50" align="right">
  <a href="#" onclick="OpenAgeRestriction(1);return false;">Family</a>
</td>
<!--Age Restriction-->
</tr>
<tr bgcolor="F5F5F5">
<td colspan="5">
  <p>A gang of 10 year olds takes on top secret missions, using fantastic home-made technology to safeguard their treehouse against attack and grown-ups.</p>
</td>
</tr>
<tr>
<td colspan="5">&nbsp;</td>
</tr>
<tr bgcolor="F5F5F5">
<td width="40">
  <b><font class="ScheduleTime"> 06:50</font></b>
</td>
<!--Time-->
<td width="420">
  <font class="ScheduleTitle">The Powerpuff Girls
  <!--Title-->
  </font>
</td>
<td width="17"></td>
<td width="188"></td>
<!--SMS Reminder-->
<td width="50" align="right">
  <a href="#" onclick="OpenAgeRestriction(1);return false;">Family</a>
</td>
<!--Age Restriction-->
</tr>
<tr bgcolor="F5F5F5">
<td colspan="5">
  <p>The wild and wacky escapades of three girls with extraordinary powers. Blossom, Buttercup and Bubbles use their superpowers to fight crime and villainy in Townsville.</p>
</td>
</tr>
<tr>
<td colspan="5">&nbsp;</td>
</tr>

The first table row we're interested in is the one that tells us which channels we are looking at and what this day's date is (lightly formatted for readability):

<tr>
<td colspan="5">
  <font class="ScheduleSchedule">Today\'s Schedule for :</font>
  <font class="ScheduleChannel">Cartoon Network (Africa)</font>
</td>
</tr>
<tr>
<td colspan="5">&nbsp;</td>
</tr>
<tr>
<td colspan="5">
  <font class="ScheduleDate">17&nbsp;April&nbsp;2007</font>
</td>
</tr>

The name of the channel resides in a font tag whit a class attribute of "ScheduleChannel" and the date we're working with also resides in a font tag with a class attribute of "ScheduleDate". How does the search for this information translate into code?I will be using a XPath query (Hpricot supports both XPath and CSS selector based queries) to find the first font tag that has a class attribute that I am searching for:

def channel
  @channel = @hp.at("font[@class='ScheduleChannel']").inner_html
end

def date
  @date = @hp.at("font[@class='ScheduleDate']").inner_html
end

That's all pretty plain Jane so far. Here is what a typical table row looks like that contains the time of the program (reformatted for readability):

<tr bgcolor="F5F5F5">
<td width="40">
  <b><font class="ScheduleTime"> 06:25</font></b>
</td>
<!--Time-->
<td width="420">
  <font class="ScheduleTitle">Codename: Kids Next Door
  <!--Title-->
  </font>
</td>
<td width="17"></td>
<td width="188"></td>
<!--SMS Reminder-->
<td width="50" align="right">
  <a href="#" onclick="OpenAgeRestriction(1);return false;">Family</a>
</td>
<!--Age Restriction-->
<tr>
<tr bgcolor="F5F5F5">
<td colspan="5">
  <p>A gang of 10 year olds takes on top secret missions, using fantastic home-made technology to safeguard their treehouse against attack and grown-ups.</p>
</td>
</tr>

The time is similarly found in a font tag with a class tag of "ScheduleTime" and the program is found in a font tag with a class attribute of "ScheduleTitle". The program synopsis is however wrapped in a table column with a span of 5 and a paragraph tag.

def time
@time = @hp.at("font[@class='ScheduleTime']").inner_html
end

def title
@title = @hp.at("font[@class='ScheduleTitle']").inner_html
end

def synopsis
@synopsis = (@hp/"td[@colspan=5]/p").first.inner_html
end

You will notice that the extraction of the time and title holds no surprises. The synopsis extraction however is something new. I chose to use a CSS selector search for the synopsis by looking for the first td tag that has a colspan=5 attribute, followed by a p tag's contents (inner_html).

If you were to print the values of the relevant variables you would see that there is still some cleaning up that needs to be done on them before they can be considered for programmatic consumption:

Channel: Cartoon Network (Africa)
Date: 17&nbsp;April&nbsp;2007
Time:  06:25
Title: Codename: Kids Next Door              <!--Title-->


Synopsis: A gang of 10 year olds takes on top secret missions, using fantastic home-made technology to safeguard their treehouse against attack and grown-ups.

The channel and time looks fine so we'll just skip them for now. The date has some HTML entities in it so let's remove them using the handy HTMLEntities (I recommend installing from the gem) lib. The problem is that if they sneaked in some HTML entities in the date they may choose to do this elsewhere as well so let's not trust the input and ensure we sanitise all input in a generic way:

def initialize(url)
@coder = HTMLEntities.new
p sanitize(url)
end

def sanitize(string)
@coder.decode(string)
end

The only problem with this is that that HTMLEntities uses UTF-8 encoding which outputs (on my system) something like this for the date value:

"17\\302\\240April\\302\\2402007"

Not really ideal ... let's use the iconv lib to get the UTF-8 string forced into a US-ASCII encoding:

def initialize(url)
@coder = HTMLEntities.new
@ic = Iconv.new('US-ASCII//TRANSLIT', 'UTF-8')
p sanitize(url)
end

def sanitize(string)
@ic.iconv(@coder.decode(string))
end

Right, now to get back on track after that slight detour. To recap, we now have a strategy to get all the items we're interested in but the solution above is a little naive because it assumes we only have one day with one program. The complete schedule for a channel could span several days, each having variable amounts of programs per day.

One can also extend the ideas above to make it a lot more usable by downloading multiple channels for you and possibly pretty print it, send it to yourself via email or drop it in a db for later processing or display.

I'll cover these in followup articles to come ...

(Almost) Overriding helpers in rails

2007-03-30T12:36:00.000-07:00

In a project I am currently working on I use the select_datetime() helper to provide me with year + month + day + hour + minute drop down boxes. After using the UI for a while the customer mentioned that they do not require the full minute listing (from 00 to 59 minutes) for the minute drop down box and would prefer to only have 15 minute increments to shorten the drop down and waste less scrolling time when choosing the minutes.

Umm ... OK!

This type of functionality can be provided in a few ways (that I
know of ):

Wrapper: Create your own helper wrapper
Replace: Replace the original helper method in the module where it comes from
RTFM: Read the docs first and then code

select_datetime() is actually a composition of two other helpers:

select_time() is another composition of:

Because we're simply interested in changing the behaviour of the minutes we will only be changing select_minute().

Wrapper
This approach is pretty straight forward. All you need to do is add your own helper as a wrapper for the helper you wish to masquerade. Unfortunately we cannot use this technique here because the helper we're using (select_datetime()) cannot provide us with the relevant functionality we want by simply massaging the helper's input or output.

Bummer.

Replace
Your intuition may be itching at this point and that annoying inner voice you never ignore could be saying:

'Why don't you just simply override the select_datetime() in application_helper.rb or ActionView::Base class, lilly-livered limp-wristed sissy!'.

In this case your inner voice needs to be muzzled. Helpers are included as mixins. This means the methods in the mixed in module become available to the classes using them as if they were part of the actual class.

The PickAxe book says it succinctly:

'In fact, mixed-in modules effectively behave as superclasses.'

Unfortunately this also means that if you change the methods in a module they will change for all classes that use the mixed-in module. If you want to change the helper behaviour you need to change the behaviour in the module that is providing the helper as a mixin, in this case ActionView::Helpers::DateHelper.

Off we go and open ActionView::Helpers::DateHelper to start mucking with select_minute() so that we can support periods as the customer requested. Here is the relevant code:

# File vendor/rails/actionpack/lib/action_view/helpers/date_helper.rb, line 190

190: def select_minute(datetime, options = {})
191: val = datetime ? (datetime.kind_of?(Fixnum) ? datetime : datetime.min) : ''
192: if options[:use_hidden]
193: hidden_html(options[:field_name] || 'minute', val, options)
194: else
195: minute_options = []
196: 0.step(59, options[:minute_step] || 1) do |minute|
197: minute_options << ((val == minute) ? 198: %(#{leading_zero_on_single_digits(minute)}\n) : 199: %(#{leading_zero_on_single_digits(minute)}\n) 200: ) 201: end 202: select_html(options[:field_name] || 'minute', minute_options, options) 203: end 204: end

Alrighty! Let's add that stepping code to provide the periodic minutes ...

RTFM
Damn! Had I spent a little time drilling down through the documentation I would have noticed that the select_minute() helper already provides the required functionality via its :minute_step option.

Here is what the documentation says about this helper:

select_minute(datetime, options = {})

Returns a select tag with options for each of the minutes 0 through 59 with the
current minute selected. Also can return a select tag with options by minute_step
from 0 through 59 with the 00 minute selected The minute can also be substituted
for a minute number. Override the field name using the :field_name option, 'minute'
by default.

So, all of this may seem a little pointless but the lesson here is check the docs before you take-off at light speed. I look forward to writing an actual article on overriding helpers in rails soon.