Monday, October 12, 2009

B0rked ports on Snow Leopard

If you are using MacPort to manage OSS on your box and you recently upgraded to Snow Leopard you will find error messages like this when trying to use the ports system:
$ port selfupdate
dlopen(/Library/Tcl/macports1.0/MacPorts.dylib, 10): no suitable image
found. Did find:
/Library/Tcl/macports1.0/MacPorts.dylib: mach-o, but wrong architecture
while executing
"load /Library/Tcl/macports1.0/MacPorts.dylib"
("package ifneeded macports 1.0" script)
invoked from within
"package require macports"
(file "/opt/local/bin/port" line 39)
The issue is simply that you have all your libs and binaries managed via MacPorts compiled for the i386 architecture and not x86-64 (as required for Snow Leopard).

The solution is simple, but, arduous. You need to install the new version of MacPorts for Snow Leopard, make a backup list of the installed ports, delete them and reinstall the ones you still want.

You may want to pay close attention to the variants of the ports that you had previously installed when reinstalling them.

Also, you need the latest version of Xcode (a version greater than v3.0 will do) installed.


See the following two URLs for more info:
Happy recompiling!

Tuesday, September 22, 2009

OS X Snow Leopard and broken scrolling in MacVim

My upgrade to Snow Leopard has been pretty smooth sailing bar one annoying hitch. After the upgrade MacVim stopped scrolling properly.

When you scroll down only the last few lines on the screen update and scrolling up only the first few lines. This requires you CTRL-L to redraw the window every time you've finished scrolling.


Looks like this is caused by MacVim's support for ATSUI which has been deprecated in favour of Core Text in Snow Leopard.

If you are experiencing this you can simply turn the ATSUI renderer off by unchecking MacVim -> Preferences -> Advanced -> Use ATSUI renderer.

This is switched off by default so most people won't be affected by this. At the time of this post there was no real indication from the MacVim project if they'd be switching to Core Text in the future in addition to using the ATSUI renderer.

Sunday, September 13, 2009

iTerm, and the case of the missing DISPLAY

If you're running as your X server on OS X you generally need to use the xterm window it launches for you as part of the system or your specific xinitrc to connect to remote hosts via SSH when you want to enable X11 forwarding (and the propagation of your DISPLAY variable).

I prefer iTerm and luckily the solutions to switch to using it instead of the xterm launched by X11 is pretty easy to implement.

The first option is to make a user specific copy of xinitrc and modify the line that launches xterm to rather launch iTerm. If you don't have a user specific xinitrc (~/.xinitrc) you can grab a copy from /etc/X11/xinit/xinitrc or /usr/X11R6/lib/X11/xinit/xinitrc.

Near the bottom of the xinitrc you'll see a few lines that reference xterm that you can comment our and substitute with:

/usr/bin/open /Applications/

If you access remote X applications very infrequently it may simply be easier to ensure that X11 is running and then export your DISPLAY variable manually in iTerm:

$ export DISPLAY=:0

The final solution is to make all terms aware of your DISPLAY variable by editing your local environment plist (~/.MacOSX/environment.plist):

$ open ~/.MacOSX/environment.plist

This will open the plist in the Property List Editor where you can add a new child (DISPLAY) under the root node with a value of ':0'. For this to take effect you need to log out/in. You could also have dropped 'export DISPLAY=:0' into your local ~/.bash_profile to achieve the same results.

Untrusted SSH clients or Gdk-WARNING **: Connection to display localhost:10.0 appears to be untrusted

While connecting to a local RH system via 'ssh -X' I was seeing errors like:

Connection to display localhost:10.0 appears to be untrusted. Pointer and
keyboard grabs and inter-client communication may not work as expected.

OpenSSH implements the X11 SECURITY extension that provides two clients types: trusted & untrusted.

Trusted clients can do anything with the display while untrusted clients cannot inject synthetic events (mouse movement, keypresses, etc.) or read data from other windows (e.g. take screenshots).

In theory you should be able to run most applications via the untrusted category but it seems that a fair amount of client apps still do not implement this correctly so YMMV.

If you want to initialize a SSH connection with X11 forwarding support and using the trusted security type you simply replace -X with -Y:

$ ssh -Y USER@HOST

TIP: Add a '-C' to your ssh connections to speed things up. Nothings for free but I am sure the CPU/speed trade-off will suit most users.

Friday, September 11, 2009

Installing ghc on OS X and getting "./configure: line 11082: syntax error: unexpected end of file"

While trying to install ghc from MacPorts I kept getting this:
---> Fetching ghc
---> Attempting to fetch ghc-6.10.3-src.tar.bz2 from
---> Attempting to fetch ghc-6.10.3-src-extralibs.tar.bz2 from
---> Attempting to fetch testsuite-6.10.3.tar.bz2 from
---> Attempting to fetch ghc-6.8.2-darwin-i386-leopard-bootstrap.tar.bz2 from
---> Verifying checksum(s) for ghc
---> Extracting ghc
---> Applying patches to ghc
---> Configuring ghc
Error: Target org.macports.configure returned: configure failure: shell command
" cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.
macports.org_release_ports_lang_ghc/work/ghc-6.10.3" && ./configure 
--with-gmp-includes=/opt/local/include --with-gmp-libraries=/opt/local/lib 
work/ghc-bootstrap/bin/ghc' --with-gcc=/usr/bin/gcc-4.0 " returned error 2
Command output: checking build system type... i386-apple-darwin9.7.0
checking host system type... i386-apple-darwin9.7.0
checking target system type... i386-apple-darwin9.7.0
Canonicalised to: i386-apple-darwin
checking version of ghc... 6.8.2
checking for nhc... no
checking for nhc98... no
checking for hbc... no
checking for ld... /usr/bin/ld
./configure: line 11082: syntax error: unexpected end of file

Turns out there's some old cruft in the ghc autoconf macros that were causing the break. The fix was a simple:

$ sudo port selfupdate && sudo port clean ghc && sudo port install ghc

You can see the filed bug and changeset at MacPorts for more information.

Tuesday, September 1, 2009

Update: Home made nginx + phusion passenger OS X port

The new v2.2.5 version of Phusion Passenger has been released.

If you want to roll this into OS X without having to use their installer you can simply follow the previous article to get this done.

Important things to note is that wherever you used /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4* you would now use /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.5* (the Portfile and the nginx.conf come to mind).

Friday, July 24, 2009

Mounting a disk image with Windows tools only

Coming from the *NIX/Mac world I frequently find myself mounting ISO images as local disks for software installation, etc.

The same requirement recently came up for me on a Windows platform and I was a little stumped as I did not want to spend any money on this (as I see this as an intrinsic part of the the OS) by buying Alcohol or Daemon Tools.

To my surprise Microsoft actually has a (unsupported) tool (Virtual CD-ROM Control Panel) and driver that provides virtual CD-ROM services.

Unfortunately the solution is XP-only. Download the executable above and install the service. Its readme is quite short and to the point so getting this going is pretty easy.

Have fun!

Wednesday, July 8, 2009

Auto-sizing jQuery-UI Dialog

I recently upgraded all my jquery and jquery-ui (and some supporting) libs. Part of this upgrade was also to get the latest version of nyroModal working.

After several hours of chasing issues with the lib I decided to rip it out of my app and rather use the jquery-ui dialog plugin that ships with the native lib.

A large bugbear with nyroModal has always been that I could never get the dialogs to flow around the different content lengths that I was populating them with properly. After retrofitting the jquery-ui dialog plugin I ran into the similar issues I had with nyroModal.

Whenever the content of my dialog was greater than the distance from the top of the dialog window to the bottom of the viewport, the dialog would simply disappear off the viewport (regardless of whether it had a scrollbar or not).

What I wanted was for the dialog to automagically adjust its vertical height based on the size of the content it contains relative to the top of the dialog position and the bottom of the viewport.

The fact that I am using FBBorderLayout (yep, I am aware this project has been superseded by the jQuery UI plugin but I am taking baby steps here) to set up a nice layout that stops the page from scrolling down and contributes to the dialog sizing off the viewport.

The scene
Based on the documentation on the jquery-ui dialog plugin page I created something like this.

As you can see from this page the dialog stretched off the screen with no scrollbar or way to scroll down to see the content of the dialog.

Ok, not quite what we wanted so I added the following style code to the head to try and get the dialog container to automatically flow and add a scrollbar if the content was too big.

And? It had no effect whatsoever. I tried all possible styling that I could find to try and get the dialog container to play nice but to no avail.

Unlikely rescuer
I scoured the web for a possible solution. All I found were a lot of questions about this phenomenon but no real usable solutions or suggestions (beyond the styling I'd already tried).

I then thought of trying to calculate the height programatically and set the height of the dialog contents prior to displaying it.

You'll notice I stopped the dialog from displaying automatically by adding the 'autoOpen: false' option to the dialog initialization. This is required so that I am able to calculate the height before displaying the dialog.

Here's the complete source file for posterity.


Sunday, July 5, 2009

Installing the MySQL gem on OS X

If you manage your system packages via the MacPorts system you may run into some problems when trying to install the native MySQL driver gem on OS X.

Typically, you'd see output like this when trying to install the gem:

$ sudo gem install mysql
Building native extensions. This could take a while...
ERROR: Error installing mysql:
ERROR: Failed to build gem native extension.

/opt/local/bin/ruby extconf.rb
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... no
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.

Provided configuration options:

Gem files will remain installed in /opt/local/lib/ruby/gems/1.8/gems/mysql-2.7 for inspection.
Results logged to /opt/local/lib/ruby/gems/1.8/gems/mysql-2.7/gem_make.out

Based on those errors the configure process seems to be failing when looking for MySQL libs. The default path to the libs based on extconf.rb seems to be /usr/local which is not where the ports system installs MySQL.

All you need to do is point to the correct mysql_config and let the gem install command line know of this to get things going:

$ sudo gem install mysql -- --with-mysql-config=/opt/local/bin/mysql_config5
Building native extensions. This could take a while...
Successfully installed mysql-2.7
1 gem installed

All's well that ends well.

Wednesday, July 1, 2009

nginx/passenger: Exception RangeError in PhusionPassenger::Railz::ApplicationSpawner (bignum too big to convert into `long')


I am running nginx+passenger on my OS X laptop and after getting the passenger module installed I got the following stacktrace in /opt/local/var/log/nginx/error.log:

*** Exception RangeError in PhusionPassenger::Railz::ApplicationSpawner (bignum too big to convert into `long') (process
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:363:in `setgid'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:363:in `switch_to_user'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:328:in `lower_privilege'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:250:in
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:230:in `report_app_init_st
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:237:in
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:193:in `start_sy
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:162:in `start'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:213:in
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:261:in `spawn_rail
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server_collection.rb:126:i
n `lookup_or_add'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:255:in `spawn_rail
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server_collection.rb:80:in
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server_collection.rb:79:in
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:254:in `spawn_rail
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:153:in `spawn_appl
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:286:in `handle_spa
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:351:in `__send__
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:351:in `main_loo
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:195:in `start_sy
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/bin/passenger-spawn-server:61

So, what's up?

The old switcheroo
Passenger implements User Switching which works around the age old problem that plagues most other scripting languages where they run in the same context as the web server they are accessed through.

The default behaviour is for rails apps to be started as the owner of config/environment.rb. There are however some points that should be noted:

  • The owner of environment.rb must have read access to the rails application folder, and read/write access to the rails application logs folder.

  • This feature is only available if nginx is started by root. This is the case on most nginx installations.

  • Under no circumstances will applications be run as root. If environment.rb is owned as root or by an unknown user, then the rails application will run as the user specified by passenger_default_user.

In my case the project was owned by root so changing that to a local user fixed the problem.

Home made nginx + phusion passenger OS X port

The instructions to do this are similar to those in my previous article.

Install passenger gem
The first step is to install the Phusion Passenger gem:
$ sudo gem install -V -r passenger
Installing RDoc documentation for passenger-2.2.4...

Take note of where the passenger gem installed the nginx module code (ext/nginx/*) that will be required further down. In this case the install directory will be /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/.

Compile nginx passenger module
You also need to compile the passenger nginx module if it was not already compiled:

$ cd /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/
$ sudo rake nginx
g++ ext/nginx/HelperServer.cpp -o ext/nginx/HelperServer -Iext -Iext/common -D_REENTRANT -I/usr/local/include -Wall -g -DPASSENGER_DEBUG -DBOOST_DISABLE_ASSERTS ext/nginx/libpassenger_common.a ext/nginx/libboost_oxt.a -lpthread

Modify the nginx Portfile
Ensure you have nginx port installed. Your Portfile will most likely be in /opt/local/var/macports/sources/

Add the following variant block to the end of the file:

variant passenger description {Enable Phusion Passenger (mod_rails) support} {
configure.args-append --add-module=/opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/nginx

Rebuild/install nginx
When you try and reinstall nginx now it will use the updated Portfile and add the passenger module to the mix:

$ sudo port uninstall nginx && sudo port clean nginx
$ sudo port -vvvvv install nginx +passenger

Nginx passenger configuration
To enable and take advantage of the passenger nginx module you need to edit the /opt/local/etc/nginx/nginx.conf config. Here's an example:

#user nobody;
worker_processes 1;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/;

events {
worker_connections 1024;

http {
include mime.types;
default_type application/octet-stream;

#log_format main '$remote_addr - $remote_user [$time_local] $request '
# '"$status" $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';

#access_log logs/access.log main;

sendfile on;
#tcp_nopush on;

#keepalive_timeout 0;
keepalive_timeout 65;

gzip on;

# Enable Phusion Passenger nginx module
passenger_root /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4;
passenger_ruby /opt/local/bin/ruby;

server {
listen 80;
root /var/www/foo/public; # <--- be sure to point to 'public'!
passenger_enabled on;

Ensure that the port, server_name and root variables have valid values for your environment.

If you want to use a non-production rails environment you can add the following to the block after passenger_enabled:

rails_env development;

Once done you can restart nginx and you should see the following processes that confirm things are running:

$ ps axw | egrep "nginx|passenger"
55935 ?? Ss 0:00.00 nginx: master process nginx
55936 ?? S 0:00.00 nginx: worker process
55925 s001 S 0:00.01 PassengerNginxHelperServer /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4 /opt/local/bin/ruby 3 4 0 6 0 300 1 nobody 4294967294 4294967294 /tmp/passenger.55923
55938 s001 R+ 0:00.00 egrep nginx|passenger

The final test to see if everything is working is to open a browser and to browse to the server_name and port defined above. Errors will be reported in /opt/local/var/log/nginx/error.log as well as the development/production logs of the rails project you're running.

Home made nginx + phusion passenger Debian/Ubuntu dep

If you are using Ubuntu Hardy then you could try using Brightbox's repo instead of building your own deb.

If you are however running some other distro then the following (non-definitive) instructions are for you.

Warning: it may be safer to install nginx + passenger via the passenger-install-nginx-module script as it will match passenger with a tested version of nginx.

The instructions below simply add passenger support to the latest nginx deb in your distro of choice.

Install passenger gem
The first step is to install the Phusion Passenger gem:
$ sudo gem install -V -r passenger
Installing RDoc documentation for passenger-2.2.4...

Take note of where the passenger gem installed the nginx module code that will be required further down:
$ gem contents passenger | egrep '/ext/nginx/' | head -n 1

In this case the install directory will be /var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx.

Compile nginx passenger module
You also need to compile the passenger nginx module if it was not already compiled:

$ cd /var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx
$ sudo rake nginx

(in /var/lib/gems/1.8/gems/passenger-2.2.4)
mkdir -p ext/nginx/libboost_oxt/boost
g++ -Iext -D_REENTRANT -I/usr/local/include -Wall -g -DPASSENGER_DEBUG -DBOOST_DISABLE_ASSERTS -o ext/nginx/libboost_oxt/boost/exceptions.o -c ext/boost/src/pthread/exceptions.cpp
g++ ext/nginx/HelperServer.cpp -o ext/nginx/HelperServer -Iext -Iext/common -D_REENTRANT -I/usr/local/include -Wall -g -DPASSENGER_DEBUG -DBOOST_DISABLE_ASSERTS ext/nginx/libpassenger_common.a ext/nginx/libboost_oxt.a -lpthread

Install build packages

$ sudo aptitude install fakeroot debhelper dpkg-dev autotools-dev libpcre3-dev libssl-dev

Modify APT sources.list
Open /etc/apt/sources.list and ensure you add deb-src lines for each of the universe repos. The easiest way to do this is to simply copy the universe repo lines that exist and change the starting deb to deb-src:

deb jaunty universe
deb-src jaunty universe
deb jaunty-updates universe
deb-src jaunty-updates universe

Replace '' with the repo closest to you and also replace 'jaunty' with your distro. Ensure you do a 'sudo aptitude update' to load the latest package indexes.

Set up a packaging environment

$ mkdir -p /var/tmp/nginx-passenger && cd /var/tmp/nginx-passenger
$ apt-get source nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done
NOTICE: 'nginx' packaging is maintained in the 'Svn' version control system at:
Need to get 537kB of source archives.
Get:1 jaunty/universe nginx 0.6.35-0ubuntu1 (dsc) [1305B]
Get:2 jaunty/universe nginx 0.6.35-0ubuntu1 (tar) [525kB]
Get:3 jaunty/universe nginx 0.6.35-0ubuntu1 (diff) [11.0kB]
Fetched 537kB in 0s (1504kB/s)
gpg: Signature made Sat 21 Feb 2009 07:02:34 AM EST using DSA key ID 3FE63E00
gpg: Can't check signature: public key not found
dpkg-source: extracting nginx in nginx-0.6.35
dpkg-source: info: unpacking nginx_0.6.35.orig.tar.gz
dpkg-source: info: applying nginx_0.6.35-0ubuntu1.diff.gz
$ ls -la
total 548
drwxr-xr-x 3 root root 4096 2009-06-29 08:55 .
drwxrwxrwt 3 root root 4096 2009-06-29 08:39 ..
drwxr-xr-x 8 root root 4096 2009-06-29 08:55 nginx-0.6.35
-rw-r--r-- 1 root root 11018 2009-02-21 08:04 nginx_0.6.35-0ubuntu1.diff.gz
-rw-r--r-- 1 root root 1305 2009-02-21 08:04 nginx_0.6.35-0ubuntu1.dsc
-rw-r--r-- 1 root root 524987 2009-02-21 08:04 nginx_0.6.35.orig.tar.gz
$ rm -f nginx_*
$ mv nginx-0.6.35 nginx-passenger-0.6.35

Package customizations
In order to build our nginx-passenger custom package we need to modify the following packaging resources:

  1. control file

  2. changelog file

  3. rules file

  4. nginx.install file

  5. nginx.logrotate file

  6. passenger.conf file

  7. postinst file

Control file

$ cd nginx-passenger-0.6.35/debian

Edit the control file and change the Source, Maintainer, Uploaders, Package, Provides, Conflicts and Description fields:

Source: nginx-passenger
Maintainer: Charl Matthee
Uploaders: Charl Matthee
Package: nginx-passenger
Provides: httpd, nginx
Conflicts: nginx
Description: small, but very powerful and efficient web server
Nginx (engine x) is a web server created by Igor Sysoev and kindly provided
to the open-source community. This server can be used as standalone HTTP
server and as a reverse proxy server before some Apache or another big
server to reduce load to backend servers by many concurrent HTTP-sessions.
This is a custom nginx package that includes Phusion Passenger
support for nginx.

Note: spacing in the control file is significant.

More information on the control file internals can be found here.

Add a new changelog entry in the changelog file:

nginx-passenger (0.6.35-0custom1) jaunty; urgency=low

* New custom Passenger Phusion nginx module addition

-- Charl Matthee Mon, 29 Jun 2009 09:08:36 +0100

nginx (0.6.35-0ubuntu1) jaunty; urgency=low

* New upstream bugfix release

-- Daniel Hahler Fri, 20 Feb 2009 21:01:23 +0100

More information on the changelog file internals can be found here.

Add the passenger module inclusion to the configure script options when nginx is built by modifying the './configure $(CONFIGURE_OPTS)' block in the rules file like this:

./configure $(CONFIGURE_OPTS) --conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log --pid-path=/var/run/ \
--lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log \
--http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy \
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-http_stub_status_module \
--with-http_flv_module --with-http_ssl_module --with-http_dav_module \
--with-http_realip_module --add-module=PASSENGER_LIB_DIR

Where PASSENGER_LIB_DIR is the path you noted above when you installed the passenger gem. So, the updated block looks like this:

./configure $(CONFIGURE_OPTS) --conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log --pid-path=/var/run/ \
--lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log \
--http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy \
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-http_stub_status_module \
--with-http_flv_module --with-http_ssl_module --with-http_dav_module \
--with-http_realip_module --add-module=/var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx/

More infomration on the changelog file internals can be found here.

Move this file to the correct name based on the package name and add the installation instructions for the default nginx configuration that is required to turn passenger support on:

$ mv nginx.install nginx-passenger.install
$ cat nginx-passenger.install
objs/nginx usr/sbin
html/* var/www/nginx-default
debian/conf/* etc/nginx
debian/passenger.conf etc/nginx/conf.d/
debian/NEWS.Debian usr/share/doc/nginx

Move this file to the correct name based on the package name:

$ mv nginx.logrotate nginx-passenger.logrotate

Create the following file in the debian directory:

passenger_root /var/lib/gems/1.8/gems/passenger-2.2.4;
passenger_ruby /usr/bin/ruby1.8;

Ensure those paths point to the correct passenger isntall and ruby directories.

Add the following steps to the postinst script after the '#DEBHELPER#' token:

# Phusion Passenger module settings for nginx
echo "***********************************************************************"
echo "For each virtual host you want to add Phusion Passenger support for you"
echo "need to create a config like this in /etc/nginx/site-available:"
echo " server {
listen 80;
server_name DOMAIN_NAME;
root /var/www/RAILS_PROJECT/public; #<--- be sure to point to 'public'!
passenger_enabled on;
echo "Don't forget to add a symlink from this file(s) to"
echo "/etc/nginx/site-enabled to enable the configuration in nginx."
echo "***********************************************************************"

This will warn administrators that install this package that they still need to do some manual work to get a virtual host set up to take advantage of the passenger module.

Package rebuild
The package can then be rebuilt just like any other custom deb:

$ cd /var/tmp/nginx-passenger/nginx-passenger-0.6.35
$ dpkg-buildpackage -rfakeroot
dpkg-buildpackage: set CFLAGS to default value: -g -O2
dpkg-deb: building package `nginx-passenger' in `../nginx-passenger_0.6.35-0custom1_i386.deb'.
dpkg-deb: ignoring 1 warnings about the control file(s)
signfile nginx-passenger_0.6.35-0custom1.dsc
gpg: skipped "Charl Matthee ": secret key not available
gpg: [stdin]: clearsign failed: secret key not available

dpkg-genchanges >../nginx-passenger_0.6.35-0custom1_i386.changes
dpkg-genchanges: warning: the current version (0.6.35-0custom1) is smaller than the previous one (0.6.35-0ubuntu1)
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload; Debian-native package (full source is included)
dpkg-buildpackage: warning: Failed to sign .dsc and .changes file

Those signing errors were triggered because I have did not have the required key infrastructure installed. They can be ignored for the moment.

Install the new package:

$ sudo dpkg -i ../nginx-passenger_0.6.35-0custom1_i386.deb
Setting up nginx-passenger (0.6.35-0custom1) ...

Create a config like this for each virtual host you want to add Phusion
Passenger support to. They should be created in

server {
listen 80;
server_name DOMAIN_NAME;
root /var/www/RAILS/public; # <--- be sure to point to 'public'!
passenger_enabled on;

Don't forget to add a symlink from this file(s) to
/etc/nginx/site-enabled to enable the configuration in nginx.

Nginx passenger configuration
To enable and take advantage of the passenger nginx module you need to create a /etc/nginx/conf.d/passenger.conf config:

passenger_root /var/lib/gems/1.8/gems/passenger-2.2.4;
passenger_ruby /usr/bin/ruby1.8;

This will of course have been done when we installed our custom package.

Next, add a virtual host for each of the rails applications you want to support by dropping a file like 'foo' into /etc/nginx/sites-available/:

server {
listen 80;
root /var/www/foo/public; # <--- be sure to point to 'public'!
passenger_enabled on;

Ensure that the port, server_name and root variables have valid values for your environment.

If you want to use a non-production rails environment you can add the following to the block after passenger_enabled:

rails_env development;

To enable the virtual host(s) you need to ensure that there's a symlink linking the virtual host in sites-available to sites-enabled:

$ sudo ln -s /etc/nginx/sites-available/foo /etc/nginx/sites-enabled/foo
$ ls -la /etc/nginx/sites-enabled
total 8
drwxr-xr-x 2 root root 4096 2009-06-29 12:33 .
drwxr-xr-x 5 root root 4096 2009-06-29 10:51 ..
lrwxrwxrwx 1 root root 34 2009-06-02 07:25 default -> /etc/nginx/sites-available/default
lrwxrwxrwx 1 root root 30 2009-06-29 12:33 foo -> /etc/nginx/sites-available/foo

Once done you can restart nginx and you should see the following processes that confirm things are running:

$ ps axf | egrep "nginx|passenger"
28608 pts/1 S+ 0:00 \_ egrep nginx|passenger
28560 pts/1 Sl 0:00 PassengerNginxHelperServer /var/lib/gems/1.8/gems/passenger-2.2.4 /usr/bin/ruby1.8 3 4 0 6 0 300 1 nobody 33 33 /tmp/passenger.28557
28580 ? Ss 0:00 nginx: master process /usr/sbin/nginx
28581 ? S 0:00 \_ nginx: worker process

The final test to see if everything is working is to open a browser and to browse to the server_name and port defined above. Errors will be reported in /var/log/nginx/error.log as well as the development/production logs of the rails project you're running.

Thursday, June 11, 2009

Commandline PDF Manipulation

Want to do any of the following from a *NIX based system commandline with PDF documents:
  • Merge PDF Documents

  • Split PDF Pages into a New Document

  • Rotate PDF Pages or Documents

  • Decrypt Input as Necessary (Password Required)

  • Encrypt Output as Desired

  • Fill PDF Forms with FDF Data or XFDF Data and/or Flatten Forms

  • Apply a Background Watermark or a Foreground Stamp

  • Report on PDF Metrics such as Metadata, Bookmarks, and Page Labels

  • Update PDF Metadata

  • Attach Files to PDF Pages or the PDF Document

  • Unpack PDF Attachments

  • Burst a PDF Document into Single Pages

  • Uncompress and Re-Compress Page Streams

  • Repair Corrupted PDF

One way to accomplish any of these would be to use pdftk (PDF Toolkit).

Something I need to do quite often is concatenate (join) several PDF documents into one document. This can quite trivially be achieved with the following from the commandline:

$ pdftk 01.pdf 02.pdf cat output all_together_now.pdf

Above we concatenated 01.pdf and 02.pdf to form the new combined all_together_now.pdf document.

Wednesday, April 22, 2009

Broken memcached gem/libmemchached deb on Ubuntu

This post follows on a previous post that was OS X centric.

To fix this issue follow the same steps (starting with the download of libmemcached-0.25.14.tar.gz) detailed here.

Because the source installation of libmemcached dropped the libs in a non-standard place (from a Debian/Ubuntu viewpoint) we need a little more magic before things will work.

As root, create /etc/
# Manual installation of libmemcached dropped the libs in a non-standard place
Run ldconfig to load the new config and test to ensure your script can now access libmemcached properly through the memcached gem.

Sunday, March 22, 2009

Unpacking a RPM on Ubuntu (or Debian)

# alien --scripts PACKAGE.rpm
# ar -x PACKAGE.deb
You now have two tarballs control.tar.gz and data.tar.gz.

The first is the information dpkg needs to do a proper installation and configuration of the package, the second is contains the binaries and data files.

Monday, February 23, 2009

Broken memcached gem/libmemchached port on OS X

Update (2009/04/22):
This post was originally only targeted at OS X but I have since noticed Ubuntu also having this same issue. Jump to my additional instructions for Ubuntu.

Recipe for disaster:
  1. Use MacPorts for your package management
  2. Use the ruby memcached gem
  3. Use the libmemcached port as a dependency on the gem above
  4. Upgrade your ports
I upgraded all my ports on my MBP recently and one of the libs that was upgraded in the process was libmemcached (upgraded from v0.25 to v0.26). Unfortunately this has broken my access to memcached on my system (the script dies when trying to connect).

Not knowing what the problem may be I also upgraded my gem to v0.14 which then could not build. A quick search for the issue found this entry on the memcached discussion forum.

How now brown cow?
Evan Weaver, the maintainer of the memcached project, mentioned that libmemcached v0.26 lacks some critical patch and suggested people may have to wait for v0.27 to rectify things.

For those who cannot wait and have inconsistent, broken systems, Evan goes on to mention that you should currently use the libmemcached lib and memcached gem pair from here.

Here he also provides a convenient compatibility matrix so you can see how to align the different versions of libmemcached and the memcached gem.

Unfortunately this messes with things because libmemcached now needs to be installed as a source package in a system managed with MacPorts.


Unfortunately the choices are limited as the versions of libmemcached that ships as a port are either v0.26 (the latest, but broken for our purposes) or v0.25 (which aligns with the older v0.13 of the memcached gem).

First off remove any libmemcached ports you may have installed:
$ sudo port uninstall libmemcached
Download the libmemcached source archive, extract, build and install it:
ibm-99tvvxc:msp charl$ cd /var/tmp/
$ wget
--2009-02-24 10:50:40--
Connecting to||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 428047 (418K) [application/x-gzip]
Saving to: `libmemcached-0.25.14.tar.gz'

100%[==========================================================>] 428,047 135K/s in 3.1s

2009-02-24 10:50:49 (135 KB/s) - `libmemcached-0.25.14.tar.gz' saved [428047/428047]

$ tar zxvf libmemcached-0.25.14.tar.gz
$ cd libmemcached-0.25.14
$ ./configure
checking build system type... i386-apple-darwin9.6.0
checking host system type... i386-apple-darwin9.6.0
checking target system type... i386-apple-darwin9.6.0
checking for a BSD-compatible install... /usr/bin/install -c
$ make
Making all in docs
/usr/bin/pod2man -c "libmemcached" -r "" -s 3 libmemcached.pod > libmemcached.3
/usr/bin/pod2man -c "libmemcached" -r "" -s 3 libmemcached_examples.pod > libmemcached_examples.3
$ sudo make install
Making install in docs
make[2]: Nothing to be done for `install-exec-am'.
test -z "/usr/local/share/man/man1" || ../config/install-sh -c -d "/usr/local/share/man/man1"
/usr/bin/install -c -m 644 './memcat.1' '/usr/local/share/man/man1/memcat.1'
You may want to keep the source directory around so that you can easily uninstall (run sudo make uninstall in the source directory) the source installation later when the relevant aligned packaged versions are available for installation.

The final step is to install the gem:
$ sudo env ARCHFLAGS="-arch i386" gem install -r -V memcached --no-rdoc --no-ri
GET 200 OK:
GET 200 OK:
connection reset after 2 requests, retrying
GET 200 OK:
Installing gem memcached-0.14
Successfully installed memcached-0.14
1 gem installed

We now have libmemcached v0.25.14 and the memcached v0.14 gem installed and we can go on with our lives.

Sunday, February 22, 2009

Silly Rails Environment Optimisation

Do you find supplying the ./script base directory for any of the commands you want to run in it annoying?

Why not practice some Convention over Configuration and assume these commands will always be run from within a valid rails project directory.

Based on this assumption you can drop the following into your .bash_profile (OS X) or some system-wide profile to facilitate some laziness:
# Rails aliases
alias about="./script/about"
alias console="./script/console"
alias dbconsole="./script/dbconsole"
alias destroy="./script/destroy"
alias generate="./script/generate"
alias performance="./script/performance"
alias plugin="./script/plugin"
alias process="./script/process"
alias runner="./script/runner"
alias server="./script/server"
Don't forget to load it into you current shell with:
$ . ~/.bash_profile
Now you can simply run 'console', 'dbconsole', 'server', etc. when you're in a valid rails project directory without having to append that annoying base directory.

Caveat: If similar commands exist on your system or are introduced by a non-rails package a conflict will arise and the alias version of the command will win. Also, TAB-completion now no longer works for these commands and you'll have to type them out in their entirety.

Friday, February 20, 2009

What's next with acts_as_state_machine?

I've been using acts_as_state_machine and there seems to be at least two convenience methods missing that I find quite useful:
  1. next_state - what state will we be transitioning to
  2. next_states - list of states that we still need to transition to
Drop the following in a suitable place (your model using acts_as_state_machine will do) and abuse it to your hearts content:
# Monkey patch acts_as_state_machine to allow us to see what the next states are
# for the current state
module ScottBarron
module Acts
module StateMachine
module InstanceMethods
def next_state(state=nil)
state ||= current_state()
self.class.read_inheritable_attribute(:transition_table).each_value do |event|
event.each do |transition|
return if transition.from == state

def next_states(state=nil)
state ||= current_state()
states = []
while state = next_state(state)
states << state
Thanks to Lourens Naude's' post on acts_as_state_machine for providing me with some ideas and to Scott Barron for the great acts_as_state_machine plugin.

Wednesday, February 11, 2009

Signing your own gems

Chapter 9 (Signing Your Gems) of the RubyGems User Guide contains everything you need to know to do this.

Here's a quick overview of the steps required.

Build a certificate and private key pair
$ cd /to/where/you/wish/to/keep/your/cert_and_key
$ gem cert --build
Two files are generated from this: gem-private_key.pem & gem-public_cert.pem.

Ensure you keep gem-private_key.pem somewhere safe that only you have access to.

Modify your build specification
Add the following spec to your existing gem's gemspec or to the relevant Rakefile ("s" is your Gem::Specification instance):
$ s.signing_key = 'gem-private_key.pem'
$ s.cert_chain = ['gem-public_cert.pem']
Rebuild your package and you're done!

Your own gem repository

Setting up your own repository is quite simple. All you need is the following:
  • Web server
  • Builder gem
  • A repository
  • Client setup
Web server
I used nginx but you should be able to tailor this to your needs without too much effort.

Create a directory to contain your local gem repository:
$ sudo mkdir -p /var/www/gems-repo/gems
Note: to get everything working correctly you need to ensure you have that /gems subdirectory off your base directory configured in nginx. You can find more info on this at the RubyGems FAQ.

Add a virtual host config in /etc/nginx/sites-available for your gems repository (I'm going to assume
$ cat /etc/nginx/sites-available/
server {
listen 80;
access_log /var/log/nginx/;
root /var/www/gems-repo;
Ensure you drop a symlink in /etc/nginx/sites-enabled/ for your virtual host and restart nginx.

Install the builder gem
$ sudo gem install builder
Build your repository
Jump into the base directory defined above (/var/www/gems-repo). Copy all your own gems into /var/www/gems-repo/gems/. Then generate the relevant resource files for rubygems:
$ sudo gem generate_index -d /var/www/gems-repo
Loading 1 gems from /var/www/gems-repo
Loaded all gems
Generating quick index gemspecs for 1 gems
Generating specs index
Generating latest specs index
Generating quick index
Generating latest index
Generating Marshal master index
Generating YAML master index for 1 gems (this may take a while)
Compressing indicies
You would re-run this every time you add to or make changes to the repository.

Client setup
Finally run the following on all local clients that need access to the repository:
$ sudo gem sources -a
You can ensure the extra source was added by running 'gem sources'. If you now do a search for you gem that you added above you should see it listed as a target.

Tuesday, January 27, 2009

Stunneling (SSL Tunnels) MySQL on Ubuntu

We have really restrictive IT policies at my current employer which dictates that no client/server communication can happen in the clear. This has some interesting implications as most software servers do not support encryption or encrypted tunnels in any way.

What are the options?
  • Write our own software that supports client/server encryption
  • Only use existing applications that support client/server encryption
  • Treat the encryption layer as an abstraction that the client/server is not aware of

The first two options have obvious drawbacks in terms of time and choice. The third seems to be the best way to transparently insert encryption into an existing architecture without requiring massive changes.

This approach can be done by:

  • SSH tunnels
  • OpenVPN tunnels
  • Stunnel

There may be other options but these are the main ones I have had experience with. SSH tunnels are very convenient but tend to be unstable requiring restarts every now and again.

OpenVPN has also provided very little stability requiring frequent restarts.

Lets explore the setup of the third option then, stunnel.

The stunnel home page describes stunnel as:

Stunnel is a program that allows you to encrypt arbitrary TCP connections inside
SSL (Secure Sockets Layer) available on both Unix and Windows. Stunnel can allow
you to secure non-SSL aware daemons and protocols (like POP, IMAP, LDAP, etc) by
having Stunnel provide the encryption, requiring no changes to the daemon's code.

Sounds promising. They have a really comprehensive examples section on their site which I highly recommend you have a look at.

In my case I needed to get a MySQL master/slave pair to replicate via a secure channel. MySQL seems to support SSL but the package requires a rebuild with the required options which would mean I need a non-standard MySQL package which makes me itchy. I also needed a similar solution for several other applications that need secure communications channels so I've decided on this solution throughout my architecture.

See this Ubuntu thread for more info on rebuilding the MySQL package with SSL support if you're so inclined.

Our installation targets are an Ubuntu servers and will therefore be flavored accordingly but you should be able to easily adapt anything mentioned here for your platform of choice.

If you are going to be using this for the same purposes I am ensure that you have a working MySQL master/slave setup. You don't want to troubleshoot multiple applications if something is not playing nice.

A simple aptitude install stunnel4 on the master and slave should do the trick.

Now, keep in mind that if one of the nodes you have stunnel installed on is acting as both a stunnel client and stunnel server you need to have at least one stunnel server running for the client entries in a config file and one for the server entries in a config file.

Not heading this warning will result in you seeing errors like this on the client:

2009.01.28 07:02:16 LOG7[11508:3082910608]: SSL state (connect): before/connect initialization
2009.01.28 07:02:16 LOG7[11508:3082910608]: SSL state (connect): SSLv3 write client hello A
2009.01.28 07:02:16 LOG7[11508:3082910608]: SSL alert (write): fatal: handshake failure
2009.01.28 07:02:16 LOG3[11508:3082910608]: SSL_connect: 1408F10B: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
2009.01.28 07:02:16 LOG5[11508:3082910608]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket

And this on the server:

2009.01.28 07:10:06 LOG7[13426:3082861456]: SSL state (connect): before/connect initialization
2009.01.28 07:10:06 LOG7[13426:3082861456]: SSL state (connect): SSLv3 write client hello A
2009.01.28 07:10:06 LOG7[13426:3082861456]: SSL alert (write): fatal: handshake failure
2009.01.28 07:10:06 LOG3[13426:3082861456]: SSL_connect: 1408F10B: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
2009.01.28 07:10:06 LOG5[13426:3082861456]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket

My convention is to use the default config file (/etc/stunnel/stunnel.conf) that ships with the stunnel4 package for the client configuration and a modified copy of that as a config (/etc/stunnel/mysql-master.conf, in this specific case) for the server.

If you will not be using the MySQL master server as a stunnel client then you can simply ensure that your /etc/stunnel/stunnel.conf is the same as my /etc/stunnel/mysql-master.conf.

Keep in mind that multiple config files in /etc/stunnel/ will cause multiple stunnel servers to be run with each of the files found there.

First off, let's generate our own key (/etc/ssl/master.pem) that we'll be using for encryption. This key must be treated as a secure file that should have the relevant access controls placed on it.

root@master:~# cd /etc/ssl/
root@master:~# openssl req -new -x509 -nodes -days 730 -out master.pem -keyout master.pem
root@master:~# chmod 600 master.pem
root@master:~# dd if=/dev/urandom of=temp_file count=2
root@master:~# openssl dhparam -rand temp_file 512 >> master.pem
root@master:~# ln -sf master.pem `openssl x509 -noout -hash < coretools01.pem`.0

Now, change the following lines in /etc/stunnel/mysql-master.conf:

cert = /etc/ssl/master.pem
;client = yes
debug = 7
output = /var/log/stunnel4/stunnel.log

accept = 443
connect =
TIMEOUTclose = 0

This config will ensure we're running as a stunnel server, listening for incoming SSL connections on and shunting the unencrypted connection to (ensure your MySQl master is listening for connections here).

Fire stunnel up on the master with /etc/init.d/stunnel4 start && tail -f /var/log/stunnel4/stunnel.log.

Just modify the default /etc/stunnel/stunnel.conf that was installed via the package by changing the lines below:

;cert = /etc/stunnel/mail.pem
;key = /etc/stunnel/mail.pem
debug = 7
output = /var/log/stunnel4/stunnel.log
client = yes

accept =
connect =
TIMEOUTclose = 0

This sets up the MySQL slave machine as a stunnel client, forwarding all unencrypted connections on via SSL to Just replace with the correct FQDN for your master server. You also need to ensure that your MySQL slave's config points to for its replication requirements.

Fire stunnel up on the slave with /etc/init.d/stunnel4 start && tail -f /var/log/stunnel4/stunnel.log.

An initial test can be done from the slave with the mysql command line utility in a separate terminal session by connecting to a suitable user/db configured on the master with access from 'localhost' or '':

root@slave:~# mysql -u replication -h -P 3307 -p test
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 21
Server version: 5.0.51a-3ubuntu5.4-log (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.



Now, apply your general method for determining if your master and slave is in sync. You can use the logs at /var/log/stunnel4/stunnel.log if you hit any snags. Once you're happy with everything ensure you disable the debugging in the stunnel config files.

About Me

My photo
I love solving real-world problems with code and systems (web apps, distributed systems and all the bits and pieces in-between).