Saturday, April 19, 2008

Baking CakePHP with nginx

I've switched all my web server related infrastructure to nginx which is a high performance HTTP server (amongst other things) which blows apache out of the water (IMNSHO).

If you're a CakePHP user you'll know that it has some special rewriting requirements which can be found in $ROOT/.htaccess:

$ cat .htaccess

RewriteEngine on
RewriteRule ^$ app/webroot/ [L]
RewriteRule (.*) app/webroot/$1 [L]

Unfortunately this .htaccess file is an apache mechanism to achieve the URL rewriting that is required to get CakePHP to play nice. The apache rewrite rules do not translate directly to something nginx can use though so some work is required to get things going.

Being lazy (and believing that no problem I discover is unique to me) I had a look a round and found this article by Chris Hartjes which needed a some mods for it to work for my setup:

# CakePHP rewrite rules
location / {
root /opt/local/html/live_site;
index index.php;

# Serve static page immediately
if (-f $request_filename) {
break;
}

if (!-f $request_filename) {
rewrite ^/(.+)$ /index.php?url=$1 last;
break;
}
}

Here's my complete nginx.conf to give you an idea of how everything fits together:


user nobody;
worker_processes 1;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/nginx.pid;


events {
worker_connections 1024;
}

http {
include etc/nginx/mime.types;
default_type application/octet-stream;
server_names_hash_bucket_size 128;

sendfile on;
keepalive_timeout 20;
tcp_nodelay on;

server {
listen 80;
server_name localhost;
rewrite_log on;

#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root share/nginx/html;
}

# Serve static content directly with some caching goodness
location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico)$ {
root /opt/local/html/live_site/app/webroot;
access_log off;
expires 1d;
}

# CakePHP rewrite rules
location / {
root /opt/local/html/live_site;
index index.php;

# Serve static pages immediately
if (-f $request_filename) {
break;
}

# Rewrite all other URLs
if (!-f $request_filename) {
rewrite ^/(.+)$ /index.php?url=$1 last;
break;
}
}

# Pass PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {
root /opt/local/html/live_site;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
include /opt/local/etc/nginx/fastcgi_params;
}
}
}



Friday, April 18, 2008

PHP5 SimpleXML and CDATA

I am in the process of porting a rather large PHP4 application to PHP5 (just in time for PHP6, yes, yes, I know) for one of my customers. Most of the application is pure imperative programming so the switch has been rather painless.

Unfortunately (for me) they have made rather judicious use of the PHP4 domxml extension libraries which no longer exist in PHP5 (where they have been replaced with the dom extension).

Moving from domxml to dom was straight forward (looping over tags, tags, attributes and text inside tags are addresses differently) so I chose to simply re-write the code to use the new dom extensions instead of opting for a translation library like the one provided by Alexandre Alapetite.

One exception to this was the use of < !--[CDATA[...]]--> blocks which SimpleXML simply seemed to discard when creating a new object.

A quick look around Google (here, here and here) and I found what needed to be done to address SimpleXML's ignorant behaviour.

The SimpleXML constructor allows you to pass in extra libxml2 parameters which allow you to get further functionality out of the library. The one I was interested was of course:

LIBXML_NOCDATA (integer)
Merge CDATA as text nodes

So, simply changing my constructor from:
$xml = new SimpleXMLElement($text)
to:
$xml = new SimpleXMLElement($text, LIBXML_NOCDATA)
was all that was required for me to gain access to those < !--[CDATA[...]]--> structures.


Tuesday, April 1, 2008

Aspell, PSPELL, PHP and OS X

Until quite recently there was no way to use Aspell (via the PSPELL PHP libs) on an OS X host that was using the MacPorts system for package management.

This was simply because of the lack of a pspell variant for the php{45} packages.

Port of call
The ports system has the notion of a variant for packages that are conditional modifications of port installation behaviour.

There are two types of variants: user-selected variants and platform variants.

User-selected variants are options selected by a user when a port is installed while platform variants are selected automatically by MacPorts' base according to the OS or hardware platform (Darwin, FreeBSD, Linux, i386, PPC, etc.).

More stuff and less fluff
There are several ways of working this. The first would be to log a ticket at the MacPorts trac with a request to add the variant.

Depending on the package maintainer's load you may be a response pretty quickly. I've generally gotten something back within days of logging the ticket.

In the meantime your universe cannot come to a halt waiting for someone else to add the next greatest thing as a variant to your favourite package. So here's some manual steps to get things up and running in the meantime:

  • Install Aspell
  • Recompile and install PHP with the required PSPELL support
  • Check php to ensure the new PSPELL libs are active
  • Do a simple test to see if everything is working as it should

Install Aspell
I'll assume you're using MacPorts for your your package management on your OS X host.

Grab the aspell application, libs and whichever dictionaries catch your fancy:

$ sudo port install aspell aspell-dict-en


Be sure to install at least one dictionary or your spell checking days will be rather deflated.

Recompile and install PHP with the required PSPELL support
To manually add a compilation flag you need to edit the Portfile that comes with your installed PHP version. Mine is located at:

/opt/local/var/macports/sources/rsync.macports.org/release/ports/www/php5/Portfile

Edit the Portfile and add “–with-pspell=${prefix}” to configure.args. You can then re-ininstall PHP and it should now use the modified Portfile to compile PHP.

Check php to ensure the new PSPELL libs are active
Use the following script from the command line to determine if your PHP was re-installed with the required PSPELL bits enabled:

$ php -r 'phpinfo();' | grep PSpell
PSpell Support => enabled

Do a simple test to see if everything is working as it should
You should be able to just use the example from the PHP documentation page to ensure everything is fine:

$ cat /tmp/t_pspell.php

$ php -f /tmp/t_pspell.php
Sorry, wrong spelling

The packaging gods exist!
A few days after creating this ticket on the MacPorts trac I got a response from the package maintainer informing me that the variant had been added to all the relevant PHP packages.

Cool!

If your ports distribution files are up-to-date you should now be able to do the following to see which variants are available for your PHP version of choice:

$ port info php5
php5 5.2.5, Revision 2, www/php5 (Variants: universal, darwin_6, darwin_7, macosx, apache, apache2, fastcgi, gmp, imap, pspell, tidy, mssql, snmp, macports_snmp, mysql3, mysql4, mysql5, oracle, postgresql, sqlite, ipc, pcntl, pear, readline, sockets)
http://www.php.net/

PHP is a widely-used general-purpose scripting language that is especially suited for developing web sites, but can also be used for command-line scripting.

Library Dependencies: libxml2, libxslt, openssl, zlib, bzip2, libiconv, expat, gettext, tiff, mhash, libmcrypt, curl, pcre, jpeg, libpng, freetype
Platforms: darwin freebsd
Maintainers: ryandesign@macports.org jwa@macports.org

Now that pspell is listed as a variant you can install PHP with this variant by doing the following, after removing previous PHP version:

$ sudo port install php5 +pspell



Friday, February 22, 2008

Pre-queue content-filter connection overload

In the last while I've been seeing the following error pop up in my logwatch report for postfix:


*Warning: Pre-queue content-filter connection overload


At first I was concerned that the pre-queue content-filtering subsystem of postfix was somehow being overwhelmed and I was possibly loosing mail. Digging around a bit more lead to these types of log entries that seemed like they were the subject of the logwatch report:


Feb 21 13:01:45 pyxidis postfix/smtpd[5994]: connect from unknown[unknown]
Feb 21 13:01:45 pyxidis postfix/smtpd[5994]: lost connection after CONNECT from unknown[unknown]
Feb 21 13:01:45 pyxidis postfix/smtpd[5994]: disconnect from unknown[unknown]


Huh?!
From what I can glean here the log entries above indicate that a SMTP connection was established with the kernel but the connecting host hot potatoed it before postfix was able to process the connection.

When postfix tries to process the connection there's nobody home because the kernel had already removed the connection and it dumps something like the lines above to the mail log.

Wherefor?
Here's the scoop from the logwatch docs:


This sometimes occurs in reaction to a portscan or broken bots, or when postfix is overloaded, due to excessive header_checks / body_checks content filtering, or even too few smtpd processes to service the demand. One could reduce the number of header_checks and body_checks, and possibly set smtpd_timeout to 60 (seconds). The key is that existing clients are overloading the number of smtpd daemons. The postfix-logwatch section configuration variable is postfix_ConnectionLostOverload, and the command line option is --connectionlostoverload. If you consider this sub-section to be meaningless, set the level limiter value to 0 and the sub-section will be suppressed.


I was not going to change my header or body checks (because they keep the unwashed spammers at bay) so I opted for tuning my smtpd_timeout down to 30 seconds in main.cf.

Because this looks like it is a possible resource issue you could also up the amount of smtpd processes allowed to service the pre-queue content-filtering subsystem.


Wednesday, February 20, 2008

Firefox 3 (beta x) and Firebug

18 February 2008

Firefox 3 (beta x) and Firebug

My favourite extension, by far, for Firefox is Firebug. It is a combination of strace and tcpdump for web applications. It allows you to drill down into all aspects of the send-response loop between your browser and a web app.

If you're doing any JavaScript/AJAX development this tool will be invaluable to you!

Living on the edge though, as you do, I upgraded the version of Firefox I was running to v3.0b3 and too my annoyance Firebug now no longer worked.

Damn.

Lucky for me the crew at Fireclipse have enhanced Firebug 1.05 by Joe Hewitt with enhancements and bug fixes. I simply grabbed the XPI from here and installed it from within Firefox.

Et VoilĂ !

Another Reality Dysfunction is averted and life continues unimpeded.


Dovecot time Machine

05 January 2008

Dovecot time Machine

I have a XEN virtual machine which was complaining about time. Dovecot seemed to be the most vocal with errors like this in the logs:


dovecot: IMAP(charl): Time just moved backwards by 1 seconds. I'll sleep now until we're back in present. http://wiki.dovecot.org/TimeMovedBackwards


I run openntpd (OpenBSD NTP daemon) on the box but that was not seemingly keeping the date, well, up-to-date. Manually running ntpdate was also not providing the sync I sought and because this is a XEN box I have no access to hwclock.

Travelling back in time
A bit of digging on my provider's forums though showed that they were controlling the time syncing and had forgotten to turn the time sync back on after some troubleshooting they were doing.

That's all good and well but my box was still not syncing.

The Dovecot linked article in the log error message simply states:


With Xen you should run ntpd only in dom0. Other domains should synchronize time automatically (see this Xen FAQ).


XEN wisdom
The XEN FAQ has the following relevant things to say about time on a XEN box:


Q: My xen machines doesn't accept setting its time. Basically, it's stuck to RTC time. ntpdate, date, hwclock all "seem" to work, but they don't actually change the system time. The only way I have to change it right now is to change it in the BIOS.
A: Only affects 1.0. Fixed in newer versions.

Q: Where does a domain get its time from?
A: Briefly, Xen reads the RTC at start of day and by default will track that with the precision of the periodic timer crystal. Xen's estimate of the wall-clock time can only be updated by domain 0. If domain 0 runs ntpdate, ntpd, etc. then the synchronised time will automatically be pushed down to Xen every minute (and written to the RTC every 11 minutes, just as normal x86 Linux does). All other domains always track Xen's wall-clock time: setting the date, or running ntpd, on these domains will not affect their wall-clock time. Note that the wall-clock time exported by Xen is UTC --- all domains must have appropriate timezone handling (i.e. a correct /etc/localtime file).

Q: Is there is some cross-domain time synchronization : are they always in perfect sync, or should I run some kind of ntp in each subdomain ? Or only domain 0 would be enough ?
A:If you want each domain to keep its own time, there are two ways to cause a domain to run its wallclock independently from Xen:
1. Specify 'independent_wallclock' on the command line.
2. 'echo 1 >/proc/sys/xen/independent_wallclock'

To reenable tracking of Xen wallclock:
1. 'echo 0 >/proc/sys/xen/independent_wallclock'


"Shut it down, Shut it down forever!"
Following the FAQ I modified /proc/sys/xen/independent_wallclock and added it to /etc/sysctl.conf ("xen.independent_wallclock = 1") so that the change would survive a reboot.

I also changed my timezone to the same local as where I am working currently by doing the following:

$ sudo cp /usr/share/zoneinfo/Australia/Sydney /etc/localtime

Time marches on
openntpd now keeps time synced and Dovecot no longer complains about running in the future.


Thursday, December 27, 2007

Ubuntu's little upstart

One of my customers has a mix of Debian and Ubuntu servers installed in their network with a treasure trove of distribution versions from Debian GNU/Linux 3.0 (Woody) to Ubuntu 7.10 (Gutsy Gibbon). They have a fair amount of custom debs which need to coexist on these servers.

While installing a new Gutsy box I noticed that several of the in-house debs were failing while trying to modify /etc/inittab (our preferred way to keep things running that should never die). This was quite confusing to me as the last Ubuntu server I worked with was Ubuntu 6.06.1 LTS (Dapper Drake) which did not exhibit this weirdness.

Down the rabbit hole
Imagine my amazement when I tried to find /etc/inittab and it was completely missing! My reality reset, I checked again and it was still missing.

At first I thought some critical package (sysvinit in the versions of Ubuntu I know) was somehow missing after the base install. I jumped on the Ubuntu packages site and did a search for 'sysvinit'. Sure enough, there it was but the file list showed no inittab.

Suddenly I felt like I a four year old whose mommy had lost them at the Mall.

Further down the spiral
The file list did however list a bunch of items that referred to something called 'upstart-compat-sysv'. Browsing to this package listed the follwoing description for the package:

This package contains compatibility tasks and utilities that emulate the behaviour of the original sysvinit package, including runlevels, and ensures that the initscripts in /etc/rc*.d are still run.

OK. I faintly hear someone announcing that my mother is looking for me at the Mall's security office.

So if upstart-compat-sysv _emulates_ the original sysvinit, then what has _replaced_ it?

Upstart, show thy face
Google provided me with this link that points to the Ubuntu upstart page which went a long way towards uniting me and my mommy:

Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.

It was originally developed for the Ubuntu distribution, but is intended to be suitable for deployment in all Linux distributions as a replacement for the venerable System-V init.

Sounds even more pervasive than a simple missing iniitab!

Features:
  • Tasks and Services are started and stopped by events
  • Events are generated as tasks and services are started and stopped
  • Events may be received from any other process on the system
  • Services may be respawned if they die unexpectedly
  • Bi-directional communication with init daemon to discover which jobs are running, why jobs failed, etc.
To my amazement upstart has been turned on by default since Ubuntu 6.10 (Edgy Eft) which explains why I was totally in the dark (but upstart was seemingly nothing new to everyone that had been following the normal upgrade path).

So, upstart was not only superseding everything we were trying to do with inittab but also changed the way one interacts with scripts via /etc/event.d.

Why reinvent the wheel?
With the Ubuntu decision to move to the 2.6 kernel, and all the hotplug facilities it provides, they were left with several problems in Dapper. The kernel could now completely cope with hardware coming and going but that had a knock-on effect in that there was now no way to guarantee that particular devices were available at particular point in the boot process.

For example: Dapper cannot mount USB disks in /etc/fstab because it is not guaranteed that the block device exists at the point in the mount process where that happens.

Several other reasons are also provided on the Ubuntu site.

At first, the team decided to look at the available alternatives in third party projects such as Solaris SMF, Apple's launchd, the LSB initserv/chkconfig tools and initNG. None of these met the design criteria that the team had set out for themselves and in true OSS style they decided to roll their own.

The design and implementation documentation is pretty clear so I won't repeat it here.

Epilogue
In the end I simply had to add some logic to the custom debs to check if the system was running upstart and do The Right Thing(TM) (using upstart if it was available or falling back to using inittab) based on that. That amounted to dropping a relevant file in /etc/event.d/ for every server that we previously ran from inittab for systems that were using upstart.

Further Reading



Friday, October 5, 2007

Func it up with JavaScript

UPDATE: David Pollak has a great introduction to FP via JavaScript and Ruby.

Using functional programming paradigms in JavaScript are non-existent, clumsy, verbose and difficult to read in most cases. "Oliver Steel":http://osteele.com/ has built an excellent little library that does the grunt work for you when trying to get your func on.

To Func || ! Func
Most people go about their imperial programming days without a thought of functional programming techniques and how they can be applied to their daily problems. In most part this is due to the inherent difficulties with _doing_ functional programming in their tool of choice.

I urge you to do some further investigation (read Why Functional Programming Matters) into functional programming techniques even if you think you'll never use them anywhere. This is not intended to be an exercise in (academic) pointlessness but to place you outside of your comfort zone and expand your thinking across different domains. The depth of knowledge gained from this will enable you to solve problems from a larger pool of tools (sometimes allowing you to bring functional programming paradigms to bare on a problem or simply augmenting your existing tools for a more efficient or elegant solution to problems).

Learn to get your Func on!

From a pure functional programming language perspective this approach to problem solving offers the following advantages over the imperative and OOP approaches:
  • No (re-)assignment
  • No side effects
  • No flow of control
Functional calls can therefore have no other effect that to compute its result. In a pure functional language there are no assignments statements. Once you assign a value to a variable the variable never changes. In this sense variables in a functional language have more in common with algebraic variables that the normal programming stock we're used to.

At first this seems like a debilitating restriction but after giving your brain some time to expand you'll find that this simple restriction eliminates one of the largest sources of bugs in programming and makes the order of execution irrelevant as no side-effect can change the value of an expression and it can be evaluated at any time.

Gone are the days of worrying about orchestrating the flow control of your program. Your programs are now referentially transparent because expressions, variables and their values can be freely evaluated and replaced at any time.

Elements of Func
From a strict academic sense functional programming refers to programs that has a main body which is a function that receives it's input as its arguments and delivers the output/transformation as it's result.

So far this definition should not seem too foreign to most people that have worked with c/c++. Where this departs from the general imperative meme is that the main function is generally defined in terms of other functions, which in
turn are defined in terms of still more functions, until at the lowest level the functions are first-class citizens (language primitives).

These functions are much like ordinary mathematical functions (in that the same input will always deliver the same output).

Higher-order programming (HOP), function level programming (FLP) and partial function application (PFA) are all styles used in functional programming.

Programming Transcendence
HOP is the ability to use functions as values, in other words you can pass functions as arguments to other functions and functions can be returned as a value of other functions. An example of HOP in JavaScript would be something like the simple sort() method that you can apply to an array.

In its simplest form the sort() function takes an unordered/ordered array and sorts the array:

var a = [2,3,1,4]
document.write(a.sort())
// prints "1,2,3,4"

The sort() method however allows you to use a comparison function as an optional argument, allowing you to pass it a function as a parameter, ergo implementing HOP. Let's assume we've got an array of date objects that we want to sort in a chronological order:

array_of_dates.sort{ function (x, y) { return x.date - y.date; } }

Here we pass in an anonymous function as our comparison function to sort(). The anonymous function is called for each object in the array of dates and it must return a negative value when x < x ="="> y.

This technique is best used when you have at least two functions that perform the same take with a slight variance. Here you would then combine the functions by replacing the part(s) that are different with a function call to a separate function which is passed in to the more general function as a function parameter.

The Functional library implements string lambdas that allow you to express some of the functional programming tools more succinctly. The traditional JavaScript way of doing say a map or filter would be something like this:

map(function(x){return x+1}, [1,2,3]) // returns [2,3,4]
filter(function(x){return x>2}, [1,2,3,4]] // returns [3,4]
some(function(w){return w.length < 3}, 'are there any short words?'.split(' ')) // returns false

Instead, string lambdas allow you to write this in the following way:

map('x+1', [1,2,3])
select('x>2', [1,2,3,4])
some('_.length < 3', 'are there any short words?'.split(' '))

Here are some other way to bend a program to your functional will using simply map, reduce and filter:

// Double the items in a list:
map('*2', [1,2,3]) // [2, 4, 6]

// Find just the odd numbers:
filter('%2', [1,2,3,4]) // [1, 3]

// Find just the evens:
filter(not('%2'), [1,2,3,4]) // [2, 4]

// Find the length of the longest word:
reduce(Math.max, 0, map('_.length', 'how long is the longest word?'.split(' '))) // 7

// Parse a binary array:
reduce('2*x+y', 0, [1,0,1,0]) // 10

// Parse a (non-negative) decimal string:
reduce('x*10+y', 0, map('.charCodeAt(0)-48', '123'.split(/(?=.)/))) // 123
Much more succinct, clear to read and easier to understand.

Func Levels
Value-level programming manipulates values, transforming a sequence of inputs into an output. Function-level programming manipulates functions, applying operations to functions to construct a new function. This new function transforms the inputs into outputs.

How can we make JavaScript dance to a functional-level programming paradigm using the Functional library as meter? Here's some example's:

// Find the reciprocal only of values that test true:
map(guard('1/'), [1,2,null,4]) // [1, 0.5, null, 0.25]

// Apply '10+' only to even values, leaving the odd ones alone:
map(guard('10+', not('%2')), [1,2,3,4]) // [1, 12, 3, 14]

// Write a version of map that only applies to the evens:
var even = not('%2');
var mapEvens = map.prefilterAt(0, guard.rcurry(even));
mapEvens('10+', [1,2,3,4])

// Find the first power of two that's greater than 100:
until('>100', '2*')(1) // 128

// Or, the first three-digit power of two (these are equivalent):
until('String(_).length>2', '2*')(1)
until(compose('>2', pluck('length'), String), '2*')(1)
until(sequence(String, pluck('length'), '>2'), '2*')(1)
Hot/Medium/Mild Curry?
Partial function application (aka currying) transforms a function that takes n arguments into a function that takes only one argument and returns a curried function of n - 1 arguments.

In English please! OK, let's try:

Currying is the process of partially, or incrementally, supplying arguments to a function. Curried functions are delayed functions expecting the remainder of the arguments to be supplied. Once all the arguments are supplied, the function evaluates normally. So, curried functions lead to lazy execution of the complete function.

From the definitions above it is clear that partial function application, or specialisation, creates a new function out of an old one. To illustrate how we apply this with Functional we'll implement between(x, y, z) which determines whether y is bounded by x and z. We then curry the first and last arguments to produce a function that tests whether a number is positive:

// Function that needs to be curried
function increasing(a, b, c)
{
return a < b && b < c;
}

// Define the set of positive numbers via lazy evaluation
var positive = increasing.partial(0, _, Infinity);

// Determine if each of the values -1, 0 and 1 fall in our range
map(positive, [-1, 0, 1]) // [false, false, true]

// Define the set of negative numbers via lazy evaluation
var negative = increasing.partial(-Infinity, _, 0);

// Determine if each of -1, 0 and 1 fall in our range
map(negative, [-1, 0, 1]) // [true, false, false]
Currying leads to lazy evaluation which allows you to work with structures like the infinite sets we created above. Cool eh?!

Epilogue
Functional does a great job at making your life easier if you want to experiment with functional programming in JavaScript without getting yourself tangled up in the verbose, standard syntax. The creator does however offer a word of warning with regards to performance if you use this lib in production. Functional is also confirmed to work in Firefox 2.0, Safari 3.0, and MSIE 6.0.


About Me

My photo
I love solving real-world problems with code and systems (web apps, distributed systems and all the bits and pieces in-between).