While installing a new Gutsy box I noticed that several of the in-house debs were failing while trying to modify /etc/inittab (our preferred way to keep things running that should never die). This was quite confusing to me as the last Ubuntu server I worked with was Ubuntu 6.06.1 LTS (Dapper Drake) which did not exhibit this weirdness.
Down the rabbit hole
Imagine my amazement when I tried to find /etc/inittab and it was completely missing! My reality reset, I checked again and it was still missing.
At first I thought some critical package (sysvinit in the versions of Ubuntu I know) was somehow missing after the base install. I jumped on the Ubuntu packages site and did a search for 'sysvinit'. Sure enough, there it was but the file list showed no inittab.
Suddenly I felt like I a four year old whose mommy had lost them at the Mall.
Further down the spiral
The file list did however list a bunch of items that referred to something called 'upstart-compat-sysv'. Browsing to this package listed the follwoing description for the package:
This package contains compatibility tasks and utilities that emulate the behaviour of the original sysvinit package, including runlevels, and ensures that the initscripts in /etc/rc*.d are still run.
OK. I faintly hear someone announcing that my mother is looking for me at the Mall's security office.
So if upstart-compat-sysv _emulates_ the original sysvinit, then what has _replaced_ it?
Upstart, show thy face
Google provided me with this link that points to the Ubuntu upstart page which went a long way towards uniting me and my mommy:
Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.
It was originally developed for the Ubuntu distribution, but is intended to be suitable for deployment in all Linux distributions as a replacement for the venerable System-V init.
Sounds even more pervasive than a simple missing iniitab!
Features:
- Tasks and Services are started and stopped by events
- Events are generated as tasks and services are started and stopped
- Events may be received from any other process on the system
- Services may be respawned if they die unexpectedly
- Bi-directional communication with init daemon to discover which jobs are running, why jobs failed, etc.
So, upstart was not only superseding everything we were trying to do with inittab but also changed the way one interacts with scripts via /etc/event.d.
Why reinvent the wheel?
With the Ubuntu decision to move to the 2.6 kernel, and all the hotplug facilities it provides, they were left with several problems in Dapper. The kernel could now completely cope with hardware coming and going but that had a knock-on effect in that there was now no way to guarantee that particular devices were available at particular point in the boot process.
For example: Dapper cannot mount USB disks in /etc/fstab because it is not guaranteed that the block device exists at the point in the mount process where that happens.
Several other reasons are also provided on the Ubuntu site.
At first, the team decided to look at the available alternatives in third party projects such as Solaris SMF, Apple's launchd, the LSB initserv/chkconfig tools and initNG. None of these met the design criteria that the team had set out for themselves and in true OSS style they decided to roll their own.
The design and implementation documentation is pretty clear so I won't repeat it here.
Epilogue
In the end I simply had to add some logic to the custom debs to check if the system was running upstart and do The Right Thing(TM) (using upstart if it was available or falling back to using inittab) based on that. That amounted to dropping a relevant file in /etc/event.d/ for every server that we previously ran from inittab for systems that were using upstart.
Further Reading
