Showing posts with label debian. Show all posts
Showing posts with label debian. Show all posts

Wednesday, July 1, 2009

Home made nginx + phusion passenger Debian/Ubuntu dep

If you are using Ubuntu Hardy then you could try using Brightbox's repo instead of building your own deb.

If you are however running some other distro then the following (non-definitive) instructions are for you.

Warning: it may be safer to install nginx + passenger via the passenger-install-nginx-module script as it will match passenger with a tested version of nginx.

The instructions below simply add passenger support to the latest nginx deb in your distro of choice.

Install passenger gem
The first step is to install the Phusion Passenger gem:
$ sudo gem install -V -r passenger
[...]
Installing RDoc documentation for passenger-2.2.4...

Take note of where the passenger gem installed the nginx module code that will be required further down:
$ gem contents passenger | egrep '/ext/nginx/' | head -n 1
/var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx/Configuration.c

In this case the install directory will be /var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx.

Compile nginx passenger module
You also need to compile the passenger nginx module if it was not already compiled:

$ cd /var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx
$ sudo rake nginx

(in /var/lib/gems/1.8/gems/passenger-2.2.4)
mkdir -p ext/nginx/libboost_oxt/boost
g++ -Iext -D_REENTRANT -I/usr/local/include -Wall -g -DPASSENGER_DEBUG -DBOOST_DISABLE_ASSERTS -o ext/nginx/libboost_oxt/boost/exceptions.o -c ext/boost/src/pthread/exceptions.cpp
[...]
g++ ext/nginx/HelperServer.cpp -o ext/nginx/HelperServer -Iext -Iext/common -D_REENTRANT -I/usr/local/include -Wall -g -DPASSENGER_DEBUG -DBOOST_DISABLE_ASSERTS ext/nginx/libpassenger_common.a ext/nginx/libboost_oxt.a -lpthread


Install build packages

$ sudo aptitude install fakeroot debhelper dpkg-dev autotools-dev libpcre3-dev libssl-dev


Modify APT sources.list
Open /etc/apt/sources.list and ensure you add deb-src lines for each of the universe repos. The easiest way to do this is to simply copy the universe repo lines that exist and change the starting deb to deb-src:

deb http://au.archive.ubuntu.com/ubuntu/ jaunty universe
deb-src http://au.archive.ubuntu.com/ubuntu/ jaunty universe
deb http://au.archive.ubuntu.com/ubuntu/ jaunty-updates universe
deb-src http://au.archive.ubuntu.com/ubuntu/ jaunty-updates universe

Replace 'au.archive.ubuntu.com' with the repo closest to you and also replace 'jaunty' with your distro. Ensure you do a 'sudo aptitude update' to load the latest package indexes.

Set up a packaging environment

$ mkdir -p /var/tmp/nginx-passenger && cd /var/tmp/nginx-passenger
$ apt-get source nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done
NOTICE: 'nginx' packaging is maintained in the 'Svn' version control system at:
svn://svn.debian.org/svn/collab-maint/deb-maint/nginx/trunk
Need to get 537kB of source archives.
Get:1 http://au.archive.ubuntu.com jaunty/universe nginx 0.6.35-0ubuntu1 (dsc) [1305B]
Get:2 http://au.archive.ubuntu.com jaunty/universe nginx 0.6.35-0ubuntu1 (tar) [525kB]
Get:3 http://au.archive.ubuntu.com jaunty/universe nginx 0.6.35-0ubuntu1 (diff) [11.0kB]
Fetched 537kB in 0s (1504kB/s)
gpg: Signature made Sat 21 Feb 2009 07:02:34 AM EST using DSA key ID 3FE63E00
gpg: Can't check signature: public key not found
dpkg-source: extracting nginx in nginx-0.6.35
dpkg-source: info: unpacking nginx_0.6.35.orig.tar.gz
dpkg-source: info: applying nginx_0.6.35-0ubuntu1.diff.gz
$ ls -la
total 548
drwxr-xr-x 3 root root 4096 2009-06-29 08:55 .
drwxrwxrwt 3 root root 4096 2009-06-29 08:39 ..
drwxr-xr-x 8 root root 4096 2009-06-29 08:55 nginx-0.6.35
-rw-r--r-- 1 root root 11018 2009-02-21 08:04 nginx_0.6.35-0ubuntu1.diff.gz
-rw-r--r-- 1 root root 1305 2009-02-21 08:04 nginx_0.6.35-0ubuntu1.dsc
-rw-r--r-- 1 root root 524987 2009-02-21 08:04 nginx_0.6.35.orig.tar.gz
$ rm -f nginx_*
$ mv nginx-0.6.35 nginx-passenger-0.6.35


Package customizations
In order to build our nginx-passenger custom package we need to modify the following packaging resources:

  1. control file

  2. changelog file

  3. rules file

  4. nginx.install file

  5. nginx.logrotate file

  6. passenger.conf file

  7. postinst file



Control file

$ cd nginx-passenger-0.6.35/debian

Edit the control file and change the Source, Maintainer, Uploaders, Package, Provides, Conflicts and Description fields:

Source: nginx-passenger
Maintainer: Charl Matthee
Uploaders: Charl Matthee
Package: nginx-passenger
Provides: httpd, nginx
Conflicts: nginx
Description: small, but very powerful and efficient web server
Nginx (engine x) is a web server created by Igor Sysoev and kindly provided
to the open-source community. This server can be used as standalone HTTP
server and as a reverse proxy server before some Apache or another big
server to reduce load to backend servers by many concurrent HTTP-sessions.
.
This is a custom nginx package that includes Phusion Passenger
support for nginx.

Note: spacing in the control file is significant.

More information on the control file internals can be found here.

Changelog
Add a new changelog entry in the changelog file:

nginx-passenger (0.6.35-0custom1) jaunty; urgency=low

* New custom Passenger Phusion nginx module addition

-- Charl Matthee Mon, 29 Jun 2009 09:08:36 +0100

nginx (0.6.35-0ubuntu1) jaunty; urgency=low

* New upstream bugfix release

-- Daniel Hahler Fri, 20 Feb 2009 21:01:23 +0100
[...]

More information on the changelog file internals can be found here.

Rules
Add the passenger module inclusion to the configure script options when nginx is built by modifying the './configure $(CONFIGURE_OPTS)' block in the rules file like this:

./configure $(CONFIGURE_OPTS) --conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid \
--lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log \
--http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy \
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-http_stub_status_module \
--with-http_flv_module --with-http_ssl_module --with-http_dav_module \
--with-http_realip_module --add-module=PASSENGER_LIB_DIR

Where PASSENGER_LIB_DIR is the path you noted above when you installed the passenger gem. So, the updated block looks like this:

./configure $(CONFIGURE_OPTS) --conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid \
--lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log \
--http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy \
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-http_stub_status_module \
--with-http_flv_module --with-http_ssl_module --with-http_dav_module \
--with-http_realip_module --add-module=/var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx/

More infomration on the changelog file internals can be found here.

nginx.install
Move this file to the correct name based on the package name and add the installation instructions for the default nginx configuration that is required to turn passenger support on:

$ mv nginx.install nginx-passenger.install
$ cat nginx-passenger.install
objs/nginx usr/sbin
html/* var/www/nginx-default
debian/conf/* etc/nginx
debian/passenger.conf etc/nginx/conf.d/
debian/NEWS.Debian usr/share/doc/nginx


nginx.logrotate
Move this file to the correct name based on the package name:

$ mv nginx.logrotate nginx-passenger.logrotate


passenger.conf
Create the following file in the debian directory:

passenger_root /var/lib/gems/1.8/gems/passenger-2.2.4;
passenger_ruby /usr/bin/ruby1.8;

Ensure those paths point to the correct passenger isntall and ruby directories.

postinst
Add the following steps to the postinst script after the '#DEBHELPER#' token:

# Phusion Passenger module settings for nginx
echo
echo "***********************************************************************"
echo "For each virtual host you want to add Phusion Passenger support for you"
echo "need to create a config like this in /etc/nginx/site-available:"
echo
echo " server {
listen 80;
server_name DOMAIN_NAME;
root /var/www/RAILS_PROJECT/public; #<--- be sure to point to 'public'!
passenger_enabled on;
}"
echo
echo "Don't forget to add a symlink from this file(s) to"
echo "/etc/nginx/site-enabled to enable the configuration in nginx."
echo "***********************************************************************"
echo

This will warn administrators that install this package that they still need to do some manual work to get a virtual host set up to take advantage of the passenger module.

Package rebuild
The package can then be rebuilt just like any other custom deb:

$ cd /var/tmp/nginx-passenger/nginx-passenger-0.6.35
$ dpkg-buildpackage -rfakeroot
dpkg-buildpackage: set CFLAGS to default value: -g -O2
[...]
dpkg-deb: building package `nginx-passenger' in `../nginx-passenger_0.6.35-0custom1_i386.deb'.
dpkg-deb: ignoring 1 warnings about the control file(s)
signfile nginx-passenger_0.6.35-0custom1.dsc
gpg: skipped "Charl Matthee ": secret key not available
gpg: [stdin]: clearsign failed: secret key not available

dpkg-genchanges >../nginx-passenger_0.6.35-0custom1_i386.changes
dpkg-genchanges: warning: the current version (0.6.35-0custom1) is smaller than the previous one (0.6.35-0ubuntu1)
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload; Debian-native package (full source is included)
dpkg-buildpackage: warning: Failed to sign .dsc and .changes file

Those signing errors were triggered because I have did not have the required key infrastructure installed. They can be ignored for the moment.

Install the new package:

$ sudo dpkg -i ../nginx-passenger_0.6.35-0custom1_i386.deb
[...]
Setting up nginx-passenger (0.6.35-0custom1) ...

***********************************************************************
Create a config like this for each virtual host you want to add Phusion
Passenger support to. They should be created in
/etc/nginx/site-available:

server {
listen 80;
server_name DOMAIN_NAME;
root /var/www/RAILS/public; # <--- be sure to point to 'public'!
passenger_enabled on;
}

Don't forget to add a symlink from this file(s) to
/etc/nginx/site-enabled to enable the configuration in nginx.
***********************************************************************
[...]


Nginx passenger configuration
To enable and take advantage of the passenger nginx module you need to create a /etc/nginx/conf.d/passenger.conf config:

passenger_root /var/lib/gems/1.8/gems/passenger-2.2.4;
passenger_ruby /usr/bin/ruby1.8;

This will of course have been done when we installed our custom package.

Next, add a virtual host for each of the rails applications you want to support by dropping a file like 'foo' into /etc/nginx/sites-available/:

server {
listen 80;
server_name foo.example.com;
root /var/www/foo/public; # <--- be sure to point to 'public'!
passenger_enabled on;
}

Ensure that the port, server_name and root variables have valid values for your environment.

If you want to use a non-production rails environment you can add the following to the block after passenger_enabled:

rails_env development;


To enable the virtual host(s) you need to ensure that there's a symlink linking the virtual host in sites-available to sites-enabled:

$ sudo ln -s /etc/nginx/sites-available/foo /etc/nginx/sites-enabled/foo
$ ls -la /etc/nginx/sites-enabled
total 8
drwxr-xr-x 2 root root 4096 2009-06-29 12:33 .
drwxr-xr-x 5 root root 4096 2009-06-29 10:51 ..
lrwxrwxrwx 1 root root 34 2009-06-02 07:25 default -> /etc/nginx/sites-available/default
lrwxrwxrwx 1 root root 30 2009-06-29 12:33 foo -> /etc/nginx/sites-available/foo

Once done you can restart nginx and you should see the following processes that confirm things are running:

$ ps axf | egrep "nginx|passenger"
28608 pts/1 S+ 0:00 \_ egrep nginx|passenger
28560 pts/1 Sl 0:00 PassengerNginxHelperServer /var/lib/gems/1.8/gems/passenger-2.2.4 /usr/bin/ruby1.8 3 4 0 6 0 300 1 nobody 33 33 /tmp/passenger.28557
28580 ? Ss 0:00 nginx: master process /usr/sbin/nginx
28581 ? S 0:00 \_ nginx: worker process

The final test to see if everything is working is to open a browser and to browse to the server_name and port defined above. Errors will be reported in /var/log/nginx/error.log as well as the development/production logs of the rails project you're running.


Wednesday, April 22, 2009

Broken memcached gem/libmemchached deb on Ubuntu

This post follows on a previous post that was OS X centric.

To fix this issue follow the same steps (starting with the download of libmemcached-0.25.14.tar.gz) detailed here.

Because the source installation of libmemcached dropped the libs in a non-standard place (from a Debian/Ubuntu viewpoint) we need a little more magic before things will work.

As root, create /etc/ld.so.conf.d/libmemcached.conf:
# Manual installation of libmemcached dropped the libs in a non-standard place
/usr/local/lib
Run ldconfig to load the new config and test to ensure your script can now access libmemcached properly through the memcached gem.


Friday, May 30, 2008

warning: SASL authentication failure: cannot connect to saslauthd server: No such file or directory

There's nothing quite like being in a complete coding frenzy, communicating with your customers to get feedback on critical bugs, and your mail server going to SMTP heaven on you.

The Setup
My MTA (postfix) is set up to do secure SMTP-AUTH and TLS via Cyrus SASL library (a.k.a. saslauthd via the sasl2-bin package) on an Ubuntu box.

Postfix SASL support (RFC 4954, formerly RFC 2554) is used to authenticate remote SMTP clients to the MTA and the Postfix SMTP client to a remote SMTP server.

The Error
I originally set things up via the Postfix-SMTP-AUTH-TLS-Howto and everything was working fine until earlier today when I started seeing the following log entries when trying to send mail vi the MTA:
May 30 03:03:36 pyxidis postfix/smtpd[2840]: connect from unknown[x.x.x.x]
May 30 03:03:37 pyxidis postfix/smtpd[2840]: setting up TLS connection from unknown[x.x.x.x]
May 30 03:03:40 pyxidis postfix/smtpd[2840]: Anonymous TLS connection established from unknown[x.x.x.x]: TLSv1 with cipher AES128-SHA (128/128 bits)
May 30 03:03:40 pyxidis postfix/smtpd[2840]: warning: SASL authentication failure: cannot connect to saslauthd server: No such file or directory
May 30 03:03:40 pyxidis postfix/smtpd[2840]: warning: SASL authentication failure: Password verification failed
May 30 03:03:40 pyxidis postfix/smtpd[2840]: warning: unknown[x.x.x.x]: SASL PLAIN authentication failed: generic failure
May 30 03:03:46 pyxidis postfix/smtpd[2840]: lost connection after AUTH from unknown[x.x.x.x]
May 30 03:03:46 pyxidis postfix/smtpd[2840]: disconnect from unknown[x.x.x.x]
The Solution
I checked and the saslauthd process was happily running. Next up I had a peek in /var/spool/postfix/var/run/saslauthd/ (which I had previously created as per the HOWTO above) but there were no *mux* files to be seen as there should have been.

I then dawned on me that postfix runs in a chrooted jail and that saslauthd for some reason had stopped writing the required info to the chrooted jail where postfix was running. A quick look at the saslauthd rc script and its default file showed that it no longer had the required config to do this properly.

Why? Dunno. I'll have to go do some snooping a little later.

For now thought the fix was as simple as modifying the OPTIONS variable in the /etc/defaults/saslauthd config file to be something like this:
OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"
Restart saslauthd and things start appearing where they should and mail is back in business.


Sunday, July 29, 2007

Tracking fast-paced packages on debian based systems (aka debian-volatile project)

debian-volatile
If you run some ISP services (your own mail server with virus and/or spam scanning tools) you will have run into the age old problem that the scanning tools in the stable distribution do not evolve as fast as they should to keep up with their fast-paced projects.

Even continual updates of the software in your distribution are not enough to stay up to date as the release cycle of the stable distribution is out of sync with the speed at which things change in the wild.

According to the debian-volatile project page:

The main goal of volatile is allowing system administrators to update their systems
in a nice, consistent way, without getting the drawbacks of using unstable, even
without getting the drawbacks for the selected packages. So debian-volatile will
only contain changes to stable programs that are necessary to keep them functional.

volatile-sloppy
Great effort goes into ensuring that no functional changes are made to packages in debian-volatile (so that configuration file changes, etc. are not required) for painless upgrades. Unfortunately painful upgrades are not always avoidable so a volatile-sloppy section was created to contain packages that are fast-paced but also require some functional change to how it runs, is installed or configured.

Security
You should note that the debian-volatile project is not supported by the _official_ security team. This responsibility falls to the debian-volatile team who currently has at least one member that is shared with the official debian testing security team.

How do I use it?
Add the relevant repository (volatile and/or volatile-sloppy) to your /etc/apt/sources.list file:

Sarge
deb http://volatile.debian.org/debian-volatile sarge/volatile main contrib non-free
deb http://volatile.debian.org/debian-volatile sarge/volatile-sloppy main contrib non-free

Etch
deb http://volatile.debian.org/debian-volatile etch/volatile main contrib non-free
deb http://volatile.debian.org/debian-volatile etch/volatile-sloppy main contrib non-free


Save sources.list and run _atp-get update_ which should generate something like this (your listing will vary depending on the repositories you have listed in your sources file):

# apt-get update
Get:1 http://archive.ubuntu.com dapper Release.gpg [189B]
Get:2 http://us.archive.ubuntu.com dapper Release.gpg [189B]
Get:3 http://us.archive.ubuntu.com dapper-backports Release.gpg [191B]
Get:4 http://archive.ubuntu.com dapper-updates Release.gpg [191B]
Get:5 http://volatile.debian.org etch/volatile Release.gpg [189B]
Hit http://us.archive.ubuntu.com dapper Release
Hit http://archive.ubuntu.com dapper Release
Get:6 http://volatile.debian.org etch/volatile Release [40.7kB]
Hit http://archive.ubuntu.com dapper-updates Release
Hit http://us.archive.ubuntu.com dapper-backports Release
Hit http://archive.ubuntu.com dapper/main Packages
Hit http://archive.ubuntu.com dapper/restricted Packages
Hit http://us.archive.ubuntu.com dapper/universe Packages
Hit http://us.archive.ubuntu.com dapper/universe Sources
Hit http://archive.ubuntu.com dapper/main Sources
Hit http://archive.ubuntu.com dapper/restricted Sources
Hit http://archive.ubuntu.com dapper-updates/main Packages
Hit http://archive.ubuntu.com dapper-updates/restricted Packages
Hit http://us.archive.ubuntu.com dapper-backports/main Packages
Hit http://us.archive.ubuntu.com dapper-backports/restricted Packages
Hit http://us.archive.ubuntu.com dapper-backports/universe Packages
Hit http://archive.ubuntu.com dapper-updates/main Sources
Hit http://archive.ubuntu.com dapper-updates/restricted Sources
Hit http://us.archive.ubuntu.com dapper-backports/multiverse Packages
Hit http://us.archive.ubuntu.com dapper-backports/main Sources
Hit http://us.archive.ubuntu.com dapper-backports/restricted Sources
Hit http://us.archive.ubuntu.com dapper-backports/universe Sources
Hit http://us.archive.ubuntu.com dapper-backports/multiverse Sources
Ign http://volatile.debian.org etch/volatile Release
Get:7 http://volatile.debian.org etch/volatile/main Packages [3953B]
Hit http://volatile.debian.org etch/volatile/contrib Packages
Hit http://volatile.debian.org etch/volatile/non-free Packages
Get:8 http://security.ubuntu.com dapper-security Release.gpg [191B]
Hit http://security.ubuntu.com dapper-security Release
Hit http://security.ubuntu.com dapper-security/main Packages
Hit http://security.ubuntu.com dapper-security/restricted Packages
Hit http://security.ubuntu.com dapper-security/main Sources
Hit http://security.ubuntu.com dapper-security/restricted Sources
Hit http://security.ubuntu.com dapper-security/universe Packages
Hit http://security.ubuntu.com dapper-security/universe Sources
Fetched 44.8kB in 5s (7554B/s)
Reading package lists... Done
W: GPG error: http://volatile.debian.org etch/volatile Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EC61E0B0BBE55AB3
W: You may want to run apt-get update to correct these problems
#

The inclusion of the debian-volatile release fails because we do not have a key to authenticate the repository. Adding the following will import their key (mentioned as EC61E0B0BBE55AB3 above) into your key ring:

# gpg --keyserver subkeys.pgp.net --recv-keys EC61E0B0BBE55AB3
gpg: directory `/root/.gnupg' created
gpg: new configuration file `/root/.gnupg/gpg.conf' created
gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/root/.gnupg/secring.gpg' created
gpg: keyring `/root/.gnupg/pubring.gpg' created
gpg: requesting key BBE55AB3 from hkp server subkeys.pgp.net
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key BBE55AB3: public key "Debian-Volatile Archive Automatic Signing Key (4.0/etch)" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg: imported: 1
# gpg --armor --export EC61E0B0BBE55AB3 | apt-key add -
gpg: no ultimately trusted keys found
OK
#

Another spin of _apt-get update_ (and possibly _apt-get upgrade_ if their are any outdated packages) should then do the trick.

About Me

My photo
I love solving real-world problems with code and systems (web apps, distributed systems and all the bits and pieces in-between).