Sunday, July 5, 2009

Installing the MySQL gem on OS X

If you manage your system packages via the MacPorts system you may run into some problems when trying to install the native MySQL driver gem on OS X.

Typically, you'd see output like this when trying to install the gem:

$ sudo gem install mysql
Building native extensions. This could take a while...
ERROR: Error installing mysql:
ERROR: Failed to build gem native extension.

/opt/local/bin/ruby extconf.rb
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... no
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.

Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/opt/local/bin/ruby
--with-mysql-config
--without-mysql-config
--with-mysql-dir
--without-mysql-dir
--with-mysql-include
--without-mysql-include=${mysql-dir}/include
--with-mysql-lib
--without-mysql-lib=${mysql-dir}/lib
--with-mysqlclientlib
--without-mysqlclientlib
--with-mlib
--without-mlib
--with-mysqlclientlib
--without-mysqlclientlib
--with-zlib
--without-zlib
--with-mysqlclientlib
--without-mysqlclientlib
--with-socketlib
--without-socketlib
--with-mysqlclientlib
--without-mysqlclientlib
--with-nsllib
--without-nsllib
--with-mysqlclientlib
--without-mysqlclientlib


Gem files will remain installed in /opt/local/lib/ruby/gems/1.8/gems/mysql-2.7 for inspection.
Results logged to /opt/local/lib/ruby/gems/1.8/gems/mysql-2.7/gem_make.out

Based on those errors the configure process seems to be failing when looking for MySQL libs. The default path to the libs based on extconf.rb seems to be /usr/local which is not where the ports system installs MySQL.

All you need to do is point to the correct mysql_config and let the gem install command line know of this to get things going:

$ sudo gem install mysql -- --with-mysql-config=/opt/local/bin/mysql_config5
Building native extensions. This could take a while...
Successfully installed mysql-2.7
1 gem installed

All's well that ends well.

Wednesday, July 1, 2009

nginx/passenger: Exception RangeError in PhusionPassenger::Railz::ApplicationSpawner (bignum too big to convert into `long')

Nasty.

I am running nginx+passenger on my OS X laptop and after getting the passenger module installed I got the following stacktrace in /opt/local/var/log/nginx/error.log:

*** Exception RangeError in PhusionPassenger::Railz::ApplicationSpawner (bignum too big to convert into `long') (process
56051):
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:363:in `setgid'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:363:in `switch_to_user'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:328:in `lower_privilege'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:250:in
`initialize_server'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:230:in `report_app_init_st
atus'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:237:in
`initialize_server'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:193:in `start_sy
nchronously'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:162:in `start'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:213:in
`start'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:261:in `spawn_rail
s_application'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server_collection.rb:126:i
n `lookup_or_add'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:255:in `spawn_rail
s_application'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server_collection.rb:80:in
`synchronize'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server_collection.rb:79:in
`synchronize'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:254:in `spawn_rail
s_application'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:153:in `spawn_appl
ication'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:286:in `handle_spa
wn_application'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:351:in `__send__
'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:351:in `main_loo
p'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:195:in `start_sy
nchronously'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/bin/passenger-spawn-server:61

So, what's up?

The old switcheroo
Passenger implements User Switching which works around the age old problem that plagues most other scripting languages where they run in the same context as the web server they are accessed through.

The default behaviour is for rails apps to be started as the owner of config/environment.rb. There are however some points that should be noted:

  • The owner of environment.rb must have read access to the rails application folder, and read/write access to the rails application logs folder.

  • This feature is only available if nginx is started by root. This is the case on most nginx installations.

  • Under no circumstances will applications be run as root. If environment.rb is owned as root or by an unknown user, then the rails application will run as the user specified by passenger_default_user.



In my case the project was owned by root so changing that to a local user fixed the problem.


Home made nginx + phusion passenger OS X port

The instructions to do this are similar to those in my previous article.

Install passenger gem
The first step is to install the Phusion Passenger gem:
$ sudo gem install -V -r passenger
[...]
Installing RDoc documentation for passenger-2.2.4...

Take note of where the passenger gem installed the nginx module code (ext/nginx/*) that will be required further down. In this case the install directory will be /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/.

Compile nginx passenger module
You also need to compile the passenger nginx module if it was not already compiled:

$ cd /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/
$ sudo rake nginx
[...]
g++ ext/nginx/HelperServer.cpp -o ext/nginx/HelperServer -Iext -Iext/common -D_REENTRANT -I/usr/local/include -Wall -g -DPASSENGER_DEBUG -DBOOST_DISABLE_ASSERTS ext/nginx/libpassenger_common.a ext/nginx/libboost_oxt.a -lpthread


Modify the nginx Portfile
Ensure you have nginx port installed. Your Portfile will most likely be in /opt/local/var/macports/sources/rsync.macports.org/release/ports/www/nginx/Portfile.

Add the following variant block to the end of the file:

variant passenger description {Enable Phusion Passenger (mod_rails) support} {
configure.args-append --add-module=/opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/nginx
}


Rebuild/install nginx
When you try and reinstall nginx now it will use the updated Portfile and add the passenger module to the mix:

$ sudo port uninstall nginx && sudo port clean nginx
$ sudo port -vvvvv install nginx +passenger


Nginx passenger configuration
To enable and take advantage of the passenger nginx module you need to edit the /opt/local/etc/nginx/nginx.conf config. Here's an example:

#user nobody;
worker_processes 1;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/nginx.pid;


events {
worker_connections 1024;
}


http {
include mime.types;
default_type application/octet-stream;

#log_format main '$remote_addr - $remote_user [$time_local] $request '
# '"$status" $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';

#access_log logs/access.log main;

sendfile on;
#tcp_nopush on;

#keepalive_timeout 0;
keepalive_timeout 65;

gzip on;

# Enable Phusion Passenger nginx module
passenger_root /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4;
passenger_ruby /opt/local/bin/ruby;

server {
listen 80;
server_name foo.example.com;
root /var/www/foo/public; # <--- be sure to point to 'public'!
passenger_enabled on;
}
}

Ensure that the port, server_name and root variables have valid values for your environment.

If you want to use a non-production rails environment you can add the following to the block after passenger_enabled:

rails_env development;

Once done you can restart nginx and you should see the following processes that confirm things are running:

$ ps axw | egrep "nginx|passenger"
55935 ?? Ss 0:00.00 nginx: master process nginx
55936 ?? S 0:00.00 nginx: worker process
55925 s001 S 0:00.01 PassengerNginxHelperServer /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4 /opt/local/bin/ruby 3 4 0 6 0 300 1 nobody 4294967294 4294967294 /tmp/passenger.55923
55938 s001 R+ 0:00.00 egrep nginx|passenger

The final test to see if everything is working is to open a browser and to browse to the server_name and port defined above. Errors will be reported in /opt/local/var/log/nginx/error.log as well as the development/production logs of the rails project you're running.


Home made nginx + phusion passenger Debian/Ubuntu dep

If you are using Ubuntu Hardy then you could try using Brightbox's repo instead of building your own deb.

If you are however running some other distro then the following (non-definitive) instructions are for you.

Warning: it may be safer to install nginx + passenger via the passenger-install-nginx-module script as it will match passenger with a tested version of nginx.

The instructions below simply add passenger support to the latest nginx deb in your distro of choice.

Install passenger gem
The first step is to install the Phusion Passenger gem:
$ sudo gem install -V -r passenger
[...]
Installing RDoc documentation for passenger-2.2.4...

Take note of where the passenger gem installed the nginx module code that will be required further down:
$ gem contents passenger | egrep '/ext/nginx/' | head -n 1
/var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx/Configuration.c

In this case the install directory will be /var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx.

Compile nginx passenger module
You also need to compile the passenger nginx module if it was not already compiled:

$ cd /var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx
$ sudo rake nginx

(in /var/lib/gems/1.8/gems/passenger-2.2.4)
mkdir -p ext/nginx/libboost_oxt/boost
g++ -Iext -D_REENTRANT -I/usr/local/include -Wall -g -DPASSENGER_DEBUG -DBOOST_DISABLE_ASSERTS -o ext/nginx/libboost_oxt/boost/exceptions.o -c ext/boost/src/pthread/exceptions.cpp
[...]
g++ ext/nginx/HelperServer.cpp -o ext/nginx/HelperServer -Iext -Iext/common -D_REENTRANT -I/usr/local/include -Wall -g -DPASSENGER_DEBUG -DBOOST_DISABLE_ASSERTS ext/nginx/libpassenger_common.a ext/nginx/libboost_oxt.a -lpthread


Install build packages

$ sudo aptitude install fakeroot debhelper dpkg-dev autotools-dev libpcre3-dev libssl-dev


Modify APT sources.list
Open /etc/apt/sources.list and ensure you add deb-src lines for each of the universe repos. The easiest way to do this is to simply copy the universe repo lines that exist and change the starting deb to deb-src:

deb http://au.archive.ubuntu.com/ubuntu/ jaunty universe
deb-src http://au.archive.ubuntu.com/ubuntu/ jaunty universe
deb http://au.archive.ubuntu.com/ubuntu/ jaunty-updates universe
deb-src http://au.archive.ubuntu.com/ubuntu/ jaunty-updates universe

Replace 'au.archive.ubuntu.com' with the repo closest to you and also replace 'jaunty' with your distro. Ensure you do a 'sudo aptitude update' to load the latest package indexes.

Set up a packaging environment

$ mkdir -p /var/tmp/nginx-passenger && cd /var/tmp/nginx-passenger
$ apt-get source nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done
NOTICE: 'nginx' packaging is maintained in the 'Svn' version control system at:
svn://svn.debian.org/svn/collab-maint/deb-maint/nginx/trunk
Need to get 537kB of source archives.
Get:1 http://au.archive.ubuntu.com jaunty/universe nginx 0.6.35-0ubuntu1 (dsc) [1305B]
Get:2 http://au.archive.ubuntu.com jaunty/universe nginx 0.6.35-0ubuntu1 (tar) [525kB]
Get:3 http://au.archive.ubuntu.com jaunty/universe nginx 0.6.35-0ubuntu1 (diff) [11.0kB]
Fetched 537kB in 0s (1504kB/s)
gpg: Signature made Sat 21 Feb 2009 07:02:34 AM EST using DSA key ID 3FE63E00
gpg: Can't check signature: public key not found
dpkg-source: extracting nginx in nginx-0.6.35
dpkg-source: info: unpacking nginx_0.6.35.orig.tar.gz
dpkg-source: info: applying nginx_0.6.35-0ubuntu1.diff.gz
$ ls -la
total 548
drwxr-xr-x 3 root root 4096 2009-06-29 08:55 .
drwxrwxrwt 3 root root 4096 2009-06-29 08:39 ..
drwxr-xr-x 8 root root 4096 2009-06-29 08:55 nginx-0.6.35
-rw-r--r-- 1 root root 11018 2009-02-21 08:04 nginx_0.6.35-0ubuntu1.diff.gz
-rw-r--r-- 1 root root 1305 2009-02-21 08:04 nginx_0.6.35-0ubuntu1.dsc
-rw-r--r-- 1 root root 524987 2009-02-21 08:04 nginx_0.6.35.orig.tar.gz
$ rm -f nginx_*
$ mv nginx-0.6.35 nginx-passenger-0.6.35


Package customizations
In order to build our nginx-passenger custom package we need to modify the following packaging resources:

  1. control file

  2. changelog file

  3. rules file

  4. nginx.install file

  5. nginx.logrotate file

  6. passenger.conf file

  7. postinst file



Control file

$ cd nginx-passenger-0.6.35/debian

Edit the control file and change the Source, Maintainer, Uploaders, Package, Provides, Conflicts and Description fields:

Source: nginx-passenger
Maintainer: Charl Matthee
Uploaders: Charl Matthee
Package: nginx-passenger
Provides: httpd, nginx
Conflicts: nginx
Description: small, but very powerful and efficient web server
Nginx (engine x) is a web server created by Igor Sysoev and kindly provided
to the open-source community. This server can be used as standalone HTTP
server and as a reverse proxy server before some Apache or another big
server to reduce load to backend servers by many concurrent HTTP-sessions.
.
This is a custom nginx package that includes Phusion Passenger
support for nginx.

Note: spacing in the control file is significant.

More information on the control file internals can be found here.

Changelog
Add a new changelog entry in the changelog file:

nginx-passenger (0.6.35-0custom1) jaunty; urgency=low

* New custom Passenger Phusion nginx module addition

-- Charl Matthee Mon, 29 Jun 2009 09:08:36 +0100

nginx (0.6.35-0ubuntu1) jaunty; urgency=low

* New upstream bugfix release

-- Daniel Hahler Fri, 20 Feb 2009 21:01:23 +0100
[...]

More information on the changelog file internals can be found here.

Rules
Add the passenger module inclusion to the configure script options when nginx is built by modifying the './configure $(CONFIGURE_OPTS)' block in the rules file like this:

./configure $(CONFIGURE_OPTS) --conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid \
--lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log \
--http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy \
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-http_stub_status_module \
--with-http_flv_module --with-http_ssl_module --with-http_dav_module \
--with-http_realip_module --add-module=PASSENGER_LIB_DIR

Where PASSENGER_LIB_DIR is the path you noted above when you installed the passenger gem. So, the updated block looks like this:

./configure $(CONFIGURE_OPTS) --conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid \
--lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log \
--http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy \
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-http_stub_status_module \
--with-http_flv_module --with-http_ssl_module --with-http_dav_module \
--with-http_realip_module --add-module=/var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx/

More infomration on the changelog file internals can be found here.

nginx.install
Move this file to the correct name based on the package name and add the installation instructions for the default nginx configuration that is required to turn passenger support on:

$ mv nginx.install nginx-passenger.install
$ cat nginx-passenger.install
objs/nginx usr/sbin
html/* var/www/nginx-default
debian/conf/* etc/nginx
debian/passenger.conf etc/nginx/conf.d/
debian/NEWS.Debian usr/share/doc/nginx


nginx.logrotate
Move this file to the correct name based on the package name:

$ mv nginx.logrotate nginx-passenger.logrotate


passenger.conf
Create the following file in the debian directory:

passenger_root /var/lib/gems/1.8/gems/passenger-2.2.4;
passenger_ruby /usr/bin/ruby1.8;

Ensure those paths point to the correct passenger isntall and ruby directories.

postinst
Add the following steps to the postinst script after the '#DEBHELPER#' token:

# Phusion Passenger module settings for nginx
echo
echo "***********************************************************************"
echo "For each virtual host you want to add Phusion Passenger support for you"
echo "need to create a config like this in /etc/nginx/site-available:"
echo
echo " server {
listen 80;
server_name DOMAIN_NAME;
root /var/www/RAILS_PROJECT/public; #<--- be sure to point to 'public'!
passenger_enabled on;
}"
echo
echo "Don't forget to add a symlink from this file(s) to"
echo "/etc/nginx/site-enabled to enable the configuration in nginx."
echo "***********************************************************************"
echo

This will warn administrators that install this package that they still need to do some manual work to get a virtual host set up to take advantage of the passenger module.

Package rebuild
The package can then be rebuilt just like any other custom deb:

$ cd /var/tmp/nginx-passenger/nginx-passenger-0.6.35
$ dpkg-buildpackage -rfakeroot
dpkg-buildpackage: set CFLAGS to default value: -g -O2
[...]
dpkg-deb: building package `nginx-passenger' in `../nginx-passenger_0.6.35-0custom1_i386.deb'.
dpkg-deb: ignoring 1 warnings about the control file(s)
signfile nginx-passenger_0.6.35-0custom1.dsc
gpg: skipped "Charl Matthee ": secret key not available
gpg: [stdin]: clearsign failed: secret key not available

dpkg-genchanges >../nginx-passenger_0.6.35-0custom1_i386.changes
dpkg-genchanges: warning: the current version (0.6.35-0custom1) is smaller than the previous one (0.6.35-0ubuntu1)
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload; Debian-native package (full source is included)
dpkg-buildpackage: warning: Failed to sign .dsc and .changes file

Those signing errors were triggered because I have did not have the required key infrastructure installed. They can be ignored for the moment.

Install the new package:

$ sudo dpkg -i ../nginx-passenger_0.6.35-0custom1_i386.deb
[...]
Setting up nginx-passenger (0.6.35-0custom1) ...

***********************************************************************
Create a config like this for each virtual host you want to add Phusion
Passenger support to. They should be created in
/etc/nginx/site-available:

server {
listen 80;
server_name DOMAIN_NAME;
root /var/www/RAILS/public; # <--- be sure to point to 'public'!
passenger_enabled on;
}

Don't forget to add a symlink from this file(s) to
/etc/nginx/site-enabled to enable the configuration in nginx.
***********************************************************************
[...]


Nginx passenger configuration
To enable and take advantage of the passenger nginx module you need to create a /etc/nginx/conf.d/passenger.conf config:

passenger_root /var/lib/gems/1.8/gems/passenger-2.2.4;
passenger_ruby /usr/bin/ruby1.8;

This will of course have been done when we installed our custom package.

Next, add a virtual host for each of the rails applications you want to support by dropping a file like 'foo' into /etc/nginx/sites-available/:

server {
listen 80;
server_name foo.example.com;
root /var/www/foo/public; # <--- be sure to point to 'public'!
passenger_enabled on;
}

Ensure that the port, server_name and root variables have valid values for your environment.

If you want to use a non-production rails environment you can add the following to the block after passenger_enabled:

rails_env development;


To enable the virtual host(s) you need to ensure that there's a symlink linking the virtual host in sites-available to sites-enabled:

$ sudo ln -s /etc/nginx/sites-available/foo /etc/nginx/sites-enabled/foo
$ ls -la /etc/nginx/sites-enabled
total 8
drwxr-xr-x 2 root root 4096 2009-06-29 12:33 .
drwxr-xr-x 5 root root 4096 2009-06-29 10:51 ..
lrwxrwxrwx 1 root root 34 2009-06-02 07:25 default -> /etc/nginx/sites-available/default
lrwxrwxrwx 1 root root 30 2009-06-29 12:33 foo -> /etc/nginx/sites-available/foo

Once done you can restart nginx and you should see the following processes that confirm things are running:

$ ps axf | egrep "nginx|passenger"
28608 pts/1 S+ 0:00 \_ egrep nginx|passenger
28560 pts/1 Sl 0:00 PassengerNginxHelperServer /var/lib/gems/1.8/gems/passenger-2.2.4 /usr/bin/ruby1.8 3 4 0 6 0 300 1 nobody 33 33 /tmp/passenger.28557
28580 ? Ss 0:00 nginx: master process /usr/sbin/nginx
28581 ? S 0:00 \_ nginx: worker process

The final test to see if everything is working is to open a browser and to browse to the server_name and port defined above. Errors will be reported in /var/log/nginx/error.log as well as the development/production logs of the rails project you're running.


Thursday, June 11, 2009

Commandline PDF Manipulation

Want to do any of the following from a *NIX based system commandline with PDF documents:
  • Merge PDF Documents

  • Split PDF Pages into a New Document

  • Rotate PDF Pages or Documents

  • Decrypt Input as Necessary (Password Required)

  • Encrypt Output as Desired

  • Fill PDF Forms with FDF Data or XFDF Data and/or Flatten Forms

  • Apply a Background Watermark or a Foreground Stamp

  • Report on PDF Metrics such as Metadata, Bookmarks, and Page Labels

  • Update PDF Metadata

  • Attach Files to PDF Pages or the PDF Document

  • Unpack PDF Attachments

  • Burst a PDF Document into Single Pages

  • Uncompress and Re-Compress Page Streams

  • Repair Corrupted PDF

One way to accomplish any of these would be to use pdftk (PDF Toolkit).

Something I need to do quite often is concatenate (join) several PDF documents into one document. This can quite trivially be achieved with the following from the commandline:

$ pdftk 01.pdf 02.pdf cat output all_together_now.pdf

Above we concatenated 01.pdf and 02.pdf to form the new combined all_together_now.pdf document.


Wednesday, April 22, 2009

Broken memcached gem/libmemchached deb on Ubuntu

This post follows on a previous post that was OS X centric.

To fix this issue follow the same steps (starting with the download of libmemcached-0.25.14.tar.gz) detailed here.

Because the source installation of libmemcached dropped the libs in a non-standard place (from a Debian/Ubuntu viewpoint) we need a little more magic before things will work.

As root, create /etc/ld.so.conf.d/libmemcached.conf:
# Manual installation of libmemcached dropped the libs in a non-standard place
/usr/local/lib
Run ldconfig to load the new config and test to ensure your script can now access libmemcached properly through the memcached gem.


Sunday, March 22, 2009

Unpacking a RPM on Ubuntu (or Debian)


# alien --scripts PACKAGE.rpm
# ar -x PACKAGE.deb
You now have two tarballs control.tar.gz and data.tar.gz.

The first is the information dpkg needs to do a proper installation and configuration of the package, the second is contains the binaries and data files.


Monday, February 23, 2009

Broken memcached gem/libmemchached port on OS X

Update (2009/04/22):
This post was originally only targeted at OS X but I have since noticed Ubuntu also having this same issue. Jump to my additional instructions for Ubuntu.

Recipe for disaster:
  1. Use MacPorts for your package management
  2. Use the ruby memcached gem
  3. Use the libmemcached port as a dependency on the gem above
  4. Upgrade your ports
Synopses
I upgraded all my ports on my MBP recently and one of the libs that was upgraded in the process was libmemcached (upgraded from v0.25 to v0.26). Unfortunately this has broken my access to memcached on my system (the script dies when trying to connect).

Not knowing what the problem may be I also upgraded my gem to v0.14 which then could not build. A quick search for the issue found this entry on the memcached discussion forum.

How now brown cow?
Evan Weaver, the maintainer of the memcached project, mentioned that libmemcached v0.26 lacks some critical patch and suggested people may have to wait for v0.27 to rectify things.

For those who cannot wait and have inconsistent, broken systems, Evan goes on to mention that you should currently use the libmemcached lib and memcached gem pair from here.

Here he also provides a convenient compatibility matrix so you can see how to align the different versions of libmemcached and the memcached gem.

Fix
Unfortunately this messes with things because libmemcached now needs to be installed as a source package in a system managed with MacPorts.

Yuck!

Unfortunately the choices are limited as the versions of libmemcached that ships as a port are either v0.26 (the latest, but broken for our purposes) or v0.25 (which aligns with the older v0.13 of the memcached gem).

First off remove any libmemcached ports you may have installed:
$ sudo port uninstall libmemcached
Download the libmemcached source archive, extract, build and install it:
ibm-99tvvxc:msp charl$ cd /var/tmp/
$ wget http://blog.evanweaver.com/files/libmemcached-0.25.14.tar.gz
--2009-02-24 10:50:40-- http://blog.evanweaver.com/files/libmemcached-0.25.14.tar.gz
Resolving blog.evanweaver.com... 208.78.102.192
Connecting to blog.evanweaver.com|208.78.102.192|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 428047 (418K) [application/x-gzip]
Saving to: `libmemcached-0.25.14.tar.gz'

100%[==========================================================>] 428,047 135K/s in 3.1s

2009-02-24 10:50:49 (135 KB/s) - `libmemcached-0.25.14.tar.gz' saved [428047/428047]

$ tar zxvf libmemcached-0.25.14.tar.gz
libmemcached-0.25.14/
[...]
$ cd libmemcached-0.25.14
$ ./configure
checking build system type... i386-apple-darwin9.6.0
checking host system type... i386-apple-darwin9.6.0
checking target system type... i386-apple-darwin9.6.0
checking for a BSD-compatible install... /usr/bin/install -c
[...]
$ make
Making all in docs
/usr/bin/pod2man -c "libmemcached" -r "" -s 3 libmemcached.pod > libmemcached.3
/usr/bin/pod2man -c "libmemcached" -r "" -s 3 libmemcached_examples.pod > libmemcached_examples.3
[...]
$ sudo make install
Password:
Making install in docs
make[2]: Nothing to be done for `install-exec-am'.
test -z "/usr/local/share/man/man1" || ../config/install-sh -c -d "/usr/local/share/man/man1"
/usr/bin/install -c -m 644 './memcat.1' '/usr/local/share/man/man1/memcat.1'
[...]
You may want to keep the source directory around so that you can easily uninstall (run sudo make uninstall in the source directory) the source installation later when the relevant aligned packaged versions are available for installation.

The final step is to install the gem:
$ sudo env ARCHFLAGS="-arch i386" gem install -r -V memcached --no-rdoc --no-ri
GET 200 OK: http://gems.rubyforge.org/latest_specs.4.8.gz
GET 200 OK: http://gems.github.com/latest_specs.4.8.gz
connection reset after 2 requests, retrying
GET 200 OK: http://gems.rubyforge.org/quick/Marshal.4.8/memcached-0.14.gemspec.rz
Installing gem memcached-0.14
/opt/local/lib/ruby/gems/1.8/gems/memcached-0.14/BENCHMARKS
/opt/local/lib/ruby/gems/1.8/gems/memcached-0.14/CHANGELOG
[...]
Successfully installed memcached-0.14
1 gem installed
Success!

We now have libmemcached v0.25.14 and the memcached v0.14 gem installed and we can go on with our lives.


About Me

My photo
I love solving real-world problems with code and systems (web apps, distributed systems and all the bits and pieces in-between).