Sunday, February 22, 2009

Silly Rails Environment Optimisation

Do you find supplying the ./script base directory for any of the commands you want to run in it annoying?

Why not practice some Convention over Configuration and assume these commands will always be run from within a valid rails project directory.

Based on this assumption you can drop the following into your .bash_profile (OS X) or some system-wide profile to facilitate some laziness:
# Rails aliases
alias about="./script/about"
alias console="./script/console"
alias dbconsole="./script/dbconsole"
alias destroy="./script/destroy"
alias generate="./script/generate"
alias performance="./script/performance"
alias plugin="./script/plugin"
alias process="./script/process"
alias runner="./script/runner"
alias server="./script/server"
Don't forget to load it into you current shell with:
$ . ~/.bash_profile
Now you can simply run 'console', 'dbconsole', 'server', etc. when you're in a valid rails project directory without having to append that annoying base directory.

Caveat: If similar commands exist on your system or are introduced by a non-rails package a conflict will arise and the alias version of the command will win. Also, TAB-completion now no longer works for these commands and you'll have to type them out in their entirety.


Friday, February 20, 2009

What's next with acts_as_state_machine?

I've been using acts_as_state_machine and there seems to be at least two convenience methods missing that I find quite useful:
  1. next_state - what state will we be transitioning to
  2. next_states - list of states that we still need to transition to
Drop the following in a suitable place (your model using acts_as_state_machine will do) and abuse it to your hearts content:
# Monkey patch acts_as_state_machine to allow us to see what the next states are
# for the current state
module ScottBarron
module Acts
module StateMachine
module InstanceMethods
def next_state(state=nil)
state ||= current_state()
self.class.read_inheritable_attribute(:transition_table).each_value do |event|
event.each do |transition|
return transition.to if transition.from == state
end
end
nil
end

def next_states(state=nil)
state ||= current_state()
states = []
while state = next_state(state)
states << state
end
states
end
end
end
end
end
Thanks to Lourens Naude's' post on acts_as_state_machine for providing me with some ideas and to Scott Barron for the great acts_as_state_machine plugin.


Wednesday, February 11, 2009

Signing your own gems

Chapter 9 (Signing Your Gems) of the RubyGems User Guide contains everything you need to know to do this.

Here's a quick overview of the steps required.

Build a certificate and private key pair
$ cd /to/where/you/wish/to/keep/your/cert_and_key
$ gem cert --build gemmaster@example.com
Two files are generated from this: gem-private_key.pem & gem-public_cert.pem.

Ensure you keep gem-private_key.pem somewhere safe that only you have access to.

Modify your build specification
Add the following spec to your existing gem's gemspec or to the relevant Rakefile ("s" is your Gem::Specification instance):
$ s.signing_key = 'gem-private_key.pem'
$ s.cert_chain = ['gem-public_cert.pem']
Rebuild your package and you're done!


Your own gem repository

Setting up your own repository is quite simple. All you need is the following:
  • Web server
  • Builder gem
  • A repository
  • Client setup
Web server
I used nginx but you should be able to tailor this to your needs without too much effort.

Create a directory to contain your local gem repository:
$ sudo mkdir -p /var/www/gems-repo/gems
Note: to get everything working correctly you need to ensure you have that /gems subdirectory off your base directory configured in nginx. You can find more info on this at the RubyGems FAQ.

Add a virtual host config in /etc/nginx/sites-available for your gems repository (I'm going to assume gems.example.com):
$ cat /etc/nginx/sites-available/gems.example.com
server {
listen 80;
server_name gems.example.com;
access_log /var/log/nginx/gems.example.com.access.log;
root /var/www/gems-repo;
}
Ensure you drop a symlink in /etc/nginx/sites-enabled/ for your virtual host and restart nginx.

Install the builder gem
$ sudo gem install builder
Build your repository
Jump into the base directory defined above (/var/www/gems-repo). Copy all your own gems into /var/www/gems-repo/gems/. Then generate the relevant resource files for rubygems:
$ sudo gem generate_index -d /var/www/gems-repo
Loading 1 gems from /var/www/gems-repo
.
Loaded all gems
Generating quick index gemspecs for 1 gems
.
Complete
Generating specs index
Generating latest specs index
Generating quick index
Generating latest index
Generating Marshal master index
Generating YAML master index for 1 gems (this may take a while)
.
Complete
Compressing indicies
You would re-run this every time you add to or make changes to the repository.

Client setup
Finally run the following on all local clients that need access to the repository:
$ sudo gem sources -a http://gems.example.com/
You can ensure the extra source was added by running 'gem sources'. If you now do a search for you gem that you added above you should see it listed as a target.


Tuesday, January 27, 2009

Stunneling (SSL Tunnels) MySQL on Ubuntu

We have really restrictive IT policies at my current employer which dictates that no client/server communication can happen in the clear. This has some interesting implications as most software servers do not support encryption or encrypted tunnels in any way.

What are the options?
  • Write our own software that supports client/server encryption
  • Only use existing applications that support client/server encryption
  • Treat the encryption layer as an abstraction that the client/server is not aware of

The first two options have obvious drawbacks in terms of time and choice. The third seems to be the best way to transparently insert encryption into an existing architecture without requiring massive changes.

This approach can be done by:

  • SSH tunnels
  • OpenVPN tunnels
  • Stunnel

There may be other options but these are the main ones I have had experience with. SSH tunnels are very convenient but tend to be unstable requiring restarts every now and again.

OpenVPN has also provided very little stability requiring frequent restarts.

Lets explore the setup of the third option then, stunnel.

Stunneling
The stunnel home page describes stunnel as:

Stunnel is a program that allows you to encrypt arbitrary TCP connections inside
SSL (Secure Sockets Layer) available on both Unix and Windows. Stunnel can allow
you to secure non-SSL aware daemons and protocols (like POP, IMAP, LDAP, etc) by
having Stunnel provide the encryption, requiring no changes to the daemon's code.

Sounds promising. They have a really comprehensive examples section on their site which I highly recommend you have a look at.

In my case I needed to get a MySQL master/slave pair to replicate via a secure channel. MySQL seems to support SSL but the package requires a rebuild with the required options which would mean I need a non-standard MySQL package which makes me itchy. I also needed a similar solution for several other applications that need secure communications channels so I've decided on this solution throughout my architecture.

See this Ubuntu thread for more info on rebuilding the MySQL package with SSL support if you're so inclined.

Our installation targets are an Ubuntu servers and will therefore be flavored accordingly but you should be able to easily adapt anything mentioned here for your platform of choice.

Preamble
If you are going to be using this for the same purposes I am ensure that you have a working MySQL master/slave setup. You don't want to troubleshoot multiple applications if something is not playing nice.

Installation
A simple aptitude install stunnel4 on the master and slave should do the trick.

Now, keep in mind that if one of the nodes you have stunnel installed on is acting as both a stunnel client and stunnel server you need to have at least one stunnel server running for the client entries in a config file and one for the server entries in a config file.

Not heading this warning will result in you seeing errors like this on the client:

2009.01.28 07:02:16 LOG7[11508:3082910608]: SSL state (connect): before/connect initialization
2009.01.28 07:02:16 LOG7[11508:3082910608]: SSL state (connect): SSLv3 write client hello A
2009.01.28 07:02:16 LOG7[11508:3082910608]: SSL alert (write): fatal: handshake failure
2009.01.28 07:02:16 LOG3[11508:3082910608]: SSL_connect: 1408F10B: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
2009.01.28 07:02:16 LOG5[11508:3082910608]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket

And this on the server:

2009.01.28 07:10:06 LOG7[13426:3082861456]: SSL state (connect): before/connect initialization
2009.01.28 07:10:06 LOG7[13426:3082861456]: SSL state (connect): SSLv3 write client hello A
2009.01.28 07:10:06 LOG7[13426:3082861456]: SSL alert (write): fatal: handshake failure
2009.01.28 07:10:06 LOG3[13426:3082861456]: SSL_connect: 1408F10B: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
2009.01.28 07:10:06 LOG5[13426:3082861456]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket

My convention is to use the default config file (/etc/stunnel/stunnel.conf) that ships with the stunnel4 package for the client configuration and a modified copy of that as a config (/etc/stunnel/mysql-master.conf, in this specific case) for the server.

If you will not be using the MySQL master server as a stunnel client then you can simply ensure that your /etc/stunnel/stunnel.conf is the same as my /etc/stunnel/mysql-master.conf.

Keep in mind that multiple config files in /etc/stunnel/ will cause multiple stunnel servers to be run with each of the files found there.

Server
First off, let's generate our own key (/etc/ssl/master.pem) that we'll be using for encryption. This key must be treated as a secure file that should have the relevant access controls placed on it.

root@master:~# cd /etc/ssl/
root@master:~# openssl req -new -x509 -nodes -days 730 -out master.pem -keyout master.pem
root@master:~# chmod 600 master.pem
root@master:~# dd if=/dev/urandom of=temp_file count=2
root@master:~# openssl dhparam -rand temp_file 512 >> master.pem
root@master:~# ln -sf master.pem `openssl x509 -noout -hash < coretools01.pem`.0

Now, change the following lines in /etc/stunnel/mysql-master.conf:

cert = /etc/ssl/master.pem
;client = yes
debug = 7
output = /var/log/stunnel4/stunnel.log

[mysql-master]
accept = 443
connect = 127.0.0.1:3306
TIMEOUTclose = 0

This config will ensure we're running as a stunnel server, listening for incoming SSL connections on 0.0.0.0:443 and shunting the unencrypted connection to 127.0.0.1:3306 (ensure your MySQl master is listening for connections here).

Fire stunnel up on the master with /etc/init.d/stunnel4 start && tail -f /var/log/stunnel4/stunnel.log.

Client
Just modify the default /etc/stunnel/stunnel.conf that was installed via the package by changing the lines below:

;cert = /etc/stunnel/mail.pem
;key = /etc/stunnel/mail.pem
debug = 7
output = /var/log/stunnel4/stunnel.log
client = yes

[mysql-master]
accept = 127.0.0.1:3307
connect = master.example.com:443
TIMEOUTclose = 0

This sets up the MySQL slave machine as a stunnel client, forwarding all unencrypted connections on 127.0.0.1:3307 via SSL to master.example.com:443. Just replace master.example.com with the correct FQDN for your master server. You also need to ensure that your MySQL slave's config points to 127.0.0.1:3307 for its replication requirements.

Fire stunnel up on the slave with /etc/init.d/stunnel4 start && tail -f /var/log/stunnel4/stunnel.log.

Testing
An initial test can be done from the slave with the mysql command line utility in a separate terminal session by connecting to a suitable user/db configured on the master with access from 'localhost' or '127.0.0.1':

root@slave:~# mysql -u replication -h 127.0.0.1 -P 3307 -p test
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 21
Server version: 5.0.51a-3ubuntu5.4-log (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

Yay!

Now, apply your general method for determining if your master and slave is in sync. You can use the logs at /var/log/stunnel4/stunnel.log if you hit any snags. Once you're happy with everything ensure you disable the debugging in the stunnel config files.



Monday, December 15, 2008

drb.rb:852:in `initialize': getaddrinfo: nodename nor servname provided, or not known (SocketError) (aka DRb TCPServer.open(0) failure on OS X)


Update (2009/01/01):
Problem TT moved to redmine.
Update (2009/05/13): TT resolved.

I am currently busy building an priority queue server in ruby and I have chosen to use DRb as my communications platform.

While experimenting the simple examples from the Net (see here and here) I was consistently getting the same error from inside drb.rb (/opt/local/lib/ruby/1.8/drb/drb.rb:852):

/opt/local/lib/ruby/1.8/drb/drb.rb:852:in `initialize': getaddrinfo: nodename nor servname provided, or not known (SocketError)
from /opt/local/lib/ruby/1.8/drb/drb.rb:852:in `open'
from /opt/local/lib/ruby/1.8/drb/drb.rb:852:in `open_server_inaddr_any'
from /opt/local/lib/ruby/1.8/drb/drb.rb:864:in `open_server'
from /opt/local/lib/ruby/1.8/drb/drb.rb:759:in `open_server'
from /opt/local/lib/ruby/1.8/drb/drb.rb:757:in `each'
from /opt/local/lib/ruby/1.8/drb/drb.rb:757:in `open_server'
from /opt/local/lib/ruby/1.8/drb/drb.rb:1346:in `initialize'
from /opt/local/lib/ruby/1.8/drb/drb.rb:1634:in `new'
from /opt/local/lib/ruby/1.8/drb/drb.rb:1634:in `start_service'
from ./queue-provider.rb:32

What's up?
After poking drb.rb::self.open_server_inaddr_any(host, port) with a stick a few times two issues came to light:

  1. Multiple network address families are not catered for properly in the the code.

  2. TCPServer.open(port) where port == 0 fails under OS X but not Linux

Multiple Address Families
The code in question looks like this:

def self.open_server_inaddr_any(host, port)
infos = Socket::getaddrinfo(host, nil,
Socket::AF_UNSPEC,
Socket::SOCK_STREAM, 0,
Socket::AI_PASSIVE)
family = infos.collect { |af, *_| af }.uniq
case family
when ['AF_INET']
return TCPServer.open('0.0.0.0', port)
when ['AF_INET6']
return TCPServer.open('::', port)
else
return TCPServer.open(port)
end
end

From that we can see that we only seem to expect one network address family which is a little naive. Socket::getaddrinfo() on my MacBook Pro has the following to say (where host == 'localhost'):

$ irb
irb(main):006:0> require "socket"
=> true
irb(main):007:0> host = 'localhost'
=> "localhost"
irb(main):008:0> Socket::getaddrinfo(host, nil,
irb(main):009:1* Socket::AF_UNSPEC,
irb(main):010:1* Socket::SOCK_STREAM, 0,
irb(main):011:1* Socket::AI_PASSIVE)
=> [["AF_INET6", 0, "localhost", "::1", 30, 1, 6], ["AF_INET6", 0, "localhost", "fe80::1%lo0", 30, 1, 6], ["AF_INET", 0, "localhost", "127.0.0.1", 2, 1, 6]]

When you take this as your input you'll see that we don't end up matching either 'AF_INET' or 'AF_INET6' and we fall through to return TCPServer.open(port) because the case block expects a match against an array with one element.

TCPServer.open(0) OS X Weirdness
I have used DRb on both Linux and Windblowns in the past without a hitch so I was rather surprised to run into something like this which is a show stopper on OS X. I though I'd see if I was having the same problems on Linux to have something to compare with:

$ irb
irb(main):001:0> require "socket"
=> true
irb(main):002:0> port = 0
=> 0
irb(main):003:0> TCPServer.open(port)
=> #

Works a treat! Let's try that on OS X:

$ irb
irb(main):001:0> require "socket"
=> true
irb(main):002:0> port = 0
=> 0
irb(main):003:0> TCPServer.open(port)
SocketError: getaddrinfo: nodename nor servname provided, or not known
from (irb):3:in `initialize'
from (irb):3:in `open'
from (irb):3
from :0

CRASH! BOOM! BANG!

DRb Quilt
The first issue is rather trivial to fix:

def self.open_server_inaddr_any(host, port)
infos = Socket::getaddrinfo(host, nil,
Socket::AF_UNSPEC,
Socket::SOCK_STREAM,
0,
Socket::AI_PASSIVE)
families = Hash[*infos.collect { |af, *_| af }.uniq.zip([]).flatten]
return TCPServer.open('0.0.0.0', port) if families.has_key?('AF_INET')
return TCPServer.open('::', port) if families.has_key?('AF_INET6')
return TCPServer.open(port)
end

The code now rather assumes we're dealing with an array of one or more network address families and tries the IPv4 and IPv6 families first and then falls though to TCPServer.open(port).

I have opened a TT on RubyForge for this that contains a patch from me to fix the first issue.

What is required to fix the second issue? Dunno just yet, I'll keep looking and see if anything interesting pops up in the TT.




Wednesday, November 26, 2008

Massaging Rails Models (with a happy ending)

How do you alter data in a model so that the data which is stored to and gathered from the database is first filtered/transformed?

Two ways come to mind:

  • Insert the required behavior into your model's before_save, before_create and after_initialize callbacks.
  • Manually modify your attribute accessors for the attributes in question to do the magic for you.

We'll use the following contrived Model as our example:

class Gogga < ActiveRecord::Base
end

CREATE TABLE  `foo`.`goggas` (
`id` int(11) NOT NULL auto_increment,
`secret` varchar(255) default NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1

A Gogga has an id and a secret field. Let's pretend we need to keep Gogga.secret encrypted (using the super secret ROT13 algorithm) in our db and we would like the fact that it is encrypted to be transparent to our rails app. We need to therefore handle en/decryption of the secret data transparently from the rest of the app within the model.

Callbacks
The before_save, before_create and after_initialize callbacks are well documented in the Callbacks API documentation.

The strategy behind using the callbacks is to simply insert the behavior we want at the relevant stage of the object's life cycle. Here's one way to accomplish this using the mentioned callbacks:

class Gogga < ActiveRecord::Base
def before_save
self.secret = rot13(self.secret)
end

def before_create
self.secret = rot13(self.secret)
end

def after_initialize
self.secret = rot13(self.secret)
end

protected
def rot13(corpus)
return corpus.tr!("A-Za-z", "N-ZA-Mn-za-m")
end
end

If everything works as advertised our secret attribute should now be encoded when you call create or save on its model and decoded when you call a new on its model. The major drawback of this strategy is that if you are manipulating a large list of Goggas you will be post/pre-processing each of those instances.

The Lazy Way
An alternative would be to override the default behavior of the model to auto-generate attribute accessors via method_missing in the mystical black guts of ActiveRecord::Base. There's some info on this in the API docs as well in the Overwriting default accessors section.

This would look something like this:

class Gogga < ActiveRecord::Base
def secret
return rot13(read_attribute("secret"))
end

def secret=(value)
write_attribute("secret", rot13(value))
end

protected
def rot13(corpus)
return corpus.tr!("A-Za-z", "N-ZA-Mn-za-m")
end
end

This technique has the advantage that you never really mess with the internals of the model (as the view you have of it from outside is tinted by the accessor transformations) and of course work only gets done when you need to read/write the specific attribute.

Now, when you have one or two attributes that need to be protected writing out two accessors for each is not the end of the world. However, when you have several things become messy, tedious and downright boring.

Maybe we can look at some meta-magic to DRY things up a bit in another article.


Thursday, August 21, 2008

Oh ExtJS TreePanel Click, wherefore art thou?

ExtJS is a high quality JS UI library that really eases the cross-platform blues when it comes to designing online apps. Unfortunately I find the documentation can be a be a little obtuse from time to time.

Even the trusty Google librarian cannot answer some of the questions that pop up right off the bat and a fair amount of searching is required to get some info.

TreePanel Click
I have a TreePanel that I populate through lazy loading to save the amount of data I need to send off to the client in any one request. Unfortunately for the life of me I could not work out how to determine which node in the tree was clicked (so that I could fire off an Ajax request to populate another panel with data related to the clicked node).

Attaching Events
The first thing to do would obviously be to attach a click event listener somewhere so that we can pick up when were being prodded. If you're not familiar to JavaScript and ExtJS specifically you'd be inclined to attach an event to each of the nodes.

This is wasteful in the browser context as you should be taking advantage of event bubbling. So we simply attach a listener to the whole tree component:
  Ext.onReady(function(){
// Create initial root node
root = new Ext.tree.AsyncTreeNode({
text: 'Invisible Root',
id:'0'
});

// Create the tree
tree = new Ext.tree.TreePanel({
loader: new Ext.tree.TreeLoader({
url:'/companies',
requestMethod:'GET',
baseParams:{format:'json'}
}),
renderTo:'company-tree',
root: root,
rootVisible:false
});

// Expand invisible root node to trigger load of the first level of actual data
root.expand();

// Listen for mouse clicks
Ext.get('company-tree').on("click", function(){
// Code to determine which node was clicked ...
});
});
Where did that click go?
Never having used the TreePanel control before I had no real inkling how I was going to reap the relevant id from the node that was clicked. I tried several approaches but nothing was bearing any fruit till I found this article on the ExtJS forums discussing something similar.

Armed with a new weapon I was able to refactor the listener:
  // Listen for mouse clicks
Ext.get('company-tree').on("click", function(){
node = tree.getSelectionModel().getSelectedNode();
console.log("You clicked on", node.id);
});

That'll do.


About Me

My photo
I love solving real-world problems with code and systems (web apps, distributed systems and all the bits and pieces in-between).