ssh-agent for Developers

Have you ever wanted to automate the ssh pass phrase login procedure when connecting to remote systems that have your public key in their .ssh/authorized_keys?

This is done using ssh-agent (and ssh-add) which will be on your Debian or Ubuntu system if you have the openssh-client dep installed. For other flavours of Linux, OS X or UNIX please refer to your package management documentation (or install from source) to see how you can install the required software.

On an Ubuntu system ssh-agent is started for you by default. Please refer to your system documentation for ssh-agent to find the correct way to run it on your system.

The following approach should work for situations where the client (the computer with the private key) is either a server (access is generally restricted to it via remote shell) or a desktop (includes laptops) with a graphical terminal program.

Add the following to the end of your ~/.bashrc (or other suitable shell setup configuration file):

# Run ssh-add if it has not been run already.
if  ssh-add -l | grep -q 'The agent has no identities.'
then
eval "ssh-add"
fi

Save the addition to your .bashrc (or suitable alternative) and log out and back in.

You will be presented with a request for your pass phrase you chose when creating your public/private keys. Enter it and sigh with relief as your default key(s) are cached in memory.

When you now try and log into the remote system again there will be no passwords or pass phrases required for this session.

Ruby (Hpricot) Program Guide - I

Do you live outside of South Africa and subscribe to the M-Net Africa service? Ever wanted to avoid the M-Net Africa site and just get the program guide for your region?

Well, look no further. Ruby and Hpricot to the rescue!

The M-Net Schedule site has changed quite often over the last few months so chances are good that by the time you get to this article their site may have devolved again. Doing screen scraping on web sites is generally fraught with pain, suffering and disappointment.

This is generally due to the fact that you're providing a static way to read dynamic (over time) content. Don't get discouraged though, just build notification of changes into your screen scraper and ensure that it can notify you when things have changed so that you can up date it.

To compound the problem, many sites (including the M-Net Schedule site) do not conform to the XHTML standard. This simply means that they have not used semantic tools to layout their site to abstract the structure, content and behaviour from their site. A quick validation via the W3C Markup Validation Service confirms that the parser can't even determine the content encoding.

Embrace change - it is a lot less painful (not to mention more productive ;).

Analysis of Structure
The first step of parsing content from an outside source is to analyse the structure of the content to determine what strategies you are going to employ to read and parse the content. Below is an extract of the type of content we're interested in:

<tr>
<td colspan="5">
  <font class="ScheduleSchedule">Today's Schedule for :</font>
  <font class="ScheduleChannel">Cartoon Network (Africa)</font>
</td>
</tr>
<tr>
<td colspan="5">&nbsp;</td>
</tr>
<tr>
<td colspan="5">
  <font class="ScheduleDate">17&nbsp;April&nbsp;2007</font>
</td>
</tr>
<tr bgcolor="F5F5F5">
<td colspan="5">&nbsp;</td>
</tr>
<tr bgcolor="F5F5F5">
<td width="40">
  <b><font class="ScheduleTime"> 06:25</font></b>
</td>
<!--Time-->
<td width="420">
  <font class="ScheduleTitle">Codename: Kids Next Door
  <!--Title-->
  </font>
</td>
<td width="17"></td>
<td width="188"></td>
<!--SMS Reminder-->
<td width="50" align="right">
  <a href="#" onclick="OpenAgeRestriction(1);return false;">Family</a>
</td>
<!--Age Restriction-->
</tr>
<tr bgcolor="F5F5F5">
<td colspan="5">
  <p>A gang of 10 year olds takes on top secret missions, using fantastic home-made technology to safeguard their treehouse against attack and grown-ups.</p>
</td>
</tr>
<tr>
<td colspan="5">&nbsp;</td>
</tr>
<tr bgcolor="F5F5F5">
<td width="40">
  <b><font class="ScheduleTime"> 06:50</font></b>
</td>
<!--Time-->
<td width="420">
  <font class="ScheduleTitle">The Powerpuff Girls
  <!--Title-->
  </font>
</td>
<td width="17"></td>
<td width="188"></td>
<!--SMS Reminder-->
<td width="50" align="right">
  <a href="#" onclick="OpenAgeRestriction(1);return false;">Family</a>
</td>
<!--Age Restriction-->
</tr>
<tr bgcolor="F5F5F5">
<td colspan="5">
  <p>The wild and wacky escapades of three girls with extraordinary powers. Blossom, Buttercup and Bubbles use their superpowers to fight crime and villainy in Townsville.</p>
</td>
</tr>
<tr>
<td colspan="5">&nbsp;</td>
</tr>

The first table row we're interested in is the one that tells us which channels we are looking at and what this day's date is (lightly formatted for readability):

<tr>
<td colspan="5">
  <font class="ScheduleSchedule">Today\'s Schedule for :</font>
  <font class="ScheduleChannel">Cartoon Network (Africa)</font>
</td>
</tr>
<tr>
<td colspan="5">&nbsp;</td>
</tr>
<tr>
<td colspan="5">
  <font class="ScheduleDate">17&nbsp;April&nbsp;2007</font>
</td>
</tr>

The name of the channel resides in a font tag whit a class attribute of "ScheduleChannel" and the date we're working with also resides in a font tag with a class attribute of "ScheduleDate". How does the search for this information translate into code?I will be using a XPath query (Hpricot supports both XPath and CSS selector based queries) to find the first font tag that has a class attribute that I am searching for:

def channel
  @channel = @hp.at("font[@class='ScheduleChannel']").inner_html
end

def date
  @date = @hp.at("font[@class='ScheduleDate']").inner_html
end

That's all pretty plain Jane so far. Here is what a typical table row looks like that contains the time of the program (reformatted for readability):

<tr bgcolor="F5F5F5">
<td width="40">
  <b><font class="ScheduleTime"> 06:25</font></b>
</td>
<!--Time-->
<td width="420">
  <font class="ScheduleTitle">Codename: Kids Next Door
  <!--Title-->
  </font>
</td>
<td width="17"></td>
<td width="188"></td>
<!--SMS Reminder-->
<td width="50" align="right">
  <a href="#" onclick="OpenAgeRestriction(1);return false;">Family</a>
</td>
<!--Age Restriction-->
<tr>
<tr bgcolor="F5F5F5">
<td colspan="5">
  <p>A gang of 10 year olds takes on top secret missions, using fantastic home-made technology to safeguard their treehouse against attack and grown-ups.</p>
</td>
</tr>

The time is similarly found in a font tag with a class tag of "ScheduleTime" and the program is found in a font tag with a class attribute of "ScheduleTitle". The program synopsis is however wrapped in a table column with a span of 5 and a paragraph tag.

def time
@time = @hp.at("font[@class='ScheduleTime']").inner_html
end

def title
@title = @hp.at("font[@class='ScheduleTitle']").inner_html
end

def synopsis
@synopsis = (@hp/"td[@colspan=5]/p").first.inner_html
end

You will notice that the extraction of the time and title holds no surprises. The synopsis extraction however is something new. I chose to use a CSS selector search for the synopsis by looking for the first td tag that has a colspan=5 attribute, followed by a p tag's contents (inner_html).

If you were to print the values of the relevant variables you would see that there is still some cleaning up that needs to be done on them before they can be considered for programmatic consumption:

Channel: Cartoon Network (Africa)
Date: 17 April 2007
Time:  06:25
Title: Codename: Kids Next Door              <!--Title-->


Synopsis: A gang of 10 year olds takes on top secret missions, using fantastic home-made technology to safeguard their treehouse against attack and grown-ups.

The channel and time looks fine so we'll just skip them for now. The date has some HTML entities in it so let's remove them using the handy HTMLEntities (I recommend installing from the gem) lib. The problem is that if they sneaked in some HTML entities in the date they may choose to do this elsewhere as well so let's not trust the input and ensure we sanitise all input in a generic way:

def initialize(url)
@coder = HTMLEntities.new
p sanitize(url)
end

def sanitize(string)
@coder.decode(string)
end

The only problem with this is that that HTMLEntities uses UTF-8 encoding which outputs (on my system) something like this for the date value:

"17\\302\\240April\\302\\2402007"

Not really ideal ... let's use the iconv lib to get the UTF-8 string forced into a US-ASCII encoding:

def initialize(url)
@coder = HTMLEntities.new
@ic = Iconv.new('US-ASCII//TRANSLIT', 'UTF-8')
p sanitize(url)
end

def sanitize(string)
@ic.iconv(@coder.decode(string))
end

Right, now to get back on track after that slight detour. To recap, we now have a strategy to get all the items we're interested in but the solution above is a little naive because it assumes we only have one day with one program. The complete schedule for a channel could span several days, each having variable amounts of programs per day.

One can also extend the ideas above to make it a lot more usable by downloading multiple channels for you and possibly pretty print it, send it to yourself via email or drop it in a db for later processing or display.

I'll cover these in followup articles to come ...

(Almost) Overriding helpers in rails

In a project I am currently working on I use the select_datetime() helper to provide me with year + month + day + hour + minute drop down boxes. After using the UI for a while the customer mentioned that they do not require the full minute listing (from 00 to 59 minutes) for the minute drop down box and would prefer to only have 15 minute increments to shorten the drop down and waste less scrolling time when choosing the minutes.

Umm ... OK!

This type of functionality can be provided in a few ways (that I
know of ):

• Wrapper: Create your own helper wrapper
  
• Replace: Replace the original helper method in the module where it comes from
  
• RTFM: Read the docs first and then code
  

select_datetime() is actually a composition of two other helpers:

• select_date()
  
• select_time()
  

select_time() is another composition of:

• select_hour()
  
• select_minute()
  
• select_second()
  

Because we're simply interested in changing the behaviour of the minutes we will only be changing select_minute().

Wrapper
This approach is pretty straight forward. All you need to do is add your own helper as a wrapper for the helper you wish to masquerade. Unfortunately we cannot use this technique here because the helper we're using (select_datetime()) cannot provide us with the relevant functionality we want by simply massaging the helper's input or output.

Bummer.

Replace
Your intuition may be itching at this point and that annoying inner voice you never ignore could be saying:

'Why don't you just simply override the select_datetime() in application_helper.rb or ActionView::Base class, lilly-livered limp-wristed sissy!'.

In this case your inner voice needs to be muzzled. Helpers are included as mixins. This means the methods in the mixed in module become available to the classes using them as if they were part of the actual class.

The PickAxe book says it succinctly:

'In fact, mixed-in modules effectively behave as superclasses.'

Unfortunately this also means that if you change the methods in a module they will change for all classes that use the mixed-in module. If you want to change the helper behaviour you need to change the behaviour in the module that is providing the helper as a mixin, in this case ActionView::Helpers::DateHelper.

Off we go and open ActionView::Helpers::DateHelper to start mucking with select_minute() so that we can support periods as the customer requested. Here is the relevant code:

# File vendor/rails/actionpack/lib/action_view/helpers/date_helper.rb, line 190

190: def select_minute(datetime, options = {})
191: val = datetime ? (datetime.kind_of?(Fixnum) ? datetime : datetime.min) : ''
192: if options[:use_hidden]
193: hidden_html(options[:field_name] || 'minute', val, options)
194: else
195: minute_options = []
196: 0.step(59, options[:minute_step] || 1) do |minute|
197: minute_options << ((val == minute) ? 198: %(#{leading_zero_on_single_digits(minute)}\n) : 199: %(#{leading_zero_on_single_digits(minute)}\n) 200: ) 201: end 202: select_html(options[:field_name] || 'minute', minute_options, options) 203: end 204: end

Alrighty! Let's add that stepping code to provide the periodic minutes ...

RTFM
Damn! Had I spent a little time drilling down through the documentation I would have noticed that the select_minute() helper already provides the required functionality via its :minute_step option.

Here is what the documentation says about this helper:

select_minute(datetime, options = {})

Returns a select tag with options for each of the minutes 0 through 59 with the
current minute selected. Also can return a select tag with options by minute_step
from 0 through 59 with the 00 minute selected The minute can also be substituted
for a minute number. Override the field name using the :field_name option, 'minute'
by default.

So, all of this may seem a little pointless but the lesson here is check the docs before you take-off at light speed. I look forward to writing an actual article on overriding helpers in rails soon.