TACACS+, packet loss and 'Authorization failed' errors

I've been hunting a TACACS+ issue that a customer reported whereby they would be able to log into a network device but somewhere in the duration of the session they would try and run a command and get the dreaded '% Authorization failed' error.

After a couple of seconds they would be able to continue as if nothing was amiss.

The tac_plus (I am using the Event-Driven version) logs were not showing anything out of the ordinary so this was a rather perplexing issue. The error seemed to be popping up rather randomly (and always when I did not have a tcpdump running).

While observing several devices over a larger time period I noticed that they were experiencing intermittent packet loss and this got me to wonder of the packet loss events I was noticing were not coinciding with the authorization failures.

Lucky for me tac_plus is running on a Linux host and so I thought I'd simulate packet loss between one of my own devices and the tac_plus service to see what that would result in. To implement the simulated packet loss I simply put the following rule in place:

iptables -A INPUT -m static -mode random -probability 0.5 -s /32 -j DROP

With a probability of 0.5 (this can be between 0 and 1) I was dropping around 40-50% of the packets.

Lo and behold, '% Authorization failure', scientifically reproducible.

Sharing is caring: Ruby, Perl, Memcached and MsgPack

Do you need to share data structures between ruby and perl ... FAST?

I recently saw MsgPack bubble through my RSS feeds and tagged it to go have another look. It provides very fast multi-language bindings for serialisation/de-serialisation.

A few quick experiments with the data I share (via memcached) between ruby and perl showed a write (serialisation + write to memcached) speed increase from 20s to 1.8s. Read (read from memcached + de-serialisation) performance showed similar performance increases.

All initial testing was done ruby -> memcached -> ruby but as soon as I switched to reading from memcached via perl I started getting 'extra bytes' errors from the perl side. I then tried perl -> memcached -> perl and everything was fine.

Weird.

A closer look at the data written to memcached and then read from perl showed that the data serialised with MsgPack on the ruby end was not the same as the data read by perl from memcached (validating the 'extra bytes' error).

Testing the write -> read process from perl to ruby yielded the following error:

/opt/local/lib/ruby/gems/1.8/gems/memcached-0.19.5/lib/memcached/memcached.rb:514:in `load': incompatible marshal file format (can't be read) (TypeError)

format version 4.8 required; 147.1 given

from /opt/local/lib/ruby/gems/1.8/gems/memcached-0.19.5/lib/memcached/memcached.rb:514:in `get'

from ./t_msgpack.rb:35:in `read_test'

from ./t_msgpack.rb:49

Now why on earth would I be getting a 'incompatible marshal file format' error as I am not using the ruby marshalling lib at all?

Turns out the memcached lib I use turns marshalling of ruby data on by default when you write to/read from memcached. This is most likely the best option for most cases where you don't want to use some other form of serialisation/de-serialisation but was really biting me here.

The solution is to simply stop the default behaviour of the memcached lib by using the following forms of get and set that turns on the 'raw' data handling switch for the memcached lib:

get KEY, false

set KEY, VALUE, TTL, false

The 'false' parameter at the end of those overrides the default behaviour turning default serialisation/de-serialisation via Marshall off.

Reality restored.

How do you make your Clojure REPL suck less?

Simple, rely on the venerable rlwrap that provides you with a readline wrapper around your existing REPL.

My sucky REPL looked like this:

#!/bin/sh
java -cp PATH_TO/clojure.jar clojure.main $1



Simply install the rlwrap package using your favorite package manager and change your REPL script (clj) to the following:

#!/bin/sh
rlwrap java -cp PATH_TO/clojure.jar clojure.main $1


Major suckiness averted. The added boon of this approach is that you now get all the readline goodness (history traversal, inline editing, etc.) you've come to depend on in other REPLs.