Last updated at Wed, 27 Sep 2017 15:04:38 GMT

Back in February, Exodus Intelligence released their blog entry titled "Execute My Packet", which detailed their discovery and exploitation of CVE-2016-1287.  Since then, I've fielded numerous requests for modules and witnessed much discussion generated from it.  From this discussion, I've gathered that many researchers seem to consider the Cisco ASA as an unruly beast, difficult to approach, even harder to tame.  I feel that this is far from the truth, and this article is a response to such notions.

We attempted a module and stopped.  Before explaining why, some disclosures may be in order: while I wasn't on this project with David or Jordan, I actually worked at Exodus Intelligence during the discovery of this vulnerability and the initial exploitation attempts.  Jordan's original exploit, which the public has seen, is impressive in itself, though not portable across ASA's due to loss of heap determinism given variances in device configurations.  I'm positive that given more time, he would have found an information leak necessary to circumvent that.  Unfortunately, both he and I left Exodus before the disclosure of the bug, so I can't comment on the decision to release it in such a state.

Since the initial disclosure, I've worked both with him and independently to find a fruitful memory disclosure, but to no avail.  Given enough time, I'm sure it would come about, but the bug is patched.  Releasing a module now that could be used to compromise one's own personal device running an outdated software release feels like a wasted effort at best.  Rather, with the aforementioned questions and discussions in mind, I felt that more value would be had in using this as a teaching opportunity.  Much of this article will be old hat to many of you, but on that note, you aren't the intended audience.

While some people appear to almost fear the ASA, and embedded reverse-engineering in general, I'd argue that this is simply because it is an unknown.  I believe this is actually an extremely good way to get one's feet wet in the field.  The ASA runs on a common architecture, can be had with a valid license relatively cheap, and requires no electronics knowledge to begin picking apart.  Any bugs you may eventually find could prove rather valuable.  How much better could it be?

First Steps

The first step in all of this obviously involves setting up a research environment.  Though other, cheaper options do exist, by far the easiest approach to this is purchasing an ASA.  The other options include possibly using the ASA virtual appliance (which I have not investigated at all), or virtualization of the system software via other means.  I did attempt getting it running inside of QEMU, but the amount of work required to succeed when all you want to do is debug is quite daunting, so I went with a physical device.  In a town where tech startups crash and burn everyday, obtaining a used ASA on Craigslist is infinitely easier and cheaper when considering how much your time is worth.  For those of you so inclined, many Cisco certification seekers have formed a community centered around the effort to emulate the software within QEMU and GNS3.  Patches, scripts aimed at both packing and unpacking images, and hacked binaries exist for several older versions; however, there was none available at the time of writing for the vulnerable release.  Google is your friend!

On the hardware end, if you do end up getting an actual ASA, be sure to upgrade the RAM if it's operating with anything less than 256MB.  Debugging via the serial console is slow, and you'll likely be rebooting the device a lot.  You definitely want to eliminate any bottlenecks that you can before you begin.  Cisco sold branded memory for the device at quite a premium which is guaranteed to work.  I myself decided to risk $12 on a 1GB PC-3200 184 pin DIMM from Fry's that looked as if a small animal had been using the packaging as a chew toy.  So far it has worked flawlessly.  YMMV.  As for the serial connection, I use and recommend Parallax's USB to RS-232 adapter [https://www.parallax.com/product/28030]

Platform Overview

My bookshelf is lined with tomes such as Compilers (the classic "dragon book"), Windows NT Device Driver Development, Inside OLE, and many other equally thick books.  I am all for rigorous academic discourses on various topics when the situation calls for it.  This is not such a situation.  I feel that reverse engineering is often more about knowing what you need to ignore than trying to know everything one possibly can.  That said, I'm going to gloss over a lot of details which are well defined elsewhere.  For our purposes, I believe the salient points that require focus are as follows:

  • The Cisco ASA 5505 is a tiny computer
  • It's x86
  • It has a lot of network interfaces
  • It has a removable CF card which contains the firmware image
  • It runs Linux
  • The system boot sequence involves traversing through the BIOS, ROMMON (ROM Monitor, Cisco's bootstrap program available on this and other devices), GRUB, and off into Linux land which ends by loading the lina binary, which we will speak more of later.  The boot sequence is important because in order to do what we want, we have to hijack it.

Armed with this knowledge and a Cisco ASA 5505 in either it's physical or virtual manifestation, we're ready to get started.

Un-nesting the Matryoshka Dolls

The next step in our progression is to get setup for debugging.  Thankfully, Cisco was kind enough to include gdbserver on the firmware image, but for some unknown reason, they didn't make access to it very straightforward.  "Jailbreaks" from the CLI in earlier releases have been accomplished by unpacking the firmware, editing some script or another to start /bin/sh, repacking the firmware, and hoping that you didn't screw up somewhere along the line.  You can find details regarding techniques such as these in the excellent presentation "Breaking Bricks and Plumbing Tips" by Alec Stuart-Muirk.  (Alec's presentation is actually really, really good.  You should definitely check it out after reading this)

As I mentioned earlier, shell scripts and techniques for unpacking and repacking the firmware exist online, albeit not publicly for version 9.2.4.  While I won't link them, they are worth investigating for educational purposes, as the same general approach can be used for reversing firmware for other devices.  'xxd', 'dd', a keen mind, and a little experience are all that are truly required.  I recommend you also check out devttys0's excellent tool binwalk, as it can simplify much of the process for you.

I had originally intended to extract and repack the firmware myself, but after bouncing around ideas with David Barksdale, he provided me with alternative, that being a zen-like, that's-so-stupid-why-didn't-i-think-of-that, offset.  Assuming you have a copy of the vulnerable ASA firmware (asa924-k8.bin), open the file in your favorite hex editor.  From there, proceed to offset 0x1d1a03c.  You should see something like

This certainly looks relevant to our interests, almost like it might be Linux kernel boot parameters or something.  I wonder what would happen if we used a clever 1994 era trick and overwrite some of these options with something like:

rdinit=/bin/sh_

Save the file as something like asa924-k8-hax.bin

Once we have our modified image, we can transfer it to our device using one of a few methods, such as via TFTP from the ROMMON interface, or writing it to the CF card.  I had one laying around, so I went the card writer route.  With the CF card mounted in the OS of your choice, you can simply copy the file to the top level directory.  The TFTP process is fairly simple and well documented in the product manuals, so there's no need to run out and buy one if you're lacking such.

This project helped justify Brent's obsession with hoarding archaic historical relics

When you see “Use BREAK or ESC to interrupt boot”, do exactly as it says, and you should end up with something quite similar:

From the ROMMON prompt, we can force the device to load our firmware by using

boot disk0:/asa924-k8-hax.bin

Hit enter and be patient.  The boot time is excruciatingly slow in this age of NVMe SSD's.  Before too long, you should see something like this:


Score.  We're in.  Before getting too excited, realize that having a standard shell on an ASA is not too terribly useful, but it is a crucial step towards our end goal.  Typically, the first step one takes in auditing a product for bugs is to identify and enumerate the attack surface, ie all of the inputs of the system.  In other words, where will it let you shove a bunch of A's into it.  Cisco has simplified this process for us by cramming nearly all functionality that makes an ASA more useful than a decked out Raspberry Pi into one place: the lina process.  At least in my experience, and apart from the WebVPN interface which I have not investigated, nearly all packet filtering, QoS, and protocol capabilities, among others, are handled by this process.  There may be some other binaries I've overlooked, but lina is sufficiently large and complex enough to keep one busy for a long time.

With the boot sequence hijacked, we need to find a way to cleanly start lina under gdbserver.  Fortunately, a little 'sed' goes a long way towards that end:

sed -i 's/#\(.*-g -d.*\)/\1/' /asa/scripts/rcS  
sed -i 's|-g -d|-g -s /dev/ttyS0 -d|' /asa/scripts/rcS  
exec /sbin/init

Without explaining in minute detail, these two sed commands basically edit the flags for lina in /asa/scripts/rcS, setting it to execute in debug mode on the serial console.  After executing the real init process with the last line, if everything went according to plan, you should soon see a screen such as this:

Finally!  You might think we're done at this point, and you wouldn't be foolish to assume so despite the fact that you'd be wrong.  From here it follows that one should be able to debug the lina process by connecting gdb over the serial line.  This is true save for one final hurdle.  The application includes a watch dog mechanism that will reboot the system should the process fail to respond to polling within a given time limit.  This is great news for those who want redundancy in their SOHO networking hardware, and bad news for those who want to take their sweet time investigating the machine state from the confines of gdb.  I considered this as a stopping point for this article, but I'm feeling generous.  Consider the following:

And consider the first few lines of my .gdbinit

set disassembly-flavor intel  
target remote /dev/ttyUSB1  
set *0x0A53F168=0  
file ~/lina

This should get you going.  The proof of why it works is left as an exercise to the reader.

Where Do We Go From Here?

Bug hunting on this platform is our ultimate goal, but I feel the actual process of such is tangential to the aims of this article.  Still, I feel the need to offer a few pointers.  Before anything else, as soon as you are able to pull the lina binary from the file system, go ahead and begin processing it in your favorite disassembler whether you plan to statically audit code right away or not.  Remember how I said that Cisco packed nearly, if not all, functionality of the ASA into one binary?  Yea, it's huge.  I believe it took about 2 hours in IDA Pro on my Macbook, and a little less on my PC.  In any case, it was enough time to go have a cup of coffee or four.

With regards to the Cisco heap allocator and heap exploitation in general, I never thought I'd live to hear the question "What is Doug Lea's malloc?" - and this coming from people who I feel have a decent handle on more modern implementations such as the Windows Low Fragmentation Heap.  It seems a bit like learning Riemann Sums before mastering elementary algebra.  Still, I suppose it's entirely possible and becoming more commonplace as time marches on.  If you're already familiar with a more modern allocator, that's great, dlmalloc should be easy for you to pick up.  And if you're going to be doing vulnerability research on embedded devices, you definitely should pick it up.  While modern desktop operating systems have long since abandoned dlmalloc in favor of more robust solutions, variations of it pop up frequently in embedded systems given that it's simple and relatively efficient.  The most concise overview I can think to recommend would be the excellent "Once upon a free()... " written by the all-knowing anonymous and published in Phrack 57, article 0x09.  Exodus Intelligence's original report features an excellent write up on the proprietary changes Cisco built on top of dlmalloc, so I defer all vendor specific heap questions to their blog post.

Finally, for any of you asking "ok, how do I find bugs?", I'll leave you with this: you probably already know how.  If you're reading this and truly don't know where to start, I'd recommend reading the "Binary Auditing" chapter of "The Shellcoder's Handbook" or any of the innumerable references on fuzzing.  While it's true that spotting vulnerabilities takes some intuition, I believe that intuition can be learned.  There are no silver bullets.  The biggest hurdle is taking the time to do it.  That said, I certainly feel a future blog post specifically on this subject may be in order.  Happy hunting!

Special thanks to:

  • Jordan Gruskovnjak of Crowdstrike [w0 (@jgrusko) | Twitter] - my empathetic sounding board who understands the agony of crashing and rebooting an ASA 12 hours a day.
  • David Barksdale - who admirably has less of an online presence than I do
  • Brent Cook and everyone on the Metasploit team: proofreading and moral support

References: