by Derek Abdine & Bob Rudis (photo CC-BY-SA Kalle Gustafsson)

Astute readers will no doubt remember the Shadow Brokers leak of the Equation Group exploit kits and hacking tools back in mid-August. More recently, security researchers at SilentSignal noted that it was possible to modify the EXTRABACON exploit from the initial dump to work on newer Cisco ASA (Adaptive Security Appliance) devices, meaning that virtually all ASA devices (8.x to 9.2(4)) are vulnerable and it may be interesting to dig into the vulnerability a bit more from a different perspective.

Now, "vulnerable" is an interesting word to use since:

  • the ASA device must have SNMP enabled and an attacker must have the ability to reach the device via UDP SNMP (yes, SNMP can run over TCP though it's rare to see it working that way) and know the SNMP community string
  • an attacker must also have telnet or SSH access to the devices

This generally makes the EXTRABACON attack something that would occur within an organization's network, specifically from a network segment that has SNMP and telnet/SSH access to a vulnerable device. So, the world is not ending, the internet is not broken and even if an attacker had the necessary access, they are just as likely to crash a Cisco ASA device as they are to gain command-line access to one by using the exploit. Even though there's a high probable loss magnitude1 from a successful exploit, the threat capability2 and threat event frequency3 for attacks would most likely be low in the vast majority of organizations that use these devices to secure their environments. Having said that, EXTRABACON is a pretty critical vulnerability in a core network security infrastructure device and Cisco patches are generally quick and safe to deploy, so it would be prudent for most organizations to deploy the patch as soon as they can obtain and test it.

Cisco did an admirable job responding to the exploit release and has a patch ready for organizations to deploy. We here at Rapid7 Labs wanted to see if it was possible to both identify externally facing Cisco ASA devices and see how many of those devices were still unpatched. Unfortunately, most firewalls aren't going to have their administrative interfaces hanging off the public internet nor are they likely to have telnet, SSH or SNMP enabled from the internet. So, we set our sights on using Project Sonar to identify ASA devices with SSL/IPsec VPN services enabled since:

  • users generally access corporate VPNs over the internet (so we will be able to see them)
  • many organizations deploy SSL VPNs these days versus or in addition to IPsec (or other) VPNs (and, we capture all SSL sites on the internet via Project Sonar)
  • these SSL VPN-enabled Cisco ASAs are easily identified

We found over 50,000 Cisco ASA SSL VPN devices in our most recent SSL scan.Keeping with the spirit of our recent National Exposure research, here's a breakdown of the top 20 countries:

Table 1: Device Counts by Country

Country Device count %
United States 25,644 50.9%
Germany 3,115 6.2%
United Kingdom 2,597 5.2%
Canada 1,994 4.0%
Japan 1,774 3.5%
Netherlands 1,310 2.6%
Sweden 1,095 2.2%
Australia 1,083 2.2%
Denmark 1,026 2.0%
Italy 991 2.0%
Russian Federation 834 1.7%
France 777 1.5%
Switzerland 603 1.2%
China 535 1.1%
Austria 497 1.0%
Norway 448 0.9%
Poland 410 0.8%
Finland 404 0.8%
Czech Republic 396 0.8%
Spain 289 0.6%

Because these are SSL VPN devices, we also have access to the certificates that organizations used to ensure confidentiality and integrity of the communications. Most organizations have one or two (higher availability) VPN devices deployed, but many must deploy significantly more devices for geographic coverage or capacity needs:

Table 2: List of organizations with ten or more VPN ASA devices

Organization Count
Large Japanese telecom provider 55
Large U.S. multinational technology company 23
Large U.S. health care provider 20
Large Vietnamese financial services company 18
Large Global utilities service provider 18
Large U.K. financial services company 16
Large Canadian university 16
Large Global talent management service provider 15
Large Global consulting company 14
Large French multinational manufacturer 13
Large Brazilian telecom provider 12
Large Swedish technology services company 12
Large U.S. database systems provider 11
Large U.S. health insurance provider 11
Large U.K. government agency 10

So What?

The above data is somewhat interesting on its own, but what we really wanted to know is how many of these devices had not been patched yet (meaning that they are technically vulnerable if an attacker is in the right network position). Remember, it's unlikely these organizations have telnet, SSH and SNMP enabled to the internet and researchers in most countries, including those of us here in the United States, are not legally allowed to make credentialed scan attempts on these services without permission. Actually testing for SNMP and telnet/SSH access would have let us identify truly vulnerable systems. After some bantering with the extended team (Brent Cook, Tom Sellers & jhart) and testing against a few known devices, we decided to use hping to determine device uptime from timestamps and see how many devices had been rebooted since release of the original exploits on (roughly) August 15, 2016. We modified our Sonar environment to enable hping studies and then ran the uptime scan across the target device IP list on August 26, 2016, so any system with an uptime > 12 days that has not been rebooted (or is employing some serious timestamp masking techniques) is technically vulnerable. Also remember that organizations who thought their shiny new ASA devices weren't vulnerable also became vulnerable after the August 25, 2016 SilentSignal blog post (meaning that if they thought it was reasonable not to patch and reboot it became unreasonable to think that way on August 25).

So, how many of these organizations patched & rebooted? Well, nearly 12,000 (~24%) of them prevented us from capturing the timestamps. Of the remaining ones, here's how their patch status looks:

We can look at the distribution of uptime in a different way with a histogram, making 6-day buckets (so we can more easily see "Day 12"):

This also shows the weekly patch/reboot cadence that many organizations employ.

Let's go back to our organization list and see what the mean last-reboot time is for them:

Table 3: hping Scan results (2016-08-26)

Organization Count Mean uptime (days)
Large Japanese telecom provider 55 33
Large U.S. multinational technology company 23 27
Large U.S. health care provider 20 47
Large Vietnamese financial services company 18 5
Large Global utilities service provider 18 40
Large U.K. financial services company 16 14
Large Canadian university 16 21
Large Global talent management service provider 15 Unavailable
Large Global consulting company 14 21
Large French multinational manufacturer 13 34
Large Brazilian telecom provider 12 23
Large Swedish technology services company 12 4
Large U.S. database systems provider 11 25
Large U.S. health insurance provider 11 Unavailable
Large U.K. government agency 10 40

Two had no uptime data available and two had rebooted/likely patched since the original exploit release.

Fin

We ran the uptime scan after the close of the weekend (organizations may have waited until the weekend to patch/reboot after the latest exploit news) and here's how our list looked:

Table 4: hping Scan Results (2016-08-29)

Organization Count Mean uptime (days)
Large Japanese telecom provider 55 38
Large U.S. multinational technology company 23 31
Large U.S. health care provider 20 2
Large Vietnamese financial services company 18 9
Large Global utilities service provider 18 44
Large U.K. financial services company 16 18
Large Canadian university 16 26
Large Global talent management service provider 15 Unavailable
Large Global consulting company 14 25
Large French multinational manufacturer 13 38
Large Brazilian telecom provider 12 28
Large Swedish technology services company 12 8
Large U.S. database systems provider 11 26
Large U.S. health insurance provider 11 Unavailable
Large U.K. government agency 10 39

Only one additional organization (highlighted) from our "top" list rebooted (likely patched) since the previous scan, but an additional 4,667 devices from the full data set were rebooted (likely patched).

This bird's eye view of how organizations have reacted to the initial and updated EXTRABACON exploit releases shows that some appear to have assessed the issue as serious enough to react quickly while others have moved a bit more cautiously. It's important to stress, once again, that attackers need to have far more than external SSL access to exploit these systems. However, also note that the vulnerability is very real and impacts a wide array of Cisco devices beyond these SSL VPNs. So, while you may have assessed this as a low risk, it should not be forgotten and you may want to ensure you have the most up-to-date inventory of what Cisco ASA devices you are using, where they are located and the security configurations on the network segments with access to them.

We just looked for a small, externally visible fraction of these devices and found that only 38% of them have likely been patched. We're eager to hear how organizations assessed this vulnerability disclosure in order to make the update/no update decision. So, if you're brave, drop a note in the comments or feel free to send a note to research@rapid7.com (all replies to that e-mail will be kept confidential).


1,2,3 Open FAIR Risk Taxonomy [PDF]