Last month, I wrote about the two largest incident response bottlenecks behind the massive gap in time to compromise an organization and time it takes incident response teams to verify the true incident and take appropriate action. This post is meant to go into much greater detail on the second bottleneck: incident analysis (AKA investigation).

Challenge #1: Incident analysis with existing security tools can be very frustrating

In the vast majority of organizations today, the entire process of analyzing an incident after triage consists of searching through log data. At some point, crafty responders with no budget for new tools recognized the wealth of information available in the log data regulations were forcing their organization to retain. After more than a decade, incident response became a primary use case for log aggregation solutions and any that enable IR teams gave way to those products that made it extremely easy to (a) get more data in and (b) locate the data you desire. Whether incident analysis is performed completely via search or custom-built tools, more data can be a burden without proper context. As the amount of data collected has quickly grown, IR teams are basically being dropped in a dark forest and asked to search via moonlight.

If this challenge weren't daunting enough, they are unable to keep all of the data accessible without surpassing the rest of IT's annual budget. A common cut-off for searchable data in organizations is around thirty days, so it is no wonder that any incident that occurred more than a month previous is typically discovered by a third party. For incident analysts to obtain older swaths of data takes hours or days and leads most organizations to focus only on incidents younger than the magic 30-day cut-off. This means that they could be running through that same dark forest, convinced they are close to figuring out exactly what's out there, when they suddenly reach a concrete wall blocking their path. They could potentially find a way over it, but their time is often better spent on another of their endless pile of incidents due for analysis.

Challenge #2: There is a well-known shortage of qualified incident analysts worldwide

Expert incident analysts are like wizards who can consistently use a combination of gut feel, prior experience, and regular expression mastery to dig through disparate datasets and

come to a conclusion about the cause and total impact of an incident, all the while having to resist settling on the first story that is easy to tell. If you don't understand exactly how these incident analysis wizards work their magic, don't feel bad, because very few people do. To most of us, it is similar to the scenes in "Harry Potter and the Order of the Phoenix" when Harry and his crew are being chased through a seemingly infinite warehouse of glass spheres while trying to locate a single one containing an important prophecy that only Harry could see. How did he know where to look? Isn't there some kind of card catalog system for those things? Those of us without the gift of incident analysis magic may never know.

But that is part of the problem. If more organizations are being attacked now and it takes years to develop this magic, how are we going to properly staff a team? The UserInsight team believes that we can help you find ways to move beyond search as a primary means of incident investigation to ultimately increase the number of people that can determine the root cause. This does not mean that you will never need the gift of incident analysis magic; it just means that rather than continuing the incremental gains in search result turnaround times, investigations can be sped ten-fold by contextualizing the data and easing analysis for the less experienced members of the team.

Challenge #3: It is often very difficult to investigate an incident as a team

I can guarantee that if you are lucky enough to have an incident analysis wizard in your organization, he or she is overworked. There is a seemingly infinite backlog of triaged incidents and complaints about a "fifty-hour work week" within earshot causes an eye-roll and annoyed grunt. But simply enabling the rest of the team to assist with this backlog is not the full solution. A new challenge of duplicative efforts and other inefficiencies will be introduced if all of the team members are doing incident analysis in isolation. There need to be ways for all team members to properly collaborate, understand what others have found in analysis, and participate in ongoing investigations.

The UserInsight team's solution for the problems described here is what we call the interactive incident timeline. We currently consider it to be in "Beta", but that tag is more like a Google-style Beta that means it is still being regularly enhanced as we have conversations with customers, rather than the type of Beta that means "your data could disappear at any moment". We already have received great feedback from our customer base that (a) it is helping to speed analysis by as much as 20X and (b) we can continue to improve its teamwork and collaboration aspects. We are always excited to get both kinds of feedback.

To learn more about UserInsight and Rapid7's other solutions for detecting compromised credentials, check out our compromised credentials resource page and make sure to download our complimentary information toolkit. I expect you'll quickly see how much we can help with your team's incident analysis process.