This post is the third in a series examining the roles of search and analytics in the incident-detection-to-response lifecycle. To read the first two, click here and here.

In the second blog of this series, I touched on the need for solutions more flexible than the traditional SIEM architecture focused primarily on receiving logs from the security appliances in your infrastructure. This wasn't only a passing comment; in the past five years, the approach to compromising an organization and corresponding work to defend against it have changed significantly as new technologies have emerged for use by both sides.

Attackers get in by a variety of perimeter-blind routes, which means an expansion of the data sets security teams need to interpret

Whether you use the Verizon Data Breach Investigations Report as your source of truth or rely on anecdotal evidence from the details which gradually emerge, it is clear that web application attacks and stolen credentials comprise the majority of the successful entry points in recent years. This is especially frightening for companies using perimeter security devices as their only source of prevention and/or detection. Not only are they not watching the right entry points for most of today's attacks, but even if they possess the tools and knowledge to effectively analyze it, they likely have no access to the data which would help them detect these attacks.

The unfortunate reality for security teams today is that they cannot simply forget about perimeter devices and dedicate months of their lives to overhauling their entire infrastructure while learning the most advanced practices for mitigating these primary attack vectors. Using firewalls, intrusion detection/prevention systems, and two-factor authentication on VPNs should be considered a starting point for security teams. Removing these tools from the equation would mean initial compromise techniques currently with shrinking success rates would reclaim the top of the list. This gradual evolution of intrusion techniques forces the need for a continually expanding breadth of knowledge on the security team and more direct access to other teams' knowledge when an incident warrants it.

Tracking lateral movement is near-impossible if you only have data on centralized servers

Unless you expose your servers with the most sensitive [read: monetizable] information to the internet, the attackers are not going to stop after the first compromise. Even in the most poorly secured environments, it would be rare to store the data most critical to the business in a web application on the network's edge. Once the initial compromise is successful, there are many ways to pivot to other hosts, once inside. Harvesting credentials for use in moving from system to system is a popular method of stealthy reconnaissance as more is discovered, and it can be done with polymorphic malware to evade known-hash detection or with any number of attacker-at-keyboard toolkits. This presents a massive challenge for investigations in the vast majority of organizations who are unable to monitor the wealth of information isolated on their endpoints.

As if it weren't a large enough challenge to monitor every endpoint on the network, and at your employees' homes, an attacker can also move laterally to one of your organization's cloud environments, whether it hosts employee data, customer data, or other valuable data like source code. Segmenting valuable data across distinct cloud services is a great way for modern enterprises to reduce the impact of a compromise, but they all need to be monitored and secured the same as you would any area of your physical network. Otherwise, analyzing an incident without activity from your managed clouds can feel like the scene at the end of Blue Streak when the police are powerless to follow Miles any further because he has crossed the border into Mexico. No portion of your environment, even virtual and hosted infrastructures, should be just beyond the reach of your security team.

You need to analyze machine data from inside and outside the traditional perimeter to effectively investigate today's attacks

All of this seems like a daunting assignment because it is. If you don't have the benefit of a large team who understands the structured and unstructured data across your environment and the resources to automate the correlation and internal attribution across all of these data sources, your team is going to spend a significant number of hours duplicating manual efforts to analyze the incidents you detect. There will be incidents closed in minutes because of their familiar feel to a senior analyst and there will be incidents which take junior analysts days to effectively investigate, but there will be little time for other duties outside of alert triage or incident analysis.

If you have a different user interface for your perimeter devices, malware detection, endpoint detection, and cloud monitoring, you likely spend a lot of time switching between monitors and browser tabs to look for a string to tie events together. Too much of this has become a cascade of similar searches across unlike data sets. The ability to search will always be a necessity [as I wrote in the first post of this series], but siloed search is too manual and slow to stand alone as an investigative tool, especially if it means different search capabilities in different places. Search cannot be an afterthought. Each data source can provide valuable context to search results after analytics have been applied.

The paired solution needs to be a single place to pivot through relevant data from your entire environment. This was once dubbed a "single pane of glass" and today, it is at the center of what analytics solutions now aim to offer incident responders. If the data is structured and understood, speed analysis by automating the classification of data as user or system behavior. Detection and analysis solutions from other security vendors need to be a source of more context, rather than another console for independent analysis. If classification is currently a challenge, allow easy exploration through search. One place to send the breadth of data. One place to access a mix of automatically-drawn conclusions and conclusions requiring a human mind to interpret.

Click here to learn more about User Behavior Analytics.

If you want to learn more about building your incident response capabilities, check out our incident response toolkit. If searching through machine data is your current need, you can start a free trial of Logentries here and search your data within seconds.