Rapid7 Blog

Log Management  

5 Ways to Use Log Data to Analyze System Performance

Analyzing System Performance Using Log Data Recently we examined some of the most common behaviors that our community of 25,000 users looked for in their logs, with a particular focus on web server logs. In fact, our research identified the top 15 web server…

Analyzing System Performance Using Log Data Recently we examined some of the most common behaviors that our community of 25,000 users looked for in their logs, with a particular focus on web server logs. In fact, our research identified the top 15 web server tags and alerts created by our customers—you can read more about these in our https://logentries.com/doc/community-insights/ section—and you can also easily create tags or alerts based on the patterns to identify these behaviors in your systems log data. This week we are focusing on system performance analysis using log data. Again we looked across our community of over 25,000 users and identified five ways in which people use log data to analyze system performance. As always, customer data was anonymized to protect privacy. Over the course of the next week, we will be diving into each of these areas in more detail and will feature customers' first-hand accounts of how they are using log data to help identify and resolve issues in their systems and analyze overall system performance. Our research looked at more than 200,000 log analysis patterns from across our community to identify important events in log data. With a particular focus on system performance and IT operations related issues, we identified the following five areas as trending and common across our user base. 5 Key Log Analysis Insights 1. Slow Response Times Response times are one of the most common and useful system performance measures available from your log data. They give you an immediate understanding of how long a request is taking to be returned. For example, web server log data can give you insight into how long a request takes to return a response to a client device. This can include time taken for the different components behind your web server (application servers, DBs) to process the request, so it can offer an immediate view of how well your application is performing. Recording response times from the client device/browser can give you an even more complete picture since this also captures page load time in the app/browser as well as network latency. A good rule of thumb when measuring response times is to follow the 3 response time limits as outlined by Jakob Nielsen in his publication on ‘Usability Engineering' back in 1993 (still relevant today). 0.1 second is about the limit for having the user feel that the system is reacting instantaneously, 1.0 second is about the limit for the user's flow of thought to stay uninterrupted, and 10 seconds is about the limit for keeping the user's attention focused on the dialogue. Slow response time patterns almost always follow the pattern below: response_time>X In this context, response_time is the field value representing the server or client's response, and ‘X' is a threshold. If this threshold is exceeded, you want the event to be highlighted or to receive a notification so that you and your team are aware that somebody is having a poor user experience. 2. Memory Issues and Garbage Collection Outofmemory errors can be pretty catastrophic, as they often result in the application's crashing due to lack of resources. Thus, you want to know about these when they occur; creating tags and generating notifications via alerts when these events occur is always recommended. Your garbage collection behavior can be a leading indicator of outofmemory issues. Tracking this and getting notified if heap used vs free heap space is over a particular threshold, or if garbage collection is taking a long time, can be particularly useful and may point you toward memory leaks. Identifying a memory leak before an outofmemory exception can be the difference between a major system outage and a simple server restart until the issue is patched. Furthermore, slow or long garbage collection can also be a reason for a user's experiencing slow application behavior. During garbage collection, your system can slow down; in some situations it blocks until garbage collection is complete (e.g. with ‘stop the world' garbage collection). Below are some examples of common patterns used to identify the memory related issues outlined above: Out of memory exceeds memory limit memory leak detected java.lang.OutOfMemoryError System.OutOfMemoryException memwatch:leak: Ended heapDiff GC AND stats 3. Deadlocks and Threading Issues Deadlocks can occur in many shapes and sizes and can have a range of negative effects—from simply slowing your system down to bringing it to a complete halt. In short, a deadlock is a situation in which two or more competing actions are each waiting for the other to finish, and thus neither ever does. For example, we say that a set of processes or threads is deadlocked when each thread is waiting for an event that only another process in the set can cause. Not surprisingly, deadlocks feature as one of our top 5 system performance related issues that our users write patterns to detect in their systems. Most deadlock patterns simply contain the keyword ‘deadlock', but some of the common patterns follow the following structure: ‘deadlock' ‘Deadlock found when trying to get lock' ‘Unexpected error while processing request: deadlock;' 4. High Resource Usage (CPU/Disk/ Network) In many cases, a slow down in system performance may not be as a result of any major software flaw, but instead is a simple case of the load on your system increasing without increased resources available to deal with this. Tracking resource usage can allow you to see when you require additional capacity such that you can kick off more server instances (for example). Example patterns used when analyzing resource usage: metric=/CPUUtilization/ AND minimum>X cpu>X disk>X disk is at or near capacity not enough space on the disk java.io.IOException: No space left on device insufficient bandwidth 5. Database Issues and Slow Queries Knowing when a query failed can be useful, since it allows you to identify situations when a request may have returned without the relevant data and thus helps you identify when users are not getting the data they need. There can be more subtle issues, however, such as when a user is getting the correct results but the results are taking a long time to return. While technically the system may be fine (and bug-free), a slow user experience hurts your top line. Tracking slow queries allows you to track how your DB queries are performing. Setting acceptable thresholds for query time and reporting on anything that exceeds these thresholds can help you quickly identify when your user's experience is being affected. Example patterns: SqlException SQL Timeout Long query Slow query WARNING: Query took longer than X Query_time > X Continued Log Data Analysis As always, let us know if you think we have left out any important issues that you like to track in your log data. To start tracking your own system performance, sign up for our free log management tool and include these patterns listed above to automatically create tags and alerts relevant for your system. Ready to start getting insights from your applications? Try InsightOps today.

What is Syslog?

This post has been written by Dr. Miao Wang, a Post-Doctoral Researcher at the Performance Engineering Lab at University College Dublin. This post is the first in a multi-part series of posts on the many options for collecting and forwarding log data from different platforms…

This post has been written by Dr. Miao Wang, a Post-Doctoral Researcher at the Performance Engineering Lab at University College Dublin. This post is the first in a multi-part series of posts on the many options for collecting and forwarding log data from different platforms and the pros and cons of each. In this first post we will focus on Syslog, and will provide background on the Syslog protocol. What is Syslog? Syslog has been around for a number of decades and provides a protocol used for transporting event messages between computer systems and software applications. The Syslog protocol utilizes a layered architecture, which allows the use of any number of transport protocols for transmission of Syslog messages. It also provides a message format that allows vendor-specific extensions to be provided in a structured way. Syslog is now standardized by the IETF in RFC 5424 (since 2009), but has been around since the 80's and for many years served as the de facto standard for logging without any authoritative published specification. Syslog has gained significant popularity and wide support on major operating system platforms and software frameworks and is officially supported on almost all versions of Linux, Unix, and MacOS platforms. On Microsoft Windows, Syslog can also be supported through a variety of open source and commercial third-party libraries. Syslog best practices often promote storing log messages on a centralized server that can provide a correlated view on all the log data generated by different system components. Otherwise, analyzing each log file separately and then manually linking each related log message is extremely time-consuming. As a result, forwarding local log messages to a remote log analytics server/service via Syslog has been commonly adopted as a standard industrial logging solution. How does Syslog work? The Syslog standard defines three different layers, namely the Syslog content, the Syslog application and the Syslog transport. The content refers to the information contained in a Syslog event message. The application layer is essentially what generates, interprets, routes, and stores the message while the Syslog transport layer transmits the message via the network. According to the Syslog specification, there is no acknowledgement for message delivery and although some transports may provide status information, the Syslog protocol is described as a pure simplex protocol. Sample deployment scenarios in the spec show arrangements where messages are said to be created by an ‘originator' and forwarded on to a ‘collector' (generally a logging server or service used for centralized storage of log data). Note ‘relays ' can also be used between the originator and the collector and can do some processing on the data before it is sent on (e.g. filtering out events, combining sources of event data). Applications can be configured to send messages to multiple destinations, and individual Syslog components may be running in the same host machine. The Syslog Format Sharing log data between different applications requires a standard definition and format on the log message, such that both parties can interpret and understand each other's information. To provide this, RFC 5424 defines the Syslog message format and rules for each data element within each message. A Syslog message has the following format: A header, followed by structured-data (SD), followed by a message. The header of the Syslog message contains “priority”, “version”, “timestamp”, “hostname”, “application”, “process id”, and “message id”. It is followed by structured-data, which contains data blocks in the “key=value” format enclosed in square brackets “[]”, e.g. [SDID@0 utilization=“high” os=”linux”] [SDPriority@0 class=”medium”]. In the example image below, the SD is simply represented as “-“, which is a null value (nilvalue as specified by RFC 5424). After the SD value, BOM represents the UTF-8 and “su root failed on /dev/pts/7” shows the detailed log message, which should be encoded UTF-8. (For more details of the data elements of SLP, please refer to: http://tools.ietf.org/html/rfc5424) Why Syslog? The complexity of modern application and systems is ever increasing and to understand the behavior of complex systems, administrators/developers/Ops etc. often need to collect and monitor all relevant information produced by their applications. Such information often needs to be analyzed and correlated to determine how their systems are behaving. Consequently, administrators can apply data analytic techniques to either diagnose root causes once problems occur or gain an insight into current system behavior based on statistical analysis. Frequently, logs have been applied as a primary and reliable data source to fulfill such a mission for lots of reasons, some of which I've listed here: Logs can provide transient information for administrators to roll back the system to a proper status after a failure accident. E.g. when a banking system fails, all transactions lost from the main memory can be recorded in the logs. Logs can contain a rich diversity of substantial information produced by individual applications to allow administrators/developers/ops teams to understand system behavior from many aspects such as current system statistics, trend predictions, and troubleshooting. Logs are written externally by the underlying application to hard disks and external services such that by reading these log files, there will not be any direct performance impact on the monitored system. Therefore, in a production environment administrators can safely monitor running applications via their logs without worrying about impacting performance. However, a key aspect of log analysis is to understand the format of the arriving log data, especially in a heterogeneous environment where different applications may be developed using different log formats and network protocols to send these log data. Unless this is well defined, it is quite difficult to interpret log messages sent by an unknown application. To solve this issue Syslog defines a logging standard for different systems and applications to follow in order to easily exchange log information. Based on the logging protocol, Syslog helps applications effectively interpret each log attribute to understand the meaning of the log message. Ready to put Syslog into action? Try our free log management tool today.

Active vs. Passive Server Monitoring

Server monitoring is a requirement, not a choice. It is used for your entire software stack, web-based enterprise suites, custom applications, e-commerce sites, local area networks, etc. Unmonitored servers are lost opportunities for optimization, difficult to maintain, more unpredictable, and more prone to failure. While…

Server monitoring is a requirement, not a choice. It is used for your entire software stack, web-based enterprise suites, custom applications, e-commerce sites, local area networks, etc. Unmonitored servers are lost opportunities for optimization, difficult to maintain, more unpredictable, and more prone to failure. While it is very likely that your team has a log management and analysis initiative, it's important to determine if you are only responding to what the logs tell you about the past, or are you planning ahead based on the valuable log data you are monitoring and analyzing? There are two basic approaches to server monitoring: passive monitoring and active monitoring. They are as much a state of mind as a process. And there are significant differences in the kinds of value each type of server monitoring provides; each type has its own advantages and disadvantages. What is Passive Server Monitoring? Passive server monitoring looks at real-world historical performance by monitoring actual log-ins, site hits, clicks, requests for data, and other server transactions. When it comes to addressing issues in the system, the team will review historical log data, and from there they analyze the logs to troubleshoot and pinpoint issues. This was previously done with a manual pull of logs. While this helps developers identify where issues are, using a powerful modern log analysis service to simply automate an existing process is a waste. Passive monitoring only shows how your server handles existing conditions, but it may not give you much insight into how your server will deal with future ones. For example, if one of the components of the system, a database server, is likely to be overloaded when the load rate of change is reached. This is not going to be clear when server log data has already been recorded, unless your team is willing to stare at a graph in real-time, 24/7…which has been nearly the case in some NOC operations I have witnessed. What is Active Server Monitoring? The most effective way to get past these limits is by using active server monitoring. Active monitoring is the approach that leverages smart recognition algorithms to take current log data and use it to predict future states. This is done by some complex statistics (way over my head) that compare real-time to previous conditions, or past issues. For example it leverages anomaly detection, steady state analysis, and trending capabilities to predict that a workload is about to hit its max capacity. Or there is a sudden decrease in external network-received packets, a sign of public web degradation. Besides finding out what is possibly going to happen, active server monitoring also helps to avoid the time spent on log deep dives. Issues will sometimes still pass you by, and you will still need to take a deeper look, but because information is pushed to you, some of the work is already done, and you can avoid the log hunt. Oh and active monitoring can help the product and dev team from an architectural standpoint. If, for example, a key page is being accessed infrequently, or if a specific link to that page is rarely used, it may indicate a problem with the design of the referring page, or with one of the links leading to that page. A close look at the log can also tell you whether certain pages are being accessed more often than expected  —  which can be a sign that the information on those pages should be displayed or linked more prominently. Any Form of Server Monitoring is Better Than None Log analysis tools are the heart of both server monitoring approaches. Log analysis can indicate unusual activity which might slip past and already overloaded team. Another serious case is security. A series of attempted page hits that produce “page not found” or access denied” errors, for example, could just be coming from a bad external link  —  or they could be signs of an attacker probing your site. HTTP request that are pegging a server process could be a sign that a denial of service attack has begun. It is hard to make the shift from passive to active monitoring. Why? Not because you and your team are not interested in thinking ahead. But more so because many operations are entrenched in existing processes that are also reactive. And sometimes teams are just unaware that their tool can provide this type of functionality. Until one day it does it automatically for you, and you have a pleasant surprise. Active server monitoring can mean the difference between preventing problems before they get a chance to happen, or rushing to catch up with trouble after it happens. And they are the difference between a modern version of an old process, and moving forward to a modern software delivery pipeline. Ready to Make the Shift from Passive to Active Monitoring? Sign up for our free log management tool today.

12 Days of HaXmas: The Gift of Endpoint Visibility and Log Analytics

Merry HaXmas to you! Each year we mark the 12 Days of HaXmas with 12 blog posts on hacking-related topics and roundups from the year. This year, we're highlighting some of the “gifts” we want to give back to the community. And while these gifts…

Merry HaXmas to you! Each year we mark the 12 Days of HaXmas with 12 blog posts on hacking-related topics and roundups from the year. This year, we're highlighting some of the “gifts” we want to give back to the community. And while these gifts may not come wrapped with a bow, we hope you enjoy them. Machine generated log data is probably the simplest and one of the most used data source for everyday use cases such as troubleshooting, monitoring, security investigations … the list goes on. Since log data records exactly what happens in your software over time it is extremely useful for understanding what had caused an outage or security vulnerability. With technologies like InsightOps, it can also be used to monitor systems in real time by looking at live log data which can contain anything from resource usage information, to error rates, to user activity etc. So in short when used for the right job, log data is extremely powerful... until it's NOT! When is it not useful to look at logs? When your logs don't contain the data you need. How many times during an investigation have your logs contained enough information to point you in the right direction, but then fell short of giving you the complete picture. Unfortunately, it is quite common to run out of road when looking at log data; if only you had recorded 'user logins', or some other piece of data that was important with hindsight, you could figure out what user installed some malware and your investigation would be complete. Log data, by its very nature, provides an incomplete view of your system, and while log and machine data is invaluable for troubleshooting, investigations and monitoring, it is generally at its most powerful when used in conjunction with other data sources. If you think about it, knowing exactly what to log up front to give you 100% code or system coverage is like trying to predict the future. Thus when problems arise or investigations are underway, you may not have the complete picture you need to identify the true root cause. So our gift to you this HaXmas is the ability to generate log data on the fly through our new endpoint technology, InsightOPs, which enables you to  fill in any missing information during troubleshooting or investigations. InsightOps is pioneering the ability to generate log data on the fly by allowing end users to ask questions of their environment, InsightOps is pioneering the ability to generate log data on the fly by returning answers in the form of logs. Essentially, it will allow you to create synthetic logs which can be combined with your traditional log data - giving you the complete picture! It also gives you all this information in one place (so no need to combine a bunch of different IT monitoring tools to get all the information you need). You will be able to ask anything from 'what processes are running on every endpoint in my environment' to ‘what is the memory consumption' of a given process or machine. In fact, our vision is to allow users to ask any question that might be relevant for their environment such that you will never be left in the dark and never again have to say ‘if only I had logged that.' Interested in trying InsightOps for yourself? Sign up here: https://www.rapid7.com/products/insightops/ Happy HaXmas!

Announcing InsightOps - Pioneering Endpoint Visibility and Log Analytics

Our mission at Rapid7 is to solve complex security and IT challenges with simple, innovative solutions. Late last year Logentries joined the Rapid7 family to help to drive this mission. The Logentries technology itself had been designed to reveal the power of log data to…

Our mission at Rapid7 is to solve complex security and IT challenges with simple, innovative solutions. Late last year Logentries joined the Rapid7 family to help to drive this mission. The Logentries technology itself had been designed to reveal the power of log data to the world and had built a community of 50,000 users on the foundations of our real time, easy to use yet powerful log management and analytics engine. Today we are excited to announce InsightOps, the next generation of Logentries. InsightOps builds on the fundamental premise that in a world where systems are increasingly distributed, cloud-based and made up of connected/smart devices, log and machine data is inherently valuable to understand what is going on, be that from a performance perspective, troubleshooting customer issues or when investigating security threats. However, InsightOps also builds on a second fundamental premise, which is that log data is very often an incomplete view of your system, and while log and machine data is invaluable for troubleshooting, investigations and monitoring, it is generally at its most powerful when used in conjunction with other data sources. If you think about it, knowing exactly what to log up front to give you 100% code or system coverage is like trying to predict the future. Thus when problems arise or investigations are underway, you may not have the complete picture you need to identify the true root cause. To solve this problem InsightOps allows users to ask questions of specific endpoints in your environment. The endpoints return answers to these questions, in seconds, in the form of log events such that they can be correlated with your existing log data. I think of it as being able to generate 'synthetic logs' on the fly - logs designed to answer your questions as you investigate or need vital missing information. How often have you said during troubleshooting or an investigation "I wish I had logged that…”? Now you can ask questions in real time to fill in the missing details e.g. “who was the last person to have logged into this machine?” InsightOps combines both log data and endpoint information such that users can get a more complete understanding of their infrastructure and applications through a single solution. InsightOps will now deliver this IT data in one place and thus avoids the need for IT professionals to jump between several, disparate tools in order to get a more complete picture of their systems. By the way - this is the top pain point IT professionals have reported across lots and lots of conversations that we have had, and that we continue to have, with our large community of users. To say I am excited about this is an understatement - I've been building and researching log analytics solutions for more than 10 years and I truly believe the power provided by combining logs and endpoints will be a serious game changer for anybody who utilizes log data as part of their day to day responsibilities -- be that for asset management, infrastructure monitoring, maintaining compliance or simply achieving greater visibility, awareness and control over your IT environment. InsightOps will also be providing some awesome new capabilities beyond our new endpoint technology, including: Visual Search: Visual search is an exciting new way of searching and analyzing trends in your log data by interacting with auto-generated graphs. InsightOps will automatically identify key trends in your logs and will visualize these when in visual search mode. You can interact with these to filter your logs allowing you to search and look for trends in your log data without having to write a single search query. New Dashboards and Reporting: We have enhanced our dashboard technology making it easier to configure dashboards as well as providing a new, slicker look and feel. Dashboards can also be exported to our report manager where you can store and schedule reports, which can be used to provide a view of important trends e.g. reporting to management or for compliance reporting purposes. Data Enrichment: Providing additional context and structuring log data can be invaluable for easier analysis and ultimately to drive more value from your log and machine data. InsightOps enhances your logs by enriching them in 2 ways, (1) by combining endpoint data with your traditional logs to provide additional context and (2) by normalization your logs into a common JSON structure such that it is easier for users to work with, run queries against, build dashboards etc. As always check it out and let us know what you think - we are super excited to lead the way into the next generation of log analytics technologies. You can apply for access to the InsightOps beta program here: https://www.rapid7.com/products/insightops/beta-request

User Behavior Analytics and Privacy: It's All About Respect

When I speak with prospects and customers about incident detection and response (IDR), I'm almost always discussing the technical pros and cons. Companies look to Rapid7 to combine user behavior analytics (UBA) with endpoint detection and log search to spot malicious behavior in their environment.…

When I speak with prospects and customers about incident detection and response (IDR), I'm almost always discussing the technical pros and cons. Companies look to Rapid7 to combine user behavior analytics (UBA) with endpoint detection and log search to spot malicious behavior in their environment. It's an effective approach: an analytics engine that triggers based on known attack methods as well as users straying from their normal behavior results in high fidelity detection. Our conversations center on technical features and objections – how can we detect lateral movement, or what does the endpoint agent do, and how can we manage it? That's the nature of technical sales, I suppose. I'm the sales engineer, and the analysts and engineers that I'm speaking with want to know how our stuff works. The content can be complex at times, but the nature of the conversation is simple. An important conversation that is not so simple, and that I don't have often enough, is a discussion on privacy and IDR. Privacy is a sensitive subject in general, and over the last 15 years (or more), the security community has drawn battle lines between privacy and security.  I'd like to talk about the very real privacy concerns that organizations have when it comes to the data collection and behavioral analysis that is the backbone of any IDR program. Let's start by listing off some of the things that make employers and employees leery about incident detection and response. It requires collecting virtually everything about an environment. That means which systems users access and how often, which links they visit, interconnections between different users and systems, where in the world users log in from – and so forth. For certain solutions, this can extend to recording screen actions and messages between employees. Behavioral analysis means that something is always “watching,” regardless of the activity. A person needs to be able to access this data, and sift through it relatively unrestricted. I've framed these bullets in an intentionally negative light to emphasize the concerns. In each case, the entity that either creates or owns the data does not have total control or doesn't know what's happening to the data. These are many of the same concerns privacy advocates have when large-scale government data collection and analysis comes up. Disputes regarding the utility of collection and analysis are rare. The focus is on what else could happen with the data, and the host of potential abuses and misuses available. I do not dispute these concerns – but I contend that they are much more easily managed in a private organization. Let's recast the bullets above into questions an organization needs to answer. Which parts of the organization will have access to this system? Consider first the collection of data from across an enterprise. For an effective IDR program, we want to pull authentication logs (centrally and from endpoints – don't forget those local users!), DNS logs, DHCP logs, firewall logs, VPN, proxy, and on and on. We use this information to profile “normal” for different users and assets, and then call out the aberrations. If I log into my workstation at 8:05 AM each morning and immediately jump over to ESPN to check on my fantasy baseball team (all strictly hypothetical, of course), we'll be able to see that in the data we're collecting. It's easy to see how this makes employees uneasy. Security can see everything we're doing, and that's none of their business! I agree with this sentiment. However, taking a magnifying glass to typical user behavior, such as websites visited or messages sent isn't the most useful data for the security team. It might be interesting to a human resources department, but this is where checks and balances need to start. An information security team looking to bring in real IDR capabilities needs to take a long and hard look at its internal policies and decide what to do with information on user behavior. If I were running a program, I would make a big point of keeping this data restricted to security and out of the hands of HR. It's not personal, HR – there's just no benefit to allowing witch hunts to happen. It'll distract from the real job of security and alienate employees. One of the best alerting mechanisms in every organization isn't technology, it's the employees. If they think that every time they report something it's going to put a magnifying glass on every inane action they take on their computer, they're likely to stop speaking up when weird stuff happens. Security gets worse when we start using data collected for IDR purposes for non-IDR use cases. Who specifically will have access, to what information, and how will that be controlled? What about people needing unfettered access to all of this data? For starters, it's absolutely true. When Bad Things™ are detected, at some point a human is going to have to get into the data, confirm it, and then start to look at more data to begin the response. Consider the privacy implications, though; what is to stop a person from arbitrarily looking at whatever they want, whenever they want, from this system? The truth is organizations deal with this sort of thing every day anyway. Controlling access to data is a core function of many security teams already, and it's not technology that makes these decisions. Security teams, in concert with the many and varied business units they serve, need to decide who has access to all of this data and, more importantly, regularly re-evaluate that level of access. This is a great place for a risk or privacy officer to step in and act as a check as well. I would not treat access into this system any differently than other systems. Build policy, follow it, and amend regularly. Back to if I was running this program. I would borrow pretty heavily from successful vulnerability management exception handling processes. Let's say there's a vulnerability in your environment that you can't remediate, because a business critical system relies on it. In this case, we would put an exception in for the vulnerability. We justify the exception with a reason, place a compensating control around it, get management sign off, and tag an expiration date so it isn't ignored forever. Treat access into this system as an “exception,” documenting who is getting access, why, and define a period in which access will be either re-evaluated or expire, forcing the conversation again. An authority outside of security, such as a risk or privacy officer, should sign off on the process and individual access. Under what circumstances will this system be accessed, and what are the consequences for abusing that access? There need to be well-defined consequences for those that violate the rules and policies set forth around a good incident detection and response system. In the same way that security shouldn't allow HR to perform witch hunts unrelated to security, the security team shouldn't go on fishing trips (only phishing and hunts). Trawls through data need to be justified. This is for the same reasons as the HR case. Alienating our users hurts everyone in the long run. Reasonable people are going to disagree over what is acceptable and what is not, and may even disagree with themselves. One Rapid7 customer I spoke with talked about using an analytics tool to track down a relatively basic financial scam going on in their email system. They were clearly justified in both extracting the data and further investigating that user's activity inside the company. “In an enterprise,” they said, “I think there should be no reasonable expectation of privacy – so any privacy granted is a gift. Govern yourself accordingly.” Of course, not every organization will have this attitude. The important thing here is to draw a distinct line for day to day use, and note what constitutes justification for crossing that line. That information should be documented and be made readily available, not just in a policy that employees have to accept but never read. Take the time to have the conversation and engage with users. This is a great way to generate goodwill and hear out common objections before a crisis comes up, rather than in the middle of one or after. Despite the above practitioner's attitude towards privacy in an enterprise, they were torn. “I don't like someone else having the ability to look at what I'm doing, simply because they want to.” If we, the security practitioners, have a problem with this, so do our users. Let's govern ourselves accordingly. Technology based upon data collection and analysis, like user behavior analytics, is powerful and enables security teams to quickly investigate and act on attackers. The security versus privacy battle lines often get drawn here, but that's not a new battle and there are plenty of ways to address concerns without going to war. Restrict the use of tools to security, track and control who has access, and make sure the user population understands the purpose and rules that will govern the technology. A security organization that is transparent in its actions and receptive to feedback will find its work to be much easier.

Log Search Simplified

Hi, I'm Laura, UX Designer at Logentries and today I'm going to discuss how just about anyone can use Logentries to search and analyze their log data no matter what their job title or technical skill level. What is Logentries? At Logentries, the team works…

Hi, I'm Laura, UX Designer at Logentries and today I'm going to discuss how just about anyone can use Logentries to search and analyze their log data no matter what their job title or technical skill level. What is Logentries? At Logentries, the team works tirelessly to provide an easy to use log management service that allows users to stream their logs from just about anything. Logentries can accept data from almost any device that generates log data, including servers, applications, firewalls and routers. Really, any data, from any device and in any kind of format. These log events are automatically collected and sent to one secure location where users can quickly search and visualize their data to find out all they need to know. Typically, Logentries is used by DevOps and Developers while they are busy debugging, monitoring and troubleshooting. More recently, as Logentries has become part of Rapid7 information security and analytics solutions, the power of log search has grown to include a new variety of users within information security teams and IT. These professionals use Logentries search to help solve security problems, investigate incidents and help maintain compliance. So IT guys get to have all the fun? Not quite. We have a growing number of users from non-technical backgrounds who are hot on their heels. Businesses and marketing teams have recognized the potential of using Logentries to monitor behaviors, identify patterns and gather all types of interesting information to help focus their business goals or marketing campaigns. The basics of performing a search So you're not a DevOps master or some kind of IT guru? No problem. I'm going to take you through the basics of performing a search in Logentries using our very own search language LEQL (Logentries Query Language). Using this simple SQL-like language you can extract data hiding deep in your logs. Now that you know the basic query format, we will take a look at putting it into practice. For example, let's take myself, a UX Designer who wants to design solutions for an improved user experience within Logentries. But where do I start? Where do I focus my energies? First I want to discover the most popular or core features in Logentries. Easy. I can do this by using an application library such as node.js, .Net or Java libraries which allow you to log straight from the front end of your application. You can find this in the "Add a log" page and it is a quick and easy set up process. You can find a more detailed set up process and tips in the blog post "A different way to log your website usage". Your developer will help you embed the necessary code into your site and create listeners on elements you would like to track like buttons, links, pages and features. Once set up it can track metrics such as usage trends, activity, behaviors and engagement times across your application. Tracking most clicked features Now that I have the information that is important to me logging to my Logentries account, I can write some LEQL to query this data. First, navigate to the Logentries Log view. This is where logs feed into, where we can select logs and where we can search. The query search bar has two modes. "Advanced" is the mode an experienced user would use to write full LEQL queries. "Simple" mode is for users who are newbies or just need a helping hand in building their query. We are going to use simple mode to get you started. We are interested in learning how often a user visits a feature. We can find this out by tracking how often the button or link to this feature is clicked. First comes the "where" field followed by the "groupby" and "calculate" fields. Next is an icon to allow you to save your query and a time picker to allow you to select a particular time range to search. The mode can be toggled between simple and advanced via the switch mode link. Step 1 We want to search for click events so we search the keyword "clicked." Keyword search will work on all log entries regardless of their format and are case sensitive by default. This will give you a result of all the log entries containing click events. You can see the keyword highlighted in yellow. Step 2 That's great, but we want to do a more granular search. We need to break this up into exactly what features were clicked. Let's group by "features." Once this key is added to your query the calculate function automatically selects calculate(count). This returns a count of the matched search. Your results are returned as a table with all of your search criteria listed and a visualization of your data. Now you can see all the features that have been clicked and how often they were clicked giving an indication into what features users spend most of their time using. This can help me prioritize where I should focus my efforts to work towards making these features better for our users. Endless possibilities This is just one example of how you can use Logentries search to gain insight into your application usage. You could potentially find out lots of other interesting data like what screen resolution your users use most or what the most popular browsers are. You could find out how often or at what point a user drops off or cancels from a workflow such as a sign up process or a create wizard. Or even how long a user hovers on a button or how long they spend on a particular page. The possibilities are endless. To learn more about search check out our docs or start a 30 day free trial. Thanks for reading! Laura EllisUX Designer, Rapid7

Using Log Data as Forensic Evidence

This is a guest post by Ed Tittel. Ed, a regular contributor to blog.logentries.com, has been writing about information security topics since the mid-1990s. He contributed to the first five editions of the CISSP Study Guide (Sybex, 6e, 2012, ISBN: 978-1-119-31427-3) and to…

This is a guest post by Ed Tittel. Ed, a regular contributor to blog.logentries.com, has been writing about information security topics since the mid-1990s. He contributed to the first five editions of the CISSP Study Guide (Sybex, 6e, 2012, ISBN: 978-1-119-31427-3) and to two editions of Computer Forensics JumpStart (Sybex, 2e, 2011, ISBN: 978-0-470-93166-0), and still writes and blogs regularly on security topics for websites including Tom's IT Pro, GoCertify.com, CIO.com, and various TechTarget outlets including SearchSecurity.com. Learn more about or contact Ed through his website. Working with computer logs is something of an ongoing adventure in discovery. The data from such logs is amenable to many uses and applications, particularly when it comes to monitoring and maintaining security. But even after a security breach or incident has occurred, log data can also provide information about how an attack was carried out, the IP address (or addresses) from which it originated, and other packet data from network communications that could be used to identify the source of attack and possibly also, the identity of the attacker. This means presenting log data in a court of law as evidence to support specific allegations or accusations. How does that work? Documentary or Digital Evidence and the Hearsay Rule In legal matters, a special consideration called the hearsay rule normally applies to evidence that may be admitted in court for a judge or a jury to consider in assessing or disproving the truth of various assertions, or in deciding guilt or innocence for an accused party. The hearsay rule states that “testimony or documents which quote persons not in court are not admissible.” This provision in the law is intended to prevent information provided by third parties who cannot be questioned about their testimony or documents, or whose credibility or veracity can be neither proven nor impeached, from affecting the outcome of a decision of guilt or innocence. For the layperson, it's clearly tied to the notion that the accused has the right to face and question those who accuse him or her in the courtroom as the legal process works to its final conclusion. But what about digital evidence, then? Computer logs capture all kinds of information routinely, either at regular intervals or in response to specific events. Because an accused party cannot face or question software in the courtroom, does this mean that logs and other similar computer-generated data are not admissible as evidence? Absolutely not, but there are a few “catches” involved. The Business Records Exception… As is happens there are some kinds of information and documents that are NOT excluded by the hearsay rule as explained in the Federal Rule of Evidence 803(6). Most specifically, “Records of regularly conducted activity,” are excluded. These are defined in the afore-cited publication as “A memorandum, report, record, or data compilation, in any form, of acts, events, conditions, or diagnoses, made at or near the time by, or from information transmitted by, a person with knowledge, if kept in the course of a regularly conducted business activity, and if it was the regular practice of that business activity to make the memorandum, report, record or data compilation, as shown by the testimony of the custodian or other qualified witness, …, unless the source of information or the method or circumstances of preparation indicate lack of trustworthiness. The term ‘business' as used in this paragraph indicates business, institution, association, profession, occupation, and calling of every kind, whether or not conducted for profit.” Whew! That's a lot to digest, but here is what it means: As long as the party that wishes to use log data as evidence can show that it routinely collected log records before (and during) the events or activities captured in those logs, they should be admissible as evidence in court. A responsible person would have to be able to truthfully testify that logging was already in use by that time, and that the log data presented as evidence is a true and faithful (that is, unaltered) copy of the original data logged at the time the alleged events or activities occurred for that evidence to stand. But because logs are designed to provide a record of events and activities it will be close to impossible for the other side of the case to argue such evidence as inadmissible per se. As long as you can produce one or more credible witnesses, with supporting documentation (memos, file dates, and so forth) to show that logging started some time before the alleged events or activities occurred, and can provide records to show that the log data presented in court is identical to what was originally captured and has not been altered since, your logs can indeed tell their story in the courtroom. Note: my thanks to Neil Broom, President of the Technical Resource Center, and a regular forensics examiner and expert witness on digital forensics, and an author of Computer Forensics JumpStart for his clear and helpful guidance in explaining log data as legal evidence in the courtroom. Logentries by Rapid7 makes it simple to collect, analyze and ensure the security of your log data. Start centralizing your log data today with a free 30-day Logentries trial. Click here to get started.

Nexpose Logging Analytics using LogEntries

This blog shows how to use the power of LogEntries Search and Analytics to monitor your Nexpose installation. LogEntries has joined the Rapid7 family and offers several powerful capabilities to search, analyze, monitor and alert on your Nexpose installation. LogEntries is also super easy to…

This blog shows how to use the power of LogEntries Search and Analytics to monitor your Nexpose installation. LogEntries has joined the Rapid7 family and offers several powerful capabilities to search, analyze, monitor and alert on your Nexpose installation. LogEntries is also super easy to set up and maintain. I spent about five minutes getting it running. The Nexpose engineering team made it very easy by enabling the log4j appender in every installation of Nexpose. All you have to do is follow these steps to get up and running. Set up your free trial Set up a free trial on LogEntries (https://logentries.com/) by clicking on the "Start a Free Trial" button: Generate tokens for system logging You can create logging tokens by clicking on "Add a Log" and choosing the "Java" icon in the "Libraries" section and then click on "Create Log Token" at the bottom of the screen. Create as many as you want appenders (see next step). You can have an appender for every Nexpose log if you want: Configure Nexpose Logging In your Nexpose installation, copy the logentries appenders in the console's logging configuration located in /opt/rapid7/nexpose/nsc/conf/logging.xml (near the bottom of the file) and paste them into the user-log-settings.xml file in the same directory. Make sure to replace the ${logentries-*-token} with the actual token from your logentries account that you created above Each appender can have it's own token so they can be tracked using different logs in logentries. Here is an example: <appender name="le-nsc" class="com.logentries.logback.LogentriesAppender"> <Token>123725d5-10df-4aa7-b683-3e8c71251b2c</Token> <Debug>False</Debug> <Ssl>False</Ssl> <facility>USER</facility> <encoder> <pattern>${logFormat}</pattern> </encoder> </appender> Unlock the power of LogEntries Restart Nexpose and you will see logs flowing into your LogEntries account. Now you can start using all the great features of LogEntries including Live Tail, Saved Queries, Alerts, and Tagging to manage your Nexpose console. Here are some examples: Initial Log View This view will appear as soon as you click on the Log Set that you want to view. In my case, "Demo Set" is the log set that I used when creating my account and hooking up Nexpose. From here you can search and filter to find log entries of interest: Live Tailing Live Tailing is a great feature that allows you to debug or monitor issues as they are happening: Creating Tags and Alerts Tags and alerts allow you to label specific log lines based on regular expressions and also alert if anomalies occur: Wrap Up Also check out how to do the same thing with Metasploit Pro in Securing Your Metasploit Logs. I hope you have found this helpful and please share any feedback such as alerts, dashboards, or other useful tips and tricks that you have found when using Nexpose with LogEntries.

Seven Ways InsightIDR Helps Maintain PCI Compliance

“Compliance is king.” This is a familiar saying for any company that processes credit card transactions, where being compliant with the Payment Card Industry Data Security Standard, or PCI DSS, reigns supreme. Any entity that stores, processes, or transmits cardholder data must abide by the…

“Compliance is king.” This is a familiar saying for any company that processes credit card transactions, where being compliant with the Payment Card Industry Data Security Standard, or PCI DSS, reigns supreme. Any entity that stores, processes, or transmits cardholder data must abide by the requirements, which serve as best practices to securing your cardholder data environment (CDE). Nexpose and Metasploit have been designed to directly help your team meet PCI DSS, as well as comply with many other compliance standards. Created by security responders, Rapid7 InsightIDR also ties in with PCI, including helping you meet Requirement 10: Tracking and monitoring all access to network resources and cardholder data. InsightIDR joins your security detail to detect the top attack vectors behind breaches, speed up incident investigations, and help you escape the drudgery of security data management. Here are a few of the PCI requirements that InsightIDR can help your security team manage, ranging from monitoring access to your CDE and exposing risky user behavior, to fast and comprehensive incident investigations across the entire organization. To see it in action, see our 20-minute on-demand demo. Requirements 5.1 & 5.2: InsightIDR scans all endpoints for malware and identifies risky user behavior, including compromised user accounts, anomalous admin activity, and lateral movement. This endpoint visibility is accomplished for all systems through a blend of endpoint scans and the continuous Insight Agent. Requirements 6.4.1 & 6.4.2: You can monitor multiple separated environments, define network zones and alert you if access policies are violated. As an example, an organization could set a policy that no users in the “developers” group should access the network zone “PCI Production,” ensuring InsightIDR alerts them on any such violations. Requirements 7.1, 7.3: After flagging systems in your CDE as restricted assets, InsightIDR will alert you on any change in behavior. This includes suspicious authentications, users with unexpected privilege escalations, and even approved users remotely accessing the CDE from a new source asset. This detects unauthorized access, user risk, and enforces policies set by your security team. **Requirements 8.1, 8.2.4, 8.5: **InsightIDR alerts on brute forcing, pass-the-hash, and other password guessing attempts by running behavior analytics on event logs and through Intruder Traps, such as honey users and honey credentials. Requirement 10: InsightIDR is your complete solution to track and monitor all access to network resources and cardholder data. This starts with aggregation and search across any of your log files. In addition, all network activity is directly correlated to the users and assets behind them. During incident investigations, the security team can bring together log search, real time user activity, and endpoint interrogation into a single Super Timeline (see below). No more parsing through disparate log files, jumping between multiple solutions for investigations, or retracing user activity across IPs, assets, and services. Requirement 11.4: InsightIDR identifies malicious behavior earlier in the attack chain, the steps required to breach a company. Through a combination of user behavior analytics and purpose-built Intruder Traps, InsightIDR detects the top attack vectors behind breaches, including phishing, compromised credentials, and malware. Requirement 12.3, 12.5, 12.10: InsightIDR can aggregate, search, and attribute logs and alerts from Intrusion Detection/Prevention Systems (IDS/IPS) and Firewalls to the users and assets behind them. For example, with one search, the security team can identify the users generating the most IDS/IPS alerts. InsightIDR was built hand-in-hand with security teams to be the SIEM solution you always wanted, armed with the detection you will always need. It combines learnings from the Metasploit project, our penetration testing teams, and tested User Behavior Analytics (UBA) that hundreds of organizations benefit from today. You can finally get visibility and detection while meeting PCI compliance without it becoming a second full-time job. Learn more about how Rapid7 can help your team meet PCI, or sign up for a free guided demo!

Securing Your Metasploit Logs

Original post from Logentries found here: Securing your Metasploit Logs by Justin Buchanan Metasploit, backed by a community of 200,000 users and contributors is the most impactful penetration testing solution on the planet. With it, uncover weaknesses in your defenses, focus on the highest…

Original post from Logentries found here: Securing your Metasploit Logs by Justin Buchanan Metasploit, backed by a community of 200,000 users and contributors is the most impactful penetration testing solution on the planet. With it, uncover weaknesses in your defenses, focus on the highest risks, and improve your security outcomes. Your Metasploit Pro console produces a lot of important logs. It is essential to be able to review these logs, alert on them, and keep them secure. Why should I monitor these logs? The logs produced by your Metasploit Pro console are helpful when troubleshooting, and also for monitoring the usage of the Metasploit product. Metasploit Pro is impressively powerful, which also makes it crucial to closely monitor the usage. Unfortunately, you must always plan fo the worst possible scenario, including the potential for a Metasploit user to alter the logs created by the console to hide their actions. Sending these logs to a secure central location in real-time, can ensure that they remain unaltered and easy to review. What and where are the Metasploit Pro Logs? The list below details all of the logs created by your Metasploit Pro console and where they are saved. Your installation root directory may vary; by default the installation root for Linux is: /opt/metasploit and for Windows: C:\metasploit $INSTALL_ROOT/apps/pro/nginx/logs/error.log – Console web server error log $INSTALL_ROOT/apps/pro/nginx/logs/access.log – Console web server access log $INSTALL_ROOT/apps/pro/ui/log/production.log – Rails (ruby) log $INSTALL_ROOT/apps/pro/engine/config/logs/framework.log – Metasploit Framework log $INSTALL_ROOT/apps/pro/engine/prosvc_stdout.log – Metasploit RPC output log $INSTALL_ROOT/apps/pro/engine/prosvc_stderr.log – Metasploit RPC error log $INSTALL_ROOT/apps/pro/tasks – Task logs $INSTALL_ROOT/apps/pro/engine/license.log – License log As a best practice, all of the above logs should be sent to a secure, off-site, location for storage and analysis. For the purposes of this post we will focus on the three most imperative logs: tasks framework.log access.log The tasks directory The tasks directory provides text files detailing all of the actions taken by all Metasploit users.  It will record any exploit that is run, the creation of a listener, establishment of a pivot, and any other action taken through the console. Configure the Logentries Agent To capture the log data saved to the tasks directory first ensure that you have installed the appropriate Logentries Agent on the Metasploit Console machine. The Logentries Agent can automatically identify and forward the newest log file written to a directory by using a wildcard configuration. For the Linux Agent issue the following command to follow the tasks directory: sudo le follow '/opt/metasploit/apps/pro/tasks/*.txt' and with the Windows Agent: AgentService.exe follow c:\metasploit\apps\pro\tasks\*.txt Always remember to restart the Logentries service after making changes to its configuration. View in Logentries Now as new tasks are written to the directory on your console server you can see them stream into Logentries in real time, creating an immutable offsite backup of these important audit trails. framework.log framework.log is your best friend when you are trying to troubleshoot an issue you are encountering with Metasploit. All the logged errors are saved here.  When you dig into this log you will gain insight into which exploits failed, and for what reasons, as well as general stack traces. Configure the Logentries Agent In this case, because framework.log is just a single file, there is no need for special configuration. The command to follow this file with the Linux Agent would simply be: sudo le follow /opt/metasploit/apps/pro/engine/config/logs/framework.log access.log The final log discussed here is the NGINX access.log produced by the Metasploit console. The information available in this log is imperative to maintain complete audit trails of all actions taken in the console. This log will contain every request made to the web interface including the ip address of the requester, making it invaluable in an investigation. Metasploit's NGINX server is configured to log in combined log format, and as a result Logentries will be able to perform in-depth analysis on these logs with ease.  The video below provides a tutorial on using the advanced search functionalities of Logentries to query an Apache access.log, all the same features and functionality will be available with this NGINX access.log. Ready to secure your Metasploit logs? Give it a try by creating a free Logentries account today!

If You Work In Operations, Your Security Team Needs The Logs, Too

This post is the final in a series examining the roles of search and analytics in the incident-detection-to-response lifecycle. To read the previous six, click one, two, three, four, five,or six. As a final discussion topic around search and analytics, it is really important…

This post is the final in a series examining the roles of search and analytics in the incident-detection-to-response lifecycle. To read the previous six, click one, two, three, four, five,or six. As a final discussion topic around search and analytics, it is really important to talk about the different teams involved, who all have different questions to ask of the same overall data. Roles and titles can vary greatly, but for simplicity, I'll refer to the three primary groups as ‘IT Ops' [for anyone looking to maintain servers and software required for the employees and partners of the organization], ‘DevOps' [for anyone focused on the availability and quality of servers and software for external customers], and ‘security' [for the individuals looking to prevent and detect malicious activities]. Let's talk about the order in which these teams generally develop in an organization and how that leads to politics between teams and the duplication of efforts, using the now-classic Lethal Weapon series. IT operations monitored logs first, but not last For our purposes here, I'm going to consider the IT Ops team the Roger Murtaugh of the company because they were likely there first, so they were around to acquire the necessary tools and lay out “the way things are done”. When the first email servers were stood up and various infrastructure deployed, IT Ops were manually checking the health of each system and receiving floods of automated emails with complex error codes when the slightest problem inevitably occurred. This was never an easy job, but the gradual addition of ticketing systems, helplines, email distribution lists, and pagers to their processes made it possible to handle the day-to-day needs and off-hours emergencies. Centralized log management solutions were then brought in to simplify the entire monitoring and troubleshooting process, and the first automated trend charts around CPU usage, errors, and bandwidth made drastic improvements on initial notifications. Troubleshooting could finally be done from a single place, so IT Ops was happy with the way things were working. If your organization has custom-developed software, as many do today, at least one DevOps team has gotten sick of similarly SSHing into production systems to verify a new release is functional or track down a bug. For the IT Ops team, realizing this new team is looking to implement a log management solution was probably a similar shock to when Murtaugh first learned he had to work with that loose cannon, Riggs, who did things his own way. Given that change can be scary, providing high-level permissions to access your prized tools and letting another team do things “their way” both feel inherently risky. Even once these teams come to an agreement to either share a solution or work independently, they are typically minimizing very specific risks to the business with little thought to the risks with which the security team must obsess. You're mistaken if you've assumed no one else needs your logs One of my favorite cognitive biases to consider, because we can all rarely avoid it, is the "focusing illusion" best summarized as "nothing in life is as important as you think it is while you are thinking about it". This is relevant here because when you are paid, trained, and thanked for using the tools and data at your disposal for very clear purposes, you are unlikely to realize just how much value is in there for others with very different questions or that sometimes those other questions could be more important to the company than your own. Whether or not Murtaugh and Riggs in your organization have become partners, they most likely see security as a pest [like Leo Getz] or like Internal Affairs [Riggs's future wife, Lorna Cole] because the effort to reduce the risk of attacks and data leaks often requires very different efforts than keeping applications functional and available for customers, be they internal or external. Meanwhile, in reality, just as all of these characters had the same goal [of stopping serious criminals], the security team has the same overarching goal as IT Ops and DevOps: eliminating reasons the rest of the company would be unable to accomplish its goals [and yes, I remember that Leo Getz was originally a money-launderer]. If you're in IT Ops or DevOps, your security team wants you to know that they would really like access to your logs. It would help them reduce the risks they worry about, even if you don't completely understand why. It's healthier not to differentiate between “logs” and “security logs” Microsoft may have biased us all by labeling some logs as "security logs" and others for different purposes, but the easiest example of non-security logs with value to security are DHCP logs; if your security team doesn't have access to DHCP lease grants and releases, it can be very difficult to find out which machine is operating at any given IP address in an incident investigation. The same goes for machine data the DevOps team considers only relevant to them - if you can see who made the last change to the production servers before they went down, you can also see whose account might have been stolen and used to change settings preventing outside access to production systems. If you never ask the security team if your logs would be useful, you are likely to make the wrong assumptions. Besides, they should absolutely have the security tools capable of ignoring anything they don't need. Getting a single solution for all parties to use can save you a lot Though the security team should have the tools that help them find what they care about in your logs, far too many organizations have three different tools for three different teams when there could be a single one. You don't need pivot tables or fancy finance visualizations to realize that buying a log management solution for IT Ops, a separate search solution for DevOps, and also a SIEM for the security team is a very costly way to solve different problems with the same data. Even if you forget about the three different purchases, failing to take advantage of discounted tiers of pricing, you can see that three different groups have to manage three different solutions focused on search and analytics which only differ by the questions they ask and use cases to which they apply. There may have been technology challenges preventing this before, but you should demand better now. Ask for real-time search and analytics flexible enough to satisfy Murtaugh's, Riggs's, Getz's, and Cole's problems. You all want to analyze your data to remove anything which could possibly hinder your organization's progress. If you are a part of the IT Ops or DevOps teams, you can start a free trial of Logentries here and search your data within seconds.If you are on the security side of things, check out an InsightIDR demo.

Logentries Joins the Rapid7 Family

I'm very excited today to join the Rapid7 family. The acquisition is good news for Logentries customers, Rapid7 customers and all of our employees.  It means that great minds and innovative technology have come together to solve some of our thorniest IT and security…

I'm very excited today to join the Rapid7 family. The acquisition is good news for Logentries customers, Rapid7 customers and all of our employees.  It means that great minds and innovative technology have come together to solve some of our thorniest IT and security challenges.The Logentries team has been on a mission over the last few years -- Revealing the Power of Log Data to the World. While pursuing our mission, I am often asked why log data has become so valuable. The answer is simple: look at the world around us. Cloud computing has emerged as a disruptive force, mobile and internet connected devices are increasing exponentially, and IT systems are widely distributed and increasingly siloed.These trends have changed the nature of what it means to build, deploy, manage and secure the systems we all rely on for our daily lives. And when systems are breached, or closer investigation into potential issues is warranted, it's the log data that is often the source of truth.Rapid7 is dedicated to solving the security data and analytics challenge; Logentries is dedicated to solving the machine data search and analytics challenge. Uniting to form one team—uniquely positioned to offer the broadest and deepest solutions for searching and analyzing IT and security data—is the next logical iteration.It is in this spirit that I look forward to working as a member of the Rapid7 team. Together we will embrace the data challenge, and help our customers and partners better understand, manage and protect their IT infrastructure.Rapid7 is committed to continuing the development of our products.  The company has a strong reputation for effectively partnering with its customers, delivering continuous innovation and creating solutions with impact.  Sounds a lot like Logentries! You can learn more about the acquisition from the press release, this blog from Rapid7's CEO, Corey Thomas, or by joining our conference call today at 5:00 p.m. Eastern Time. The call will be accessible by telephone at 888-223-4580 (domestic) or 303-223-2683 (international). A webcast replay will be available at http://investors.rapid7.com until October 16, 2015.Andrew BurtonCEO, Logentries

Why we're welcoming Logentries to the Rapid7 family - a story of data and analytics

Those that follow Rapid7 will know that we talk a great deal about our vision of delivering security data and analytics to our customers to enable an active, analytics-driven approach to cyber security. I'm excited to let you know that today we're making an important…

Those that follow Rapid7 will know that we talk a great deal about our vision of delivering security data and analytics to our customers to enable an active, analytics-driven approach to cyber security. I'm excited to let you know that today we're making an important addition to the Rapid7 family that will help us advance this vision even further… we are acquiring the world-class, cloud-based log management and search technology company, Logentries.Organizations need real mastery of the information in their IT environment so they can better understand and respond to risk. Being able to easily search machine data and logs is a key means of doing this, enabling security pros to dig deeper for a better view of their security posture and to perform detailed investigations and forensics on security incidents.It's a great complement to our current capabilities for incident investigation, including our ability to tie events back to the specific users and assets involved, in a matter of minutes, and to seamlessly build an incident timeline.  Today, our customers tell us that with our UserInsight solution, they can complete an investigation which previously required two days in only minutes.  We plan to bring that kind of power to our customers with our new search capabilities as well.  We are committed to helping you solve problems quickly so you can focus on what matters – making your organization more secure.Logentries' technology enhances our data collection by providing our customers with really fast, scalable search, so you can find the answers you need more quickly and efficiently.Of course, when acquiring a company, great technology is only part of the picture -- even more important are all the people who make the company great.  Corporate culture is very important to Rapid7; we strongly believe one of the measures for success in a move like this is making sure the cultures mesh, and mesh well. At the core of our corporate culture is a strong commitment to putting customers first, and a dedication to driving innovation. We were impressed and excited to find that Logentries shares these core values.Another significant area of alignment in our approach is our mutual commitment to supporting a community of free users, and enabling customers to try the technology before they buy.  This has been a core tenet for Rapid7 for years, and underpins our support of the Metasploit Framework, as well as free versions and trials of Nexpose and AppSpider. Similarly, Logentries has a vast community of 50,000 for its free version, which you can try here. We believe that through these many areas of alignment we will continue to build something truly great. I am so excited to welcome the Logentries team to our family, and I am looking forward to tackling new customer challenges together in the future.Please join me in extending a very warm welcome to our new team members in Boston, MA and Dublin, Ireland!You can learn more about the acquisition from our press release, this blog from Logentries CEO, Andrew Burton, or by joining our conference call today at 5:00 p.m. Eastern Time. The call will be accessible by telephone at 888-223-4580 (domestic) or 303-223-2683 (international). A webcast replay will be available at http://investors.rapid7.com until October 16, 2015.Corey ThomasPresident & CEO, Rapid7

Q & A from the Incident Response & Investigation Webcast: "Storming the Breach, Part 1: Initial Infection Vector"

The recent webcast “Storming the Breach, Part 1: Initial Infection Vector”, with Incident Response experts Wade Woolwine and Mike Scutt sparked so many great questions from our live attendees that we didn't have time to get through all of them! Our presenters took the time…

The recent webcast “Storming the Breach, Part 1: Initial Infection Vector”, with Incident Response experts Wade Woolwine and Mike Scutt sparked so many great questions from our live attendees that we didn't have time to get through all of them! Our presenters took the time to answer additional questions after the fact... so read on for the overflow Q&A on tips and tricks for investigating common breach scenarios: Q. “What do you recommend for sandboxing the browser?” A. I'm a huge fan of Sandboxie, which is free for personal use. Sandboxie allows users to launch individual applications within a comprehensive application sandbox, and provides granular control over the application's access to the underlying operating system.  In enterprise environments, we often recommend that clients implement Microsoft's Enhanced Mitigation Experience Toolkit (EMET). IT and security personnel can implement EMET protection for any application via group policy, allowing the implementation of several anti-exploitation techniques at scale. EMET does a fantastic job of mitigating many browser-based exploits, malicious document exploits, and can be tailored to your environment.  We recommend enforcing EMET for web browsers, Flash, Acrobat, and Office/Productivity suites, as they are commonly-targeted platforms.  Q. “A lot of this information assumes that you know when the malware was dropped in a reasonable window.  For a company that gets contracted to do investigations is there a good way to efficiently identify that window that the malware was dropped?” A. This is a fantastic question, with a not-so-fantastic answer (though one I give often): it depends.  For this example I will briefly walk through a portion of my analysis methodology for Windows compromises. Note: each investigation is different, so I stress “incident response methodology” and critical thinking when training new analysts. For each incident you respond to, consider the following questions: “What evidence do I have that indicates the system was compromised?” – Use this as your starting point. If a user received a pop-up message when logging in, there is likely persistent malware on the system that a tool such as Sysinternal's AutoRuns utility will help you identify the sample. If responding to a live system, look through volatile data to determine whether there are malicious processes running on the system, unusual DLLs loaded into processes, unusual open handles, and connections to odd or known bad domains/IPs.  Use this data to determine where the malware is on disk or in the registry, and use the associated timestamps from those sources to assist in building a timeline of activity. If the initial alert came from AV / SIEM / IPS alerts, use that time period to temporarily reduce the scope of data that you're looking at, and focus on identifying newly created files (especially PE files), modified registry keys, and event log entries. “What is the earliest evidence of compromise on the system?” – Once you've identified evidence to support the system being compromised, consider how the system was compromised.  At some point either an attacker or malicious executable code interacted with the system. Work back from your earliest piece of evidence, and timeline logs, file system activity, registry activity, etc, until you've identified a potential method of ingress.  Ingress evidence often takes the form of an initial logon (for interactive attackers moving laterally / transferring files), Java or Flash artifacts from a user visiting a malicious site (or a legitimate site hosting malicious code), a user opening a malicious document, etc.  These are a few steps that help me find my bearing when responding to a breach, and are not comprehensive.  There are a number of excellent Incident Response and OS internals books on the market that will provide a greater level of detail on Incident Response methodologies. Focus on answering those questions when initially responding to a compromised system.  It is seldom enough to simply determine whether there is malware on a system, identifying the initial ingress method / root cause of the compromise is critical to ensuring appropriate remediation and mitigation strategies are deployed for the compromised system and all vulnerable systems on the network Q. “On the web application side, does it help to use IIS IP filtering to prevent malicious attacks from the outside?” A. IP filtering is useful as a temporary remediation strategy when dealing with a novice attacker, however I do not recommend relying solely on this as a defensive measure to prevent web application compromises.  Many attackers will use proxies such as TOR, other compromised systems, or larger infrastructure during an attack. I recommend following a “defense-in-depth” measure for protecting internet facing systems, to include placing a firewall tailored to your application (i.e. a Web Application Firewall), and an intrusion prevention system between your system and the internet.  The system itself should run a host-based intrusion prevention application, a file integrity monitor, a host-based firewall, and the application should be using the most restrictive permissions possible.  The system should be as isolated as possible from internal systems, and should be monitored carefully for anomalous behavior. Consider the following: if an attacker was attacking your application whilst using multiple TOR IPs and successfully compromised the application – is the application running on the system with elevated privileges? Can the attacker access critical files such as a database connector configuration file?  We frequently respond to web server breaches, and often the underlying applications are running either with elevated privileges or the application user has access to core configuration files.  Once an attacker obtains the configuration files it is trivial for them to connect to backend databases and dump the contents. Q. “For web servers, do you see specific CMOs/hosting platforms more targeted than others?  If so, is targeting because of popularity or insecurity?  Ex: Wordpress or Joomla.” A. In opportunistic attacks, absolutely.  Most of the common content management systems on the market today are heavily targeted by automated scanners or opportunistic attackers, as are most of the common plugins. In targeted attacks, we've found that it doesn't matter whether the client is running a well-known CMS or something they've developed in-house, as the attackers will diligently reconnoiter the application and identify methods of exploitation.  My mitigation strategies for these attacks remains the same: follow defense-in-depth best practices, patch your CMS and plugins, and ensure that the web application user is running with least-privileges. Q. “Given that we have to isolate individual machines to mitigate the risk of spreading infection, it doesn't seem feasible to install diagnostic tools or VM platforms on every machine in an office. Would it be more feasible to carry around mobile platforms, like a collection of diagnostic tools stored on USB drives, and are there any risks of the mobile diagnostic tool contracting and spreading that infection?” A. In these situations, I recommend either using a live-cd with the utilities you need, or purchasing a USB device with a write-block switch.  In large environments, consider creating an isolated VLAN to place the compromised systems on, alongside an analysis machine booted from a Live-CD.  This allows you to connect to the remote machine, perform your data analysis and remediation, and maintain a clean image remotely.  Note: I do not suggest this technique for any work that may be used for litigation purposes. This method is not forensically sound in any way J. Q. “You mentioned whitelist websites. How to effectively do when sites hosted around the world now. Also with akami and amazon being used to host, IP ranges constentaly change. Would proxy be best or blocked sites/regions via firewall?” A. Whitelisting is typically only a component of the recommendations we have for mitigating browser based attacks. Specifically only allowing known good sites from the browsers in your enterprise will raise the barrier of entry for attackers. Additionally, we also recommend that customers run an intrusion detection and/or prevention system at the internet egress to alert on potential patterns in traffic that could provide some investigative leads. Q. “How much of a concern do you have, during these investigations, is there much focus on collecting data in ways that can be used in litigation / prosecution? Or are you mostly concerned with finding breaches and mitigating, etc? Thanks much.” A. This heavily depends on the client and the type of engagement.  In situations where we know that the data will be used for litigation / prosecution we maintain forensically sound images of the affected systems and follow internal chain of custody protocols.  In situations where it is unlikely that the data will be used outside of the investigation we attempt to avoid impacting the underlying systems as much as possible while collecting. Q. “If we are using web-based email like Gmail and not Outlook, where else can we look for where the attacks are coming from or other logs inside the registry?” A. Enterprise Google Apps clients have a wealth of logging for email.  In a situation where a system was suspected of being compromised via a Gmail email we'd correlate timeline data from the user's browser histories, download histories, file system activity, and Gmail logging.  These data sources will typically provide enough data to determine whether the system was compromised via a malicious email opened within Gmail. Q. “What should we investigate on a ransomware infection on a PC?” A. Root cause.  The majority of the ransomware infections we run across are due to the browser rendering a page with an exploit kit that determines what vulnerable plugins and applications the browser is running.  Once you've determined when the malware dropped on the system, timeline the file system activity prior to determine whether there were artifacts from Java, Flash, etc. created just prior to the compromise.  Use that information to ensure that the rest of your systems are appropriately patched (and that your browsers are sandboxed / protected by EMET).  Additionally, correlate the timeline data with the user's browser history to identify the malicious domain if possible, and sinkhole DNS resolution for the domain (at least temporarily) to prevent additional systems from being compromised. Q. “Do you typically do full forensics for single PC issues or triage with tools like Sysinternals?” A. This depends on why we're investigating the system. IF we have indications that an attacker interacted with the system, we'll typically perform a full forensic investigation to ensure we fully understand the scope of the attacker's actions. If we've simply identified that the attacker attempted to log into the system and need to confirm whether the attempt was successful and whether they interacted with the system we'll perform a lighter triage wherein we collect targeted data from a few relevant sources. Q. “Do you have any tips for prioritizing how/where you hunt for malware on a machine that you suspect is infected?” A. If I don't have an initial indicator of compromise aside from “Thar be dragons malware!” I employ a few techniques (Windows system – Linux methodology is similar but the particulars vary considerably): 1) Check running processes, and their associated network connections. – here I'm looking for unusual process names, paths, arguments, ports, and remote IPs.  If I'm still struggling I'll enumerate the process's open handles, loaded DLLs, etc. 2) Review Prefetch  - if enabled, the Windows Prefetch can help quickly identify unusual executables that have run recently.  Review prefetch files for unusual target paths, unusual accessed files, and last execution time (for timelining). 3) Review persistent binaries – Use a tool such as Mandiant's Redline or Sysinternals Autoruns to collect a list of persistent samples (items that startup on boot / login / scheduled time).  Look for unsual items by path, name, digital signature status, etc. 4) Review AppCompat /Shimcache – Review the application compatibility cache to get a rough timeline of executables run on the system.  Look for unusual filenames & Paths, as well as “last modified” time.  Note: there are intricacies and caveats involved in using this data.  I recommend reading Mandiant's whitepaper prior to parsing this data to better understand its use: https://dl.mandiant.com/EE/library/Whitepaper_ShimCacheParser.pdf Q. “What forensic tools would you recommend for a mobile IR kit? For the field” A. 1)    Computer repair toolkit 2)    Flash drives (several large a read-only drive) 3)    Hard drives for imaging 4)    Hard drive imager (we're big fans of the CRU Ditto – it has night vision mode!) 5)    Flashlight 6)    ESD evidence bags 7)    Small switch and patch cables 8)    Analysis rig with analysis VMs, Encase/FTK/Etc. It sounds like a lot, but fits nicely into a standard laptop bag with careful organization.  Q. “Are there any known limitations to discovering malicious activity via behavioral analysis software like CrowdStrike?” A. There are always limitations when automating malicious activity detection. We always recommend a combination of people and technology for effective threat detection. Additionally, we recommend that our customers use a constant feedback mechanism to ensure that lessons learned through threat detection and incident response are fed back into the technology to reduce false-positives and increase analyst efficiencies. Q. “Do you find many organizations have SIEM tools to help corrolate the information in the logs when responding to incidents?” A. Yes, we often find customers using SIEMs. In almost all cases, the SIEM doesn't ingest logs directly from the endpoint, but rather from select systems and devices (such as Windows domain controllers and critical network devices), which makes the SIEM less relevant during investigations of individual systems. Further, we typically recommend that customers invest in expanding their SIEM deployment to include endpoint logs to improve log based threat detection. Q. “For crypto ransom ware what will be prevention step from antivirus concern or other....can we identify any encryption process is working during ransomware” A. In certain situations, a key can be derived through malware analysis and memory analysis, but we typically recommend that customers be prepared for such events by implementing a good system and configuration backup strategy. To see the full presentation on the expert's guide to investigating common breach scenarios: view the on-demand webinar now.

Featured Research

National Exposure Index 2017

The National Exposure Index is an exploration of data derived from Project Sonar, Rapid7's security research project that gains insights into global exposure to common vulnerabilities through internet-wide surveys.

Learn More

Toolkit

Make Your SIEM Project a Success with Rapid7

In this toolkit, get access to Gartner's report “Overcoming Common Causes for SIEM Solution Deployment Failures,” which details why organizations are struggling to unify their data and find answers from it. Also get the Rapid7 companion guide with helpful recommendations on approaching your SIEM needs.

Download Now

Podcast

Security Nation

Security Nation is a podcast dedicated to covering all things infosec – from what's making headlines to practical tips for organizations looking to improve their own security programs. Host Kyle Flaherty has been knee–deep in the security sector for nearly two decades. At Rapid7 he leads a solutions-focused team with the mission of helping security professionals do their jobs.

Listen Now