So you’re generating lots of log data from your cloud services like Google, Amazon and Microsoft. And you know there is a lot of value in using that data to optimize and troubleshoot those services. But, how exactly do you get access to that useful information and immediately apply it to your business operations?
In our earlier post, “The Challenges of Getting Log Data from Cloud Services,” we covered the different ways three leading cloud service providers – Google, Microsoft and Amazon – generate log data and how they suggest you utilize it.
In a nutshell, that blog highlights the challenges in extracting data for business analysis from your cloud service logs: The information is there, but the vendors don’t provide an easy way for that data to get to the BI user.
The assumption, that it will be accessed only by knowledgeable IT users, means that you add a level of work for IT, to getting any strong business value out of the data. And, if query generation requires that code be written to get a specific answer, you limit the analysis to a series of previously canned reports, which limits its value.
Third-party log management and analytics services like Logentries are almost always a requirement to get the greatest business value out of your cloud services, and even simplify day-to-day IT monitoring and optimization of those services.
Let’s quickly review the challenges before talking about why third-party tools can be so helpful.
To analyze log data strictly from the client side, Google offers the simple Google Apps Log Analyzer tool, which can take the client log files from several Google Apps tools.
When the log files are scanned, they report on the problems or issues contained within and offer guidance on solving the problems discovered. But there is no end user configuration or methodology to find answers to specific questions; it is a black box where the user puts their log data in and gets back a report on what Google thinks are the most important contents.
For analysis of the server-side components with Google apps, there are no user available tools (other than the administrative logs mentioned in part one); you are limited to using the logs you created when you built the applications. Cloud-side log data is therefore limited to what you build on your own and is available only by deploying custom code to generate canned report information.
On the flip side, many services and components of Microsoft Windows Azure already generate log data, and Windows Azure also offers a diagnostic module, which can be installed to collect diagnostic data from the many sources available within the service.
Collecting data seems simple on the surface; you import the diagnostics module into the service model and configure the data sources. There is, however, one major catch: the data does not persist unless you also configure it to be saved to Windows Azure storage, which can make viewing and managing the data more difficult.
With the various Amazon services, there is a significant amount of log data available for analysis. As we mentioned in our previous post, Amazon suggests the use of Elastic MapReduce (EMR) as a methodology for analyzing the data in the logs. But it’s not quite that straightforward. The relatively simple task of analyzing Apache web server data using Apache Hive and EMR is described in detail in this online tutorial. Following these steps allows you to build an engine connected to the web server log data that lets you use the SQL-like language supported by Hive to query the log data. This is very technical approach and not something suitable for business intelligence (BI) users in most organizations.
How Third-Party Log Management and Analytics Tools Help?
So it’s obvious why the IT or DevOps team could find itself in a frustrating situation: lots of extremely valuable information, but no easy way for that data to get to the BI user. That’s where the third-party tools come in. They can simplify day-to-day IT monitoring and optimization of these services and help get the most business value out of your cloud services – quickly.
For example, configuring and using Logentries can be simplified by using the provided agents for Linux and Windows. Agent installation provides a simplified starting point for accessing and using a variety of log data. But the flexibility of the system means that you can connect at a variety of levels from standard syslog and rsyslog to token TCP and custom API forwarders. Each level of integration gives you more detailed control and a potentially more informative look into your log data.
But one of the greatest strengths of this type of service is the ability to present the data that you want to track across the various logs in an “at-a -glance” format. If you are using Logentries, you can easily access a system snapshot through a customized, real-time dashboard. The flexibility of the charting, graphing and reporting widgets enables you to see important events, trend behavior over time, and monitor critical events from a single screen.
Using the Log Insights dashboard you can get a look into the activity across all of your monitored systems and drill down to the most active logs and events being generated. While it may be more complex to create log data sources in the Cisco cloud than in Microsoft Azure, the tools are there to walk you through the configurations and the display of data gives you a single view across multiple platforms should that be your requirement.
The real advantage here is that learning to get the most from the Logentries tool applies across all of their supported platforms. This means that you can minimize the learning curve should you decide to switch from one supported platform to another, as configuration of Logentries is versatile and can easily provide the information you need regardless of the platform. Check it out with a free account, and let us know what you think!