Last updated at Fri, 03 Nov 2017 21:28:53 GMT

Thanks to some slick work from our engineering team, we have recently released a lightweight python script that will allow you to pull your Elastic Load Balancer logs from S3 into Logentries.

In this implementation, we use AWS Lambda and leverage the S3 trigger, so the script only runs when needed.

The full documentation is available here: https://logentries.com/doc/s3-ingestion-with-lambda/

Get Setup

First, confirm that you have your Elastic Load Balancer configured to write your logs to S3 every five minutes. If you do not have this configuration in place, follow the guide available here. Be sure to assign the necessary policy to your bucket!

Next, setup the Logentires S3 Lambda integration by following the directions here.

Analyzing the log data in Logentries

When you view the data in Logentries you will notice the log events have been automatically parsed into key value pairs (KVPs). The Logentries S3 Lambda integration will automatically parse the following keys:

- ip

- request_time

- elb_status

- backend_status

- bytes_recieved 		- bytes_sent

- method

- url

- user_agent

It would be helpful to know if there were any jumps in our request time. Let’s use the request_time KVP to calculate an average over time. The visualization will even show us the time frame when any spikes occurred so we can investigate further.

In advanced mode, paste the following query:

calculate(average:request_time)

Great! Now it would be helpful to understand what different status codes we are seeing. We can look at the status codes for both the ELB and the backend. In this example lets look at our ELB status codes:

groupby(elb_status)

Add in a sort(desc) to get your top status codes on the left.

This new integration allows us to build some pretty compelling dashboards. Like this one that tracks both bytes_sent and bytes_recieved.

You can quickly manipulate your LEQL queries to generate useful information for your current investigation or monitoring effort. Our new S3 ingestion can pull all logs saved to S3, not just your ELB logs – Give it a try by creating a free Logentries account today!