Amazon S3 Logstash Configuration

Pull logs from a S3 Bucket to logstash

Step 1 - Create S3 Policy

In the top left corner of your aws console you will notice a services drop down arrow. Open it and from that menu choose IAM.

List of IAM policies

Now in the left hand menu you want to select policies. Once you have reached the policies page you want to hit the Create Policy that appears towards the top of the page.

On the create policy screen choose the json tab and enter the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Read",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
             ],
            "Resource": [
                "arn:aws:s3:::your-bucket/*"
            ]
        },
        {
            "Sid": "List",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
             ],
            "Resource": [
                "arn:aws:s3:::your-bucket"
            ]
        }
    ]
 }

Create policy editor in JSON

At the bottom of the page select review policy and finally on the next page you need to give your policy a name, now hit create policy.

Create new policy form fields

Step 2 - Create User

You are going to need to create a new user. While on the IAM page, in the left hand menu choose users.

Now at the top of the page select Add User.

Add User from Users page

Enter a username and make sure to check Programmatic Access, continue onto the next page of creation.

Fill in set user details form

In the next section you want to attach an exisiting policy. Highlight attach exisiting policies and search the policy list below for your newly created policy.

Highlight existing policies box

Continue onwards to the next step, you can choose to set any Tags here but they aren't necessary. Continue onto the User Review, check all settings are correct and select create user.

On the next screen you will be given your Access Key ID and Secret Access Key. You will need to make a note of these or alternatively download the .csv file provided.

Success message with your Access Key ID

Step 3 - Confirm S3 Bucket

Ensure your logs are being sent to an S3 bucket. The following guide from Amazon will help you achieve this if you are not doing so already:

Cloudwatch to s3

Step 4 - Start Sending Logs to a Stack

To start sending logs from an S3 Bucket to your Stack you need to configure an S3 Logstash Input.

Logit.io will verify your input before it is applied, we will contact you to confirm when this has been completed.

Go to Dashboard

Step 5 - Check Logit.io for your logs

Now you should view your logs:

Launch Dashboard

If you don't see logs take a look at How to diagnose no data in Stack below for how to diagnose common issues.

Step 6 - S3 Logging Overview

S3 is a simple cloud storage solution created by Amazon. It is preferred by it’s users as it reduces the chance of data losses considerably in comparison to using a typical on premise solution. This storage solution also benefits AWS users additionally as it’s security levels are high. S3 does not function as a database as it is simply designed to backup and store large amounts of data.

When it comes to logging for S3, by default this functionality is not enabled but it can be configured to create and store access logs. These raw log files are sent to an S3 bucket and require further parsing and processing to be useful to whoever is administrating AWS.

When processed in a log management solution, S3 server access logs can be used to see requests made to an S3 bucket. These can be highly useful when creating audit reports for security. S3 access logs become essential to review especially in events where a data breach has occured due to their ability to track data access patterns.

Thanks to our hosted ELK solution, Logit.io makes parsing, managing & reporting on your logs from a variety of AWS services fast and effective. If you need any more assistance when analysing your S3 server access logs we're here to help, feel free to reach out by contacting us via live chat & support we’ll be happy to assist.

Compact View