Amazon S3 Logstash Configuration

In the top left corner of your aws console you will notice a services drop down arrow. Open it and from that menu choose IAM.

Now in the left hand menu you want to select policies. Once you have reached the policies page you want to hit the Create Policy that appears towards the top of the page.

On the create policy screen choose the json tab and enter the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Read",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
             ],
            "Resource": [
                "arn:aws:s3:::your-bucket/*"
            ]
        },
        {
            "Sid": "List",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
             ],
            "Resource": [
                "arn:aws:s3:::your-bucket"
            ]
        }
    ]
 }

At the bottom of the page select Next.

On the next page you need to give your policy a name.

Finally scroll to the bottom of the page and now hit create policy.

You are going to need to create a new user. While on the IAM page, in the left hand menu choose users.

Now at the top of the page select Create User.

Enter a username and continue onto the next page of creation.

In the next section you want to attach an exisiting policy. Highlight attach exisiting policies.

Search the policy list below for your newly created policy.

Continue onwards to the User Review step, you can choose to set any Tags here but they aren't necessary. Check all settings are correct and select create user.

This will take you to the Users page where you will see the new user that has just been created. Click on the new user to go to the user details page.

On the user details page click the Create access key link.

Here we need to select the "Third-party service" option shown below.

Scroll down to the very bottom of the page and click the confirmation checkbox and then click the "Next" button.

Enter a description and then click the Create access key option.

On the next screen you will be given your Access Key ID and Secret Access Key. You will need to make a note of these or alternatively download the .csv file provided.

Ensure your logs are being sent to an S3 bucket. The following Getting Started guide will help you achieve this if you are not doing so already:

Getting started creating an Amazon S3 bucket and sending data to Logit.io

To start sending logs and metrics from AWS to your Stack you need to configure an AWS Input on your Logit.io Stack.

Go to Dashboard

Logit.io will verify your input before it is applied, we will contact you to confirm when this has been completed.

Data should now have been sent to your Stack.

View my data

If you don't see logs take a look at How to diagnose no data in Stack below for how to diagnose common issues.

Sending data from Amazon S3 to Logit.io is a crucial aspect of efficient log management and analysis for businesses utilizing AWS infrastructure. By efficiently integrating these two services, organizations can gain valuable insights from their log data and enhance their operational efficiency. Logit.io offers an intuitive solution that allows users to ship, analyze, and visualize logs from Amazon S3, ensuring that vital information is readily available for troubleshooting and monitoring. Via Logit.io, you can also take advantage of our Amazon ELB Application (Elastic Load Balancer) to ingest further data into our centralised platform. In addition, you can link your CloudWatch metrics to Logit.io for comprehensive log data analysis and real-time monitoring. By leveraging these services in tandem, you can enhance your operational intelligence and proactively address issues within your AWS infrastructure. Logit.io's AWS logging is known for its end-to-end integration of these components.