AWS Beanstalk Configuration

To efficiently transport logs to your stack, Filebeat stands out as a streamlined choice. A member of Elastic's Beats family, Filebeat excels at collecting diverse metrics and logs across environments. Its lightweight nature, encryption capabilities, and user-friendly configuration make it an optimal solution.

Specifically, if you're keen on applying filters to logs, Filebeat offers a straightforward approach. Installation involves minimal effort, and the tool efficiently trails specified files. This is particularly advantageous in an AWS Beanstalk environment, where various log files are generated. Opting for Filebeat ensures a smooth log shipping process and facilitates easy differentiation between log types in OpenSearch.

Rather than engaging in manual SSH access to your EC2 instance for Filebeat installation, Elastic Beanstalk streamlines the process by enabling automatic deployment of AWS services and additional software through its extension system, known as .ebextension.

To implement this, include a .ebextensions folder in the root directory of your application and add a filebeat.config file in YAML format.

Copy the configuration file below and overwrite the contents of your

files:
  "/etc/filebeat/filebeat.yml":
    mode: "000755"
    owner: root
    group: root
    content: |
      filebeat.inputs:
      - input_type: filestream
        enabled: true
        paths:
          # REQUIRED: Change the path to your logs
          - /var/log/*.log
        fields:
          type: logfile
      output.logstash:
        hosts: ["your-logstash-host:your-ssl-port"]
        loadbalance: true
        ssl.enabled: true

commands:
  1_command:
    command: "curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.11.3-x86_64.rpm"
    cwd: /home/ec2-user
  2_command:
    command: "rpm -ivh --replacepkgs filebeat-8.11.3-x86_64.rpm"
    cwd: /home/ec2-user
  3_command:
    command: "/etc/init.d/filebeat start"

Validate your YAML

It’s a good idea to run the configuration file through a YAML validator to rule out indentation errors, clean up extra characters, and check if your YAML file is valid. Yamllint.com is a great choice.

Data should now have been sent to your Stack.

View my data

If you don't see logs take a look at How to diagnose no data in Stack below for how to diagnose common issues.

If you don't see data appearing in your Stack after following the steps, visit the Help Centre guide for steps to diagnose no data appearing in your Stack or Chat to support now.

AWS Elastic Beanstalk simplifies application deployment and management in the AWS Cloud. As a fully managed platform-as-a-service (PaaS) solution, it frees developers from infrastructure complexities, enabling a focus on code creation.

Deployment is effortless, with Elastic Beanstalk handling resource provisioning, capacity management, and automatic scaling based on demand. Supporting diverse programming languages, it offers flexibility for various development needs.

Managed environments cover web servers, application servers, and databases, making Elastic Beanstalk suitable for hosting a variety of applications. Seamlessly integrated with other AWS services, it interfaces with Amazon RDS, Amazon S3, and CloudWatch for comprehensive functionality.

Automatic updates ensure a secure and up-to-date environment, and customization options allow users to modify configurations and integrate with existing AWS resources. Multi-tier application management is simplified, supporting separate environments for distinct tiers.

Built-in monitoring tools provide insights into application performance, and auto-scaling adjusts instances based on load. In a DevOps-friendly environment, Elastic Beanstalk supports continuous integration and deployment (CI/CD), streamlining the software delivery pipeline.

If you need any further assistance with migrating your log data to ELK we're here to help you get started. Feel free to get in contact with our support team by sending us a message via live chat & we'll be happy to assist.