Ready to get going? Start your 14 days free trial today

Start free trial

Have an account? Sign in

Send data via Docker to your Logstash instance provided by Logit.io

Docker

Collect and ship Docker container application logs to Logstash and Elasticsearch.

Filebeat is a lightweight shipper that enables you to send your Docker container application logs to Logstash and Elasticsearch. Configure Filebeat using the pre-defined examples below to start sending and analysing your Docker application logs.

Step 1 - Install FilebeatCopy

deb (Debian/Ubuntu/Mint)

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-7.6.2-amd64.deb
sudo dpkg -i filebeat-oss-7.6.2-amd64.deb

rpm (CentOS/RHEL/Fedora)

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-7.6.2-x86_64.rpm
sudo rpm -vi filebeat-oss-7.6.2-x86_64.rpm

Windows

  • Download the Filebeat Windows zip file from the official downloads page.
  • Extract the contents of the zip file into C:\Program Files.
  • Rename the filebeat-<version>-windows directory to Filebeat.
  • Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select Run As Administrator). If you are running Windows XP, you may need to download and install PowerShell.
  • Run the following commands to install Filebeat as a Windows service:
PS > cd 'C:\Program Files\Filebeat'
PS C:\Program Files\Filebeat> .\install-service-filebeat.ps1`
If script execution is disabled on your system, you need to set the execution policy for the current session to allow the script to run. For example: PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-filebeat.ps1.
My OS isn't here! Don't see your system? Check out the official downloads page for more options (including 32-bit versions).

Step 2 - Locate The Configuration FileCopy

deb/rpm

/etc/filebeat/filebeat.yml

Change the owner of the filebeat.yml file to root to allow access to the docker container logs.


sudo chown root:root filebeat.yml
ls -la

Skip this step if you are using windows.

Step 3 - Configure The InputsCopy

deb/rpm

On Linux we want filebeat to read the container logs from /var/lib/docker/containers/*/*.log which is where docker's container logs are stored, this is handled by default.

Add to your filebeat.inputs section the docker type.

filebeat.inputs:
- type: docker
  containers.ids:
    - '*'

Windows

On Windows we want filebeat to read the container logs from C:\ProgramData\docker\containers\

filebeat.inputs:
- type: docker
  containers:
  ids: - '*'
  path: C:\ProgramData\docker\containers\

If you do not want to read all container logs then you can specify the continer ID to logs from specific containers.

Step 4 - Configure OutputCopy

We'll be shipping to Logstash so that we have the option to run filters before the data is indexed.
Comment out the elasticsearch output block.

## Comment out elasticsearch output
#output.elasticsearch:
#  hosts: ["localhost:9200"]

Uncomment and change the logstash output to match below.

output.logstash:
    hosts: ["your-logstash-host:your-ssl-port"]
    loadbalance: true
    ssl.enabled: true

Step 5 - Validate ConfigurationCopy

Let's check the configuration file is syntactically correct.

deb/rpm

sudo filebeat -e -c /etc/filebeat/filebeat.yml

Windows

cd <EXTRACTED_ARCHIVE>
filebeat.exe -e -c filebeat.yml

Step 6 - Start FilebeatCopy

Ok, time to start ingesting data!

deb/rpm

sudo systemctl enable filebeat
sudo systemctl start filebeat

Windows

Start-Service filebeat

Step 7 - Docker Logging OverviewCopy

Docker is a platform as a service (PaaS) tool created for building & deploying applications by using containers. Developers use these isolated containers to package an application with all of its required dependencies for streamlined deployment. Docker was first created in 2013 and offers both a free open source solution and paid offering. The platform has been instrumental in the development of cloud-native applications.

Thanks to Docker’s widespread adoption, the trend towards using microservices and containerization has become a must for developers launching applications in the cloud.

Despite this, effective log analysis using Docker’s container logs can easily spiral and run into many complications when trying to keep up with the scaling required for your growing infrastructure. Due to their isolated & distributed nature, centralising Docker log messages is often overly problematic when log data is required for further analysis.

The Logit platform provides a single source of truth for container monitoring and log management. We enable your teams to have complete observability across containers, enabling your Developers & IT leaders with the ability to investigate and fix issues faster.

Read more about container monitoring

If you need any more help with migrating your Docker log data to Filebeat our engineers are here to help. Feel free to get in contact with our support team by sending us a message via live chat & we'll be happy to assist.

Toggle View

Expand View

Return to Search