About

We recommend using Filebeat to upload logs locally from Debian to Logstash.

Install

Filebeat isn't in the main debian repos, so first we need to add the elastic beats repo

apt-get install apt-transport-https
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo 'deb https://artifacts.elastic.co/packages/oss-6.x/apt stable main'  > /etc/apt/sources.list.d/beats.list

Use APT to download and update filebeat:

sudo apt-get update && sudo apt-get install filebeat-oss

Configuration

Locate the filebeat YAML config file:

/etc/filebeat/filebeat.yml

Open the file and change the following:

  1. Line 24: Set the log input to enabled:
# Change to true to enable this input configuration.
enabled: true
  1. Line 28: Specify the directory to scan for new logs. For example:
# Paths that should be crawled and fetched. Glob based paths.
paths:
    - /var/log/*

Note: * means any text, so this will pick up any file inside the filebeat_logs folder.

  1. Line 143: The elasticsearch output will be enabled/configured by default. Disable this by commenting it out:
#output.elasticsearch:
    # Array of hosts to connect to.
    #hosts: ["localhost:9200"]
  1. Line 153: Enable the logstash output and the load balancer:
output.logstash:
    # The Logstash hosts
    hosts: ["your-logstash-host:your-port"]
    loadbalance: true
    ssl.enabled: true

Save and close the file.

Start Logging

Start filebeat (starting filebeat using this method will display live activity inside the terminal. It will also display any validation errors found in the YAML file):

sudo filebeat -e -c /etc/filebeat/filebeat.yml

Any logs found inside the previously specified directory will be harvested by filebeat (this activity will be displayed in the terminal) and logged to logstash.

You can also start the service without using the filebeat command but this will not display the activity in the terminal:

sudo systemctl enable filebeat
sudo systemctl start filebeat