Filebeat Configuration

To get started first follow the steps below:

• Install
• Root access
• Verify the required port is open

Older versions can be found here 7, 6, 5

The configuration file below is pre-configured to send data to your Logit.io Stack via Logstash.

Copy the configuration file below and overwrite the contents of filebeat.yml.

# ============================== Filebeat modules ==============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  #reload.period: 10s

# ================================== Outputs ===================================
# ------------------------------ Logstash Output -------------------------------
output.logstash:
    hosts: ["your-logstash-host:your-ssl-port"]
    loadbalance: true
    ssl.enabled: true

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

If you’re running Filebeat 7 add this code block to the end. Otherwise, you can leave it out.

# ... For Filebeat 7 only ...
filebeat.registry.path: /var/lib/filebeat

If you’re running Filebeat 6 add this code block to the end. Otherwise, you can leave it out.

# ... For Filebeat 6 only ...
registry_file: /var/lib/filebeat/registry

Validate your YAML

It’s a good idea to run the configuration file through a YAML validator to rule out indentation errors, clean up extra characters, and check if your YAML file is valid. Yamllint.com is a great choice.

Depending on your data source, Filebeat provides pre-configured modules and dashboards. The following data sources have Filebeat modules available to use straight away -

Apache, Auditd, Elasticsearch, HAproxy, IIS, Kafka, Kibana, Logstash, Mongodb, MySQL, Nginx, PostgreSQL, Redis, System, Traefik

DEB/RPM

sudo  -e -c /etc//.yml

macOS

sudo ./ -e -c .yml --strict.perms=false

You’ll be running as root, so you need to change ownership of the configuration file and any configurations enabled in the modules.d directory, or run with --strict.perms=false as shown above. Read more about how to change ownership.

Windows

cd <EXTRACTED_ARCHIVE>
.\.exe -e -c .yml

Linux

sudo ./ -e -c .yml --strict.perms=false

You’ll be running as root, so you need to change ownership of the configuration file and any configurations enabled in the modules.d directory, or run with --strict.perms=false as shown above. Read more about how to change ownership.

If the yml file is invalid, will print an `error loading config file` error message with details on how to correct the problem. If you have issues starting see "How To Diagnose No Data In Stack" below to troubleshoot.

Start or restart to apply the configuration changes.

Now you should view your data:

View my data

If you don't see logs take a look at How to diagnose no data in Stack below for how to diagnose common issues.

If you don't see data appearing in your Stack after following the steps, visit the Help Centre guide for steps to diagnose no data appearing in your Stack or Chat to support now.

Filebeat is the most popular way to send logs to ELK due to its reliability & minimal memory footprint. It is the leading Beat out of the entire collection of open-source shipping tools, including Auditbeat, Metricbeat & Heartbeat.

Filebeat's origins begin from combining key features from Logstash-Forwarder & Lumberjack & is written in Go. Within the logging pipeline, Filebeat can generate, parse, tail & forward common logs to be indexed within Elasticsearch. The harvester is often compared to Logstash but it is not a suitable replacement & instead should be used in tandem for most use cases.

Earlier versions of Filebeat suffered from a very limited scope & only allowed the user to send events to Logstash & Elasticsearch. More recent versions of the shipper have been updated to be compatible with Redis & Kafka.

A misconfigured Filebeat setup can lead to many complex logging concerns that this filebeat.yml wizard aims to solve. Just a couple of examples of these include excessively large registry files & file handlers that error frequently when encountering deleted or renamed log files. Tracking numerous pipelines using this shipper can become tedious for self hosted Elastic Stacks so you may wish to consider our Hosted ELK service as a solution to this.

If you need any further assistance with migrating your log data to ELK we're here to help you get started. Feel free to get in contact with our support team by sending us a message via live chat & we'll be happy to assist.