Logstash Rabbitmq App Configuration

deb (Debian/Ubuntu/Mint)

sudo apt-get install apt-transport-https
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo 'deb https://artifacts.elastic.co/packages/oss-6.x/apt stable main' | sudo tee /etc/apt/sources.list.d/beats.list

sudo apt-get update && sudo apt-get install filebeat-oss

rpm (CentOS/RHEL/Fedora)

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
echo "[elastic-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/oss-6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md" | sudo tee /etc/yum.repos.d/elastic-beats.repo

sudo yum install filebeat-oss

deb/rpm /etc/filebeat/filebeat.yml

Filebeat does not currently have a module to process the rabbitmq application logs.

Therefore we need to add the rabbitmq application log location to the filebeat inputs. Since rabbitmq uses a multi-line log format we will need to configure a seperate log section to handle it.

Add the following to the end of the log input example, before the filebeat.config.modules section.

- type: log
  enabled: true
  paths:
    - /var/log/rabbitmq/*.log
  fields:
    type: rabbitmq
  multiline.pattern: ^\=
  multiline.match: before

We will be shipping to Logstash so that we have the option to run filters before the data is indexed.

Comment out the elasticsearch output block.

## Comment out elasticsearch output
#output.elasticsearch:
#  hosts: ["localhost:9200"]

Uncomment and change the logstash output to match below.

output.logstash:
    hosts: ["your-logstash-host:your-ssl-port"]
    loadbalance: true
    ssl.enabled: true

Let's check the configuration file is syntactically correct.

deb/rpm

filebeat -e -c /etc/filebeat/filebeat.yml

Ok, time to start ingesting data!

deb/rpm

sudo systemctl enable filebeat
sudo systemctl start filebeat