A little context

Ever since Elastic acquired Packetbeat in mid 2015, the beats community has grown at a pace. This momentum really began to accelerate when Elastic released libbeat, in late 2015, allowing the community to build it's own custom beats. As of writing there are well over 30 beats contributed from community alongside the 4 beats provided by Elastic themselves.

An interesting trend became apparent as more of these beats began to appear. Many of them were specifically collecting metrics from different services. For example a member of the comunity contributed Mysqlbeat, a beat designed to send metrics from a mysqlserver to elastic. Likely inspired by these initial 'metric' themed beats many more were created such as Nginxbeat and Apachebeat and so forth.

Unlock the trend

Elastic had their own beat called Topbeat designed to send system metrics such as CPU usage, memory utilisation, and disk space. All these beats were essentially doing the same job and Elastic decided they wanted to lower the barrier to entry just as they had done with libbeat.

In 5.0 Elastic sought to achieve this by deprecating Topbeat and replacing it with Metricbeat. The central tenet of Metricbeat is that it modular and that anyone can build a module to metric anything they like. Elastic then wrote a 'system' module which ships with Metricbeat and replicates all the functionality of Topbeat while adding a lot more too.

Topbeat+

One of the most interesting features that the 'system' module brings to the table is the capability to monitor container resource usage. Metricbeat can monitor Linux control groups, more commonly known as cgroups. Container technology use cgroups to monitor and limit resources assigned to containers. With the explosion in docker like technologies this alone is a compelling reason to upgrade.

Metric all the things

In addition to the the 'system' module Metricbeat also ships with the following modules

This allows you to instantly get a handle on the performance of any of these services.

Roll your own

Whatsmore you can write your own Metricbeat module should you need to. A lot of the work in generating a metricset and the module itself has already been done for you. Check out the Metricbeat developer guide for more information.

Upgrade me already

There is no automatic way to migrate from Topbeat to Metricbeat  but both config files are quite simple. Here is a sample Topbeat input file

input:   # In seconds, defines how often to read server statistics  period: 10 # Regular expression to match the processes that are monitored  # By default, all the processes are monitored  procs: [".*"] # Statistics to collect (all enabled by default)  stats:     # per system statistics, by default is true    system: true # per process statistics, by default is true    process: true # file system information, by default is true    filesystem: true # cpu usage per core, by default is false    cpu_per_core: false

The Metricbeat equivalent

metricbeat.modules: #------------------------------- System Module -------------------------------- module: system metricsets:     # CPU stats    - cpu # System Load stats    - load # Per filesystem stats    - filesystem # Memory stats    - memory # Network stats    - network # Per process stats    - process enabled: true period: 10s processes: ['.*']

The rest of the configuration options, outputs, name, tags etc stay the same or can be upgraded using the migration script that Elastic provide.

Here at Logit we are working with the Elastic stack every day and are actively helping people to migrate. If you'd like to work with us and tap into our expertise do contact us on social media, live chat or talk to our amazing support team.  Of course you can also get started now by signing up our 2 week free trial.