By Lee Smith

ELK

3 min read

A little context

Ever since Elastic acquired Packetbeat in mid 2015, the beats community has grown at a pace. This momentum really began to accelerate when Elastic released libbeat, in late 2015, allowing the community to build it's own custom beats. As of writing there are well over 30 beats contributed from community alongside the 4 beats provided by Elastic themselves.

An interesting trend became apparent as more of these beats began to appear. Many of them were specifically collecting metrics from different services. For example a member of the community contributed Mysqlbeat, a beat designed to send metrics from a mysql server to elastic. Likely inspired by these initial 'metric' themed beats many more were created such as Nginxbeat and Apachebeat and so forth.

Unlock the trend

Elastic had their own beat called Topbeat designed to send system metrics such as CPU usage, memory utilisation, and disk space. All these beats were essentially doing the same job and Elastic decided they wanted to lower the barrier to entry just as they had done with libbeat.

In 5.0 Elastic sought to achieve this by deprecating Topbeat and replacing it with Metricbeat. The central tenet of Metricbeat is that it modular and that anyone can build a module to metric anything they like. Elastic then wrote a 'system' module which ships with Metricbeat and replicates all the functionality of Topbeat while adding a lot more too.

Topbeat+

One of the most interesting features that the 'system' module brings to the table is the capability to monitor container resource usage. Metricbeat can monitor Linux control groups, more commonly known as cgroups. Container technology use cgroups to monitor and limit resources assigned to containers. With the explosion in docker like technologies this alone is a compelling reason to upgrade.

Metric all the things

In addition to the the 'system' module Metricbeat also ships with the following modules:

This allows you to instantly get a handle on the performance of any of these services.

Roll your own

Whatsmore you can write your own Metricbeat module should you need to. A lot of the work in generating a metricset and the module itself has already been done for you. Check out the Metricbeat developer guide for more information.

Upgrade me already

There is no automatic way to migrate from Topbeat to Metricbeat but both config files are quite simple. Here is a sample Topbeat input file:

input: # In seconds, defines how often to read server statistics  period: 10 # Regular expression to match the processes that are monitored  # By default, all the processes are monitored  procs: [".*"] # Statistics to collect (all enabled by default)  stats:     # per system statistics, by default is true    system: true # per process statistics, by default is true    process: true # file system information, by default is true    filesystem: true # cpu usage per core, by default is false    cpu_per_core: false

The Metricbeat equivalent

metricbeat.modules: #------------------------------- System Module -------------------------------- module: system metricsets:     # CPU stats    - cpu # System Load stats    - load # Per filesystem stats    - filesystem # Memory stats    - memory # Network stats    - network # Per process stats    - process enabled: true period: 10s processes: ['.*']

The rest of the configuration options, outputs, name, tags etc stay the same or can be upgraded using the migration script that Elastic provide.

If you enjoyed this post on Topbeat and Metricbeat then why not check out our blog on why centralised logging is vital, or the rest of our centralised logging blog posts.

backReturn to Blog