Start TrialStart Free TrialStart Free TrialBook Your DemoSign In

Kafka Metrics

Collect and ship Kafka server metrics to Logstash and Elasticsearch.

Metricbeat is a lightweight shipper that helps you monitor your Kafka servers by collecting metrics running on the Kafka server. Configure Metricbeat using the pre-defined examples below to collect and ship Apache Kafka service metrics and statistics to Logstash or Elasticsearch.

Follow this step by step guide to get 'logs' from your system to Logit.io:

Step 1 - Install Metricbeat

To get started first follow the steps below:

  • Install
  • Root access
  • Verify the required port is open

Older versions can be found here 7, 6, 5

Step 2 - Copy configuration file

The configuration file below is pre-configured to send data to your Logit.io Stack via Logstash.

Copy the configuration file below and overwrite the contents of metricbeat.yml.

###################### Logit.io Metricbeat Configuration #######################
# =========================== Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

# ================================== Outputs ===================================
# ------------------------------ Logstash Output -------------------------------
<div class="sw-warning">
    <div>
        <img src="/images/source-wizard/warning-triangle.svg">
    </div>
    <div>
       <b>No input available! </b> Your stack is missing the required input for this data source <a href="#" onclick="Intercom('showNewMessage')" class="btn btn-info btn-sm">Talk to support to add the input</a>
    </div>
</div> 

Step 3 - Enable module

There are several built in metricbeat modules you can use. To enable the Kafka module, run the following:

deb/rpm

metricbeat modules list
metricbeat modules enable kafka

Windows

PS > .\metricbeat.exe modules enable kafka

Step 4 - Configure module

Each module has its own configuration file where different metricsets can be enabled / disabled. Locate the configuration file for the Kafka module.

deb/rpm /etc/metricbeat/modules.d/kafka.yml

win ~/<EXTRACTED_ARCHIVE>/modules.d/kafka.yml

By default the following metricsets are disabled. To enable or disable a metric simply comment or uncomment the line out.

- module: kafka
 # metricsets:
   # - partition
   # - consumergroup
  period: 10s
  hosts: ["localhost:9092"]
  
  #client_id: metricbeat
  #retries: 3
  #backoff: 250ms

Step 5 - Start Metricbeat

Start or restart to apply the configuration changes.

Step 6 - how to diagnose no data in Stack

If you don't see data appearing in your Stack after following the steps, visit the Help Centre guide for steps to diagnose no data appearing in your Stack or Chat to support now.

Step 7 - Compatibility

The Kafka modules are tested with Kafka 0.10.2.1, 1.1.0 and 2.1.1.

Step 8 - Kafka Module Dashboard

The Kafka module comes with predefined Kibana dashboards. To view your dashboards for any of your Logit.io stacks, launch Kibana and choose Dashboards.

Prebuilt Kibana dashboard

Return to Search
Sign Up

© 2023 Logit.io Ltd, All rights reserved.