Kafka Metrics

To get started first follow the steps below:

• Install metricbeat
• Root access
• Verify the required port is open

Older versions can be found here metricbeat 7, metricbeat 6, metricbeat 5

The configuration file below is pre-configured to send data to your Logit.io Stack via Logstash.

Copy the configuration file below and overwrite the contents of metricbeat.yml.

###################### Logit.io Metricbeat Configuration #######################
# =========================== Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

# ================================== Outputs ===================================
# ------------------------------ Logstash Output -------------------------------
<div class="sw-warning">
    <div>
        <img src="/images/source-wizard/warning-triangle.svg">
    </div>
    <div>
       <b>No input available! </b> Your stack is missing the required input for this data source <a href="#" onclick="Intercom('showNewMessage')" class="btn btn-info btn-sm">Talk to support to add the input</a>
    </div>
</div> 

There are several built in metricbeat modules you can use. To enable the Kafka module, run the following:

deb/rpm

metricbeat modules list
metricbeat modules enable kafka

Windows

PS > .\metricbeat.exe modules enable kafka

Each module has its own configuration file where different metricsets can be enabled / disabled. Locate the configuration file for the Kafka module.

deb/rpm /etc/metricbeat/modules.d/kafka.yml

win ~/<EXTRACTED_ARCHIVE>/modules.d/kafka.yml

By default the following metricsets are disabled. To enable or disable a metric simply comment or uncomment the line out.

- module: kafka
 # metricsets:
   # - partition
   # - consumergroup
  period: 10s
  hosts: ["localhost:9092"]
  
  #client_id: metricbeat
  #retries: 3
  #backoff: 250ms

Start or restart metricbeat to apply the configuration changes.

Data should now have been sent to your Stack.

View my data

If you don't see logs take a look at How to diagnose no data in Stack below for how to diagnose common issues.

If you don't see data appearing in your Stack after following the steps, visit the Help Centre guide for steps to diagnose no data appearing in your Stack or Chat to support now.

The Kafka modules are tested with Kafka 0.10.2.1, 1.1.0 and 2.1.1.

The Kafka module comes with predefined Kibana dashboards. To view your dashboards for any of your Logit.io stacks, launch Logs and choose Dashboards.

The Metricbeat Module for Kafka is a specialized tool designed for the efficient monitoring of Kafka clusters by capturing a wide array of performance metrics. For those already familiar with Metricbeat's capabilities in aggregating data from different services, its extension to Kafka brings a familiar yet powerful monitoring solution to the table.

Kafka, a distributed streaming platform, plays a crucial role in data processing and real-time streaming analytics. The Metricbeat Module for Kafka leverages Kafka's JMX metrics to gather data on topics, partitions, brokers, and more, offering deep insights into throughput, latency, and system health.

This module is particularly valued by DevOps, data engineers, and system administrators tasked with overseeing Kafka clusters. Its straightforward setup and integration with the ELK Stack enable real-time monitoring and analysis, significantly enhancing operational visibility and the ability to react swiftly to changes in system performance.

Compared to other monitoring methods that may rely on bespoke or rigid formats, the Metricbeat Module for Kafka is crafted to offer ease of analysis, streamlined troubleshooting, and comprehensive operational oversight for Kafka environments.

Our platform supports the Metricbeat Module for Kafka, offering centralized management of metrics that facilitates straightforward analysis and proactive monitoring. This support removes the complexity of manually tracking and interpreting metrics across distributed Kafka clusters.

Should you need help with configuring the Metricbeat Module for Kafka or have questions about monitoring your Kafka clusters effectively, the Logit.io team is ready to assist. We encourage you to visit our Help Centre or reach out to our support team via live chat & we'll be happy to assist.