Kafka Metrics

To get started first follow the steps below:

• Install
• Root access
• Verify the required port is open

Older versions can be found here 7, 6, 5

The configuration file below is pre-configured to send data to your Logit.io Stack via Logstash.

Copy the configuration file below and overwrite the contents of metricbeat.yml.

###################### Logit.io Metricbeat Configuration #######################
# =========================== Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

# ================================== Outputs ===================================
# ------------------------------ Logstash Output -------------------------------
<div class="sw-warning">
    <div>
        <img src="/images/source-wizard/warning-triangle.svg">
    </div>
    <div>
       <b>No input available! </b> Your stack is missing the required input for this data source <a href="#" onclick="Intercom('showNewMessage')" class="btn btn-info btn-sm">Talk to support to add the input</a>
    </div>
</div> 

There are several built in metricbeat modules you can use. To enable the Kafka module, run the following:

deb/rpm

metricbeat modules list
metricbeat modules enable kafka

Windows

PS > .\metricbeat.exe modules enable kafka

Each module has its own configuration file where different metricsets can be enabled / disabled. Locate the configuration file for the Kafka module.

deb/rpm /etc/metricbeat/modules.d/kafka.yml

win ~/<EXTRACTED_ARCHIVE>/modules.d/kafka.yml

By default the following metricsets are disabled. To enable or disable a metric simply comment or uncomment the line out.

- module: kafka
 # metricsets:
   # - partition
   # - consumergroup
  period: 10s
  hosts: ["localhost:9092"]
  
  #client_id: metricbeat
  #retries: 3
  #backoff: 250ms

Start or restart to apply the configuration changes.

Now you should view your data:

View my data

If you don't see logs take a look at How to diagnose no data in Stack below for how to diagnose common issues.

If you don't see data appearing in your Stack after following the steps, visit the Help Centre guide for steps to diagnose no data appearing in your Stack or Chat to support now.

The Kafka modules are tested with Kafka 0.10.2.1, 1.1.0 and 2.1.1.

The Kafka module comes with predefined Kibana dashboards. To view your dashboards for any of your Logit.io stacks, launch Logs and choose Dashboards.