Logit.io
Kafka broker and controller logs: errors, ISR and failover
← Back to blog
11/16/2026 · 24 min read

Kafka broker and controller logs: errors, ISR and failover

Logit.io Team
Logit.io Team
Technical Content Team

Last updated 11/16/2026

Log ManagementHow To GuidesOpensearch

The Apache Kafka pack in the Logit.io Dashboard Library installs one OpenSearch Dashboards view, Logit.io Kafka broker and controller logs, that shows whether every broker is still logging, whether ERROR and FATAL lines are rising and which Kafka components they come from. It fills when Filebeat's kafka module ships the broker's own log files, such as server.log, controller.log and state-change.log, to your Logs stack, where an ingest pipeline splits each line into severity, component, Java class and exception. It reads logs, not metrics, so it tells you what the brokers said about ISR shrinks, controller failovers and failing disks, not how far behind your consumers are.

This guide is for teams running Apache Kafka, Confluent Platform or another Kafka distribution on virtual machines or bare metal who want to stop SSHing into three brokers to grep for the moment a partition went under-replicated. It covers which Kafka log files matter, how each line is parsed, how to ship them with Filebeat, how to prove the path with a throwaway topic, how to read the dashboard, which messages deserve a saved search even though Kafka logs them at INFO, how to alert on them and what to check when the panels look wrong.

Contents

What the Apache Kafka pack shows

The pack is listed as Apache Kafka in the Dashboard Library, under the messaging category, and installs a single dashboard called Logit.io Kafka broker and controller logs. It reads the filebeat-* index pattern with @timestamp as the time field, opens on the last 24 hours and refreshes every 60 seconds. Every panel starts from the same Lucene base query:

event.module:kafka AND event.dataset:"kafka.log" AND agent.type:filebeat

The two error panels add AND log.level:(ERROR OR FATAL). Everything else counts every Kafka line in the range. The eight panels run top to bottom in the order you'd use them during an incident:

PanelWhat it showsField it reads
Kafka logsKafka log lines in the rangeevent.dataset
Error logsLines at ERROR or FATAL, red above zerolog.level
Reporting hostsDistinct hosts sending Kafka logshost.name
Last event receivedNewest Kafka line in the range@timestamp
Kafka log severity over timeLines per time bucket, stacked by severitylog.level
Logs by severityShare of INFO, WARN, ERROR and the restlog.level
Logs by componentTop five components that wrote lineskafka.log.component
Recent error logsLatest ERROR and FATAL linesTime, level, host, component, class, event type and file path

It answers three questions: are the brokers' logs arriving, are errors rising and which part of Kafka is complaining. It deliberately doesn't show broker availability, consumer lag, throughput, request latency or disk usage, because none of those are in the log files. A quiet broker can be perfectly healthy. Treat the pack as the place you go when something feels wrong, and keep your Kafka metrics for the numbers.

Kafka log files and what they hold

A Kafka broker doesn't write one log. Its log4j configuration (config/log4j.properties up to Kafka 3.9, config/log4j2.yaml from Kafka 4.0) sends different loggers to different files in the directory set by kafka.logs.dir. They all use the same layout, [%d] %p %m (%c)%n, which is what makes them parseable as one dataset:

FileWhat's in itRead by default?
server.logEverything not routed elsewhere: startup and shutdown, partitions, ISR changes, replica fetchers, group coordinator, socket server, disk errorsYes
controller.logThe controller: in KRaft mode the QuorumController and metadata changes, in ZooKeeper mode the KafkaController and electionsYes
state-change.logPartition leadership and replica state transitions, written by state.change.loggerYes
kafka-authorizer.logAuthorisation decisions; denied requests are logged at INFOYes
kafka-request.logRequest logging, WARN and above by default, so usually close to emptyYes
log-cleaner.logCompaction of compacted topics such as __consumer_offsetsNo, add it yourself

The Filebeat module's default paths are controller.log*, server.log*, state-change.log* and kafka-*.log* under /opt/kafka*/logs/. The trailing * picks up the hourly rotated files such as server.log.2026-11-16-09, and the module skips .gz files. log-cleaner.log doesn't match kafka-*.log*, so if you care about compaction stalls on __consumer_offsets, list it explicitly. The garbage collection log, kafkaServer-gc.log, doesn't match either, which is what you want: it's a different format.

On a KRaft cluster with dedicated controller nodes, the controller's controller.log lives on the controller machines, not the brokers. Install Filebeat on those too, or the dashboard only ever shows half the cluster. In combined mode, where one process is both broker and controller, everything is on the same host.

How a Kafka log line becomes fields

When Filebeat's kafka module ships a line, it tags the event with event.module: kafka and event.dataset: kafka.log, joins stack-trace lines onto the line above (any line that doesn't start with [ belongs to the previous event) and records the host's time zone in event.timezone. It also names the ingest pipeline the event needs. Your Logs stack's Logstash recognises that name and hands the event to the Kafka log ingest pipeline on your stack, which does the parsing. You don't configure that pipeline name. Index templates and mappings on Logit.io stacks are managed for you (the index templates guide shows how to read them). The default Filebeat template is applied when an index is created, so if a type looks wrong, fix it in the pipeline or the shipper rather than editing the live mapping; existing daily indices keep their first type until they age out.

Kafka broker and controller log files are read by the Filebeat kafka module, routed by your stack's Logstash to the Kafka log ingest pipeline, stored in filebeat indices and shown on the Apache Kafka dashboard

The pipeline reads the timestamp in brackets, the level, the message and the logger name in parentheses at the end. If the message itself starts with a bracketed name, which most Kafka classes add to say which instance is speaking, that becomes the component. Here's what it produced for real Kafka lines when we ran them through the same pipeline definition:

Line (shortened)log.levelkafka.log.componentkafka.log.class
INFO [Partition orders-3 broker=1] Shrinking ISR from 1,2,3 to 1,3. ...INFOPartition orders-3 broker=1kafka.cluster.Partition
WARN [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Error in response for fetch request ...WARNReplicaFetcher replicaId=2, leaderId=1, fetcherId=0kafka.server.ReplicaFetcherThread
INFO [QuorumController id=3] Becoming the active controller at epoch 12, ...INFOQuorumController id=3org.apache.kafka.controller.QuorumController
ERROR Error while writing to checkpoint file /var/lib/kafka/data/... ERRORunknownorg.apache.kafka.storage.internals.log.LogDirFailureChannel
INFO Principal = User:CN=billing-service is Denied operation = WRITE ...INFOunknownkafka.authorizer.logger
INFO [Broker id=1] Transitioning 12 partition(s) to local leaders.INFOBroker id=1state.change.logger

Four details matter later:

  • The component carries IDs. Partition orders-3 broker=1 and Partition orders-4 broker=1 are different values, so on a busy cluster the Logs by component doughnut chart fragments into partitions and fetchers. That's what the pack's notes mean by component labels including node identifiers.
  • Lines without a bracketed instance get unknown. That's the literal word, set by the pipeline, and it's different from the chart's Unknown slice, which means the field is missing altogether.
  • Stack traces are split out. When the first line after the message has the usual ExceptionClass: message shape, it becomes kafka.log.trace.class and kafka.log.trace.message, for example java.io.IOException and No space left on device. Searching on the exception class is often quicker than searching the message. Traces that don't have that shape stay in message only.
  • Only ERROR and FATAL are errors. The pipeline sets event.type to error for those two levels and info for everything else, WARN included. Kafka logs a shrinking ISR at INFO and a failing replica fetcher at WARN, so neither shows up in Error logs.

Kafka writes local time with no zone, so the pipeline reads the timestamp in the zone Filebeat recorded in event.timezone. That's right as long as Filebeat and Kafka run with the same time zone. The pack's fields are mapped as keywords in the Filebeat index template, so you filter on them exactly as named, with no .keyword suffix, while message is full text.

Ship broker and controller logs with Filebeat

Install Filebeat on every broker and, on a KRaft cluster with dedicated controllers, on every controller node too. Start with the Kafka integration guide in your Logit.io account. Choose Install Integration first, which prepares your stack for this source, then install Filebeat as the guide shows for your platform.

Find the log directory

The module looks under /opt/kafka*/logs/ unless you tell it otherwise, which suits the Apache tarball unpacked in /opt and very little else. Find where each broker really writes:

# The directory the running broker was started with
ps -o args= -C java | grep -o 'kafka.logs.dir=[^ ]*'

# Or look for the files themselves
sudo find / -name 'server.log' -path '*kafka*' 2>/dev/null

The tarball usually writes to /opt/kafka/logs. Confluent Platform packages and many systemd units use /var/log/kafka. Kafka in Docker with file logging turned on writes wherever you mounted the host directory. Don't confuse the application log directory with log.dirs in server.properties: that's where Kafka stores your topic data, and Filebeat must never read it.

Configure the kafka module

Enable the module, then edit modules.d/kafka.yml. The log fileset is disabled by default, so you must enable it:

sudo filebeat modules enable kafka
- module: kafka
  log:
    enabled: true
    var.paths:
      - /var/log/kafka/server.log*
      - /var/log/kafka/controller.log*
      - /var/log/kafka/state-change.log*
      - /var/log/kafka/kafka-authorizer.log*
      - /var/log/kafka/log-cleaner.log*

Replace /var/log/kafka with your directory. Listing the files rather than using var.kafka_home makes it obvious to the next person what's collected, and adds log-cleaner.log, which the defaults miss. Missing files are skipped, so the same file works on brokers and on controllers. Then set the output in /etc/filebeat/filebeat.yml. Copy the block from the integration guide while signed in, because it fills in your stack's Logstash host and Beats SSL port. Filebeat allows only one output, so comment out the default output.elasticsearch section if it's still there:

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.logstash:
  hosts: ["your-logstash-host:your-beats-ssl-port"]
  loadbalance: true
  ssl.enabled: true

processors:
  - add_host_metadata: ~

Keep add_host_metadata. The Reporting hosts tile and the host column in Recent error logs both rely on host.name, and it's how you tell broker 1 from broker 3 in every search. Then test and start:

sudo filebeat test config
sudo filebeat test output
sudo systemctl enable --now filebeat
sudo journalctl -u filebeat -f

test output should finish with talk to server... OK. If it doesn't, fix the connection first. The Filebeat package runs as root, so it can read Kafka's log directory. If you run it as another user, give that user read access to the directory and its files. Beats data sent through your stack's Logstash lands in daily filebeat-* indices, which is the pattern the pack reads. If you have renamed the index in your Logs stack, the pack won't find your data.

Use the module, not a plain filestream input on the same files. The module sets event.module:kafka, joins stack traces and names the ingest pipeline. A plain input sends raw lines that are never split into level, component and class, so the pack never sees them.

Kafka in containers

The official apache/kafka image, most Helm charts and operators such as Strimzi log to standard output only, so there are no files for the module to read. You have two choices. Mount a volume for the log directory and switch on file logging in the broker's log4j configuration, then point var.paths at the volume from Filebeat on the host. Or keep standard output and collect it as container logs: on Kubernetes the Kubernetes container logs and cluster events packs show broker pods with everything else, but those lines don't carry event.module:kafka and won't fill this pack.

Prove the path with a throwaway topic

Don't wait for a real incident to find out the shipping is broken. Creating a topic makes every broker that hosts a replica write a few predictable lines. On any machine with the Kafka command-line tools:

bin/kafka-topics.sh --bootstrap-server broker1:9092 \
  --create --topic logit-pipeline-check --partitions 3 --replication-factor 3

# a minute later, once you've found it in Discover
bin/kafka-topics.sh --bootstrap-server broker1:9092 \
  --delete --topic logit-pipeline-check

A replication factor of 3 needs at least three brokers; use 1 on a single-broker test cluster. On a cluster with more than three brokers, only the brokers that get a replica will log anything, which --describe on the topic shows. Add --command-config with your client properties if the listener needs TLS or SASL. Then open Discover in OpenSearch Dashboards, pick filebeat-*, set the last 15 minutes and search:

event.dataset:"kafka.log" AND message:"logit-pipeline-check"

Each broker should log lines such as Created log for partition logit-pipeline-check-0 in /var/lib/kafka/data/logit-pipeline-check-0 from kafka.log.LogManager, and No checkpointed highwatermark is found for partition logit-pipeline-check-0 from the Partition logit-pipeline-check-0 broker=1 component. Check four things on the documents:

  1. host.name shows every broker that holds a replica of the test topic. A missing one means Filebeat isn't running there or is reading the wrong directory.
  2. log.level and kafka.log.class are filled. If they're empty and error.log says the grok expressions don't match, the broker isn't using Kafka's standard layout.
  3. @timestamp matches the time you ran the command, not an hour off.
  4. Delete the topic and look for the deletion lines too. If only some brokers report them, compare their Filebeat configs.

For a quick count by host and file across the whole cluster, the same check works as an aggregation in Dev Tools:

GET filebeat-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        { "term": { "event.dataset": "kafka.log" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ]
    }
  },
  "aggs": {
    "hosts": {
      "terms": { "field": "host.name", "size": 20 },
      "aggs": { "files": { "terms": { "field": "log.file.path", "size": 10 } } }
    }
  }
}

Every broker should list server.log at least, and controller nodes should list controller.log. A broker with no state-change.log entries in the last hour is normal if nothing changed leadership.

Install the pack and read it

Open your Logs stack's card in your Logit.io account and choose Dashboard Library. The Library looks back over the last 24 hours of your stack's data and ticks packs whose data it finds, so once the test topic has gone through, Apache Kafka should show Data detected. Choose Rescan stack if you started shipping after you opened the page. Tick the pack, choose Install dashboards and open it in OpenSearch Dashboards when the confirmation appears. The Dashboard Library guide covers the screen in detail.

Read it from the top:

  • Reporting hosts first. It counts hosts that sent at least one Kafka line in the range, so compare it with the brokers and controllers you run. Three brokers and three dedicated controllers that have all logged something should say six. If it's lower, check which host.name is missing before you trust any other number: a host that stopped shipping looks the same as one that had nothing to say.
  • Last event received. Kafka is chatty enough at INFO that a busy broker rarely goes quiet for long, but a small idle cluster can. The pack treats more than two hours as a collection warning rather than an outage.
  • Error logs and the severity trend. A healthy cluster usually shows zero or a handful of ERROR lines a day. A block of red across one time bucket is your incident window. Click it to zoom the whole dashboard to that period.
  • Logs by component. At normal times this is dominated by whatever logs most, often log segment rolls and group coordinator lines. During an incident the top slices tell you where to look: replica fetchers mean replication trouble, QuorumController means the controller, partitions mean ISR churn.
  • Recent error logs. Expand a row to see the full message and stack trace. The log.file.path column tells you whether it came from server.log or controller.log.

Save a copy before you change anything. Two edits pay off quickly on a real cluster: change Logs by component to split on kafka.log.class, which doesn't carry partition numbers and so stays readable, and add a WARN filter to a copy of Recent error logs, because that's where replication trouble shows first.

The pack counts ERROR and FATAL lines, but several of the messages that matter most on a Kafka cluster are logged at INFO or WARN. Save these in Discover against filebeat-*, each with event.dataset:"kafka.log" AND in front. Exact wording varies a little between Kafka versions, so check the phrases against your own logs before you rely on them:

What happenedLevelLucene query
A partition lost an in-sync replicaINFOkafka.log.class:"kafka.cluster.Partition" AND message:"Shrinking ISR"
The ISR changed, in either directionINFOkafka.log.class:"kafka.cluster.Partition" AND message:"ISR updated to"
A partition fell below min.insync.replicasINFOmessage:"ISR updated to" AND message:"under-min-isr"
A follower can't fetch from its leaderWARNkafka.log.class:"kafka.server.ReplicaFetcherThread" AND log.level:WARN
A log directory failed, often a full diskERRORkafka.log.class:*LogDirFailureChannel
A controller became active (KRaft)INFOmessage:"Becoming the active controller"
The active controller stepped down (KRaft)WARNmessage:"Renouncing the leadership"
A new controller (ZooKeeper mode)INFOmessage:"successfully elected as the controller"
A consumer group is rebalancingINFOmessage:"Preparing to rebalance group"
A client failed TLS or SASL authenticationINFOmessage:"Failed authentication"
An ACL denied a requestINFOkafka.log.class:"kafka.authorizer.logger" AND message:"Denied"
A broker failed to startERRORmessage:"Fatal error during broker startup"

Shrinking ISR is the log-side view of under-replicated partitions. A shrink followed a minute later by an ISR updated to line listing the full replica set again is a follower that paused, perhaps for garbage collection. A run of shrinks on one broker, with the same broker ID dropping out of many partitions, is that broker in trouble, and if the ISR falls below min.insync.replicas, producers using acks=all start getting NotEnoughReplicasException. Current Kafka versions mark that moment by adding (under-min-isr) to the ISR updated to line, which is why it has its own search. The broker doesn't log the producers' rejected writes at INFO, so these lines are your warning. Split a visualisation of the shrink search by host.name to see which leader is reporting it, and read the message to see which replica dropped out.

Rebalances are normal during deployments and scaling. A group that rebalances every few minutes when nothing is deploying usually has consumers exceeding max.poll.interval.ms or crashing. The message names the group and the reason, so a terms split on the message isn't useful, but filtering on the group name is. Clusters on Kafka 4.0 and later use the new group coordinator by default, which words its log lines differently, so check your own logs for the equivalent phrase.

The kafka.log.class filter for log directory failures uses a wildcard because the class moved package in recent Kafka versions: older brokers log it as kafka.server.LogDirFailureChannel, current ones as org.apache.kafka.storage.internals.log.LogDirFailureChannel. The kafka.log.trace.class field is worth a visualisation of its own: a terms split on it across the last week shows which exceptions your brokers actually throw.

Alert on Kafka errors and silence

Logs stacks include alerting with ElastAlert 2 rules, which you enable from Alerting & Notifications in your stack settings. The guide to creating alerting rules walks through it. Four rules cover most clusters. First, a burst of ERROR or FATAL lines on one host, using the frequency rule type:

name: Kafka errors on one broker
type: frequency
index: "filebeat-*"
num_events: 5
timeframe:
  minutes: 10
query_key: host.name
filter:
  - query:
      query_string:
        query: 'event.dataset:"kafka.log" AND log.level:(ERROR OR FATAL)'
realert:
  minutes: 30
alert:
  - "email"
email:
  - "[email protected]"

Second, any failed log directory. One is enough, so use the any rule type. With several log directories, the broker takes the failed one offline and keeps running, a replica short for every partition that lived there. With only one, the broker shuts down:

name: Kafka log directory failure
type: any
index: "filebeat-*"
query_key: host.name
filter:
  - query:
      query_string:
        query: 'event.dataset:"kafka.log" AND kafka.log.class:*LogDirFailureChannel'
realert:
  minutes: 60
alert:
  - "email"
email:
  - "[email protected]"

Third, ISR churn across the cluster. This one is logged at INFO, so the error rule never sees it:

name: Kafka ISR shrinking repeatedly
type: frequency
index: "filebeat-*"
num_events: 20
timeframe:
  minutes: 10
filter:
  - query:
      query_string:
        query: 'event.dataset:"kafka.log" AND kafka.log.class:"kafka.cluster.Partition" AND message:"Shrinking ISR"'
alert:
  - "email"
email:
  - "[email protected]"

Twenty shrinks in ten minutes suits a cluster with a few hundred partitions. A rolling restart produces a burst of them by design, so either silence the rule during planned work or raise the number. Fourth, a flatline rule per host, matching the pack's two-hour freshness warning. It catches a stopped Filebeat service, a moved log directory and a broker that has died outright, with two limits: it only watches hosts it has already seen, and a healthy but idle host can trip it, so lengthen the window for small or quiet clusters:

name: Kafka host has stopped logging
type: flatline
index: "filebeat-*"
threshold: 1
timeframe:
  hours: 2
query_key: host.name
filter:
  - query:
      query_string:
        query: 'event.dataset:"kafka.log"'
alert:
  - "email"
email:
  - "[email protected]"

Run each rule with Test only before enabling it, to see how often it would have fired over recent data. Treat these as a complement to metric alerts on under-replicated partitions, broker availability and consumer lag, not a replacement: the logs tell you why, the metrics tell you how much.

When the dashboard looks wrong

  • The pack isn't detected. The Library looks for event.dataset:kafka.log in the last 24 hours. If Discover finds Kafka lines but without that field, they came from a plain input, another shipper or container logs, not the module.
  • Reporting hosts is lower than your broker count. Run the Dev Tools aggregation above. A host missing entirely is a Filebeat problem: check journalctl -u filebeat for TLS errors or permission denied on the log directory. Dedicated KRaft controllers missing is usually just Filebeat not installed on them.
  • Logs by component is a wall of partitions. That's the component carrying IDs. Split a copy on kafka.log.class instead.
  • A big Unknown slice in Logs by severity. Lines that didn't match the expected layout have no log.level. Search event.dataset:"kafka.log" AND _exists_:error.log to find them. The usual cause is a customised log4j pattern. The pipeline recognises Kafka's default layout and one alternative that puts the thread name in brackets, and nothing else. Put the layout back to [%d] %p %m (%c)%n on the file appenders; the pipeline is managed and won't learn a new layout.
  • Times are an hour or more out. Kafka writes local time with no zone, and the pipeline uses the zone in event.timezone, which comes from the host Filebeat runs on. A broker container in UTC on a host set to Europe/London will be an hour out in summer. Run both in the same zone, ideally UTC. A line whose date can't be parsed, for example because the pattern prints ISO 8601 dates, keeps the time Filebeat read it and carries error.message.
  • Stack traces arrive as separate one-line documents. Something is reading the files without the module's multiline setting, or a second input is reading the same files. Remove any extra filestream or log input pointed at the Kafka directory.
  • Old lines appeared all at once. On first start Filebeat reads existing files from the beginning, including rotated server.log.* files, so a long-lived broker can send days of history. Filebeat then resumes from its saved offsets; deleting its registry, or reinstalling it without keeping the data directory, makes it read the files again.
  • Some events never arrived. If your stack has the Logstash dead letter queue turned on, events that failed to index are kept there with the reason, which is the first place to look for mapping conflicts.

Limits and trade-offs

  • Logs, not metrics. Consumer lag, under-replicated partition counts, request latency, throughput and disk usage come from JMX metrics, not log files. The log pack tells you what happened and when. Pair it with metrics for the numbers.
  • Severity is Kafka's, not yours. Kafka decides what counts as ERROR. Some serious events are INFO and some ERROR lines during a planned restart are harmless, which is why the saved searches and the ISR rule matter as much as the error tile.
  • The parser is fixed. Ingest pipelines on Logs stacks are managed, so a log4j layout other than Kafka's default, or the one alternative the pipeline knows, won't be parsed into fields. Keeping Kafka's default layout is the fix.
  • Volume can surprise you. At INFO, a cluster with thousands of partitions writes a lot during restarts and leadership changes, and DEBUG writes vastly more. Leave Kafka at INFO and check your daily log volume after the first rolling restart.
  • Logs can contain names you'd rather not spread. Authorisation and authentication lines carry client principals and IP addresses, and topic names can be revealing. Check who can view filebeat-* on your stack and keep retention to what you need.
  • Only the Filebeat module fills the pack. Kafka logs sent by Fluent Bit, the OpenTelemetry Collector or as Kubernetes container logs reach your stack and are searchable, but without event.module:kafka and the Kafka pipeline they don't appear here.

Start Free Trial

Unlock complete visibility with hosted ELK, Grafana, and Prometheus-backed Observability

Start Free Trial

FAQ

Does it work with both KRaft and ZooKeeper clusters?

Yes. Both write the same layout to the same files. Kafka 4.0 and later are KRaft-only; ZooKeeper mode applies to 3.9 and earlier. The difference is what's in controller.log and where it lives: on a ZooKeeper cluster one broker at a time is the controller and logs as Controller id=N, while on KRaft the controller quorum logs as QuorumController id=N, on dedicated controller nodes if you run them. ZooKeeper's own logs are a different format and aren't part of this pack.

We run Confluent Platform. Does it fill the pack?

Yes, for the brokers. Confluent Platform's Kafka brokers use the same log layout, usually under /var/log/kafka, so set var.paths to that directory. Schema Registry, Connect and ksqlDB write their own logs, which you can ship with the same Filebeat but which won't carry event.module:kafka.

Can it show consumer lag or under-replicated partitions?

Not as numbers, because neither is written to the logs. You'll see the ISR shrink and expand lines that explain an under-replicated partition, and the rebalance lines that often explain lag, but the counts come from Kafka's metrics.

Does it work with Amazon MSK or Confluent Cloud?

No. Managed services don't give you the broker's log files, so there's nothing for Filebeat to read. Amazon MSK can deliver broker logs to CloudWatch Logs, S3 or Firehose, and you can ship them to your Logs stack from there, but they won't fill this pack.

Can I add panels or change the dashboard?

Yes. The installed dashboard is an ordinary OpenSearch Dashboards saved object. Save a copy under a new name before you edit it. Good first changes are a component split on kafka.log.class, a WARN table and an ISR shrink count split by host.

Can I try it on the free trial?

Yes. The 14-day free trial includes every feature, including the Dashboard Library, alerting and Hosted OpenSearch MCP. Point Filebeat at one broker, create the test topic and install the pack.

Next steps

Find each broker's real log directory, list the files in modules.d/kafka.yml including log-cleaner.log, and install Filebeat on dedicated controllers as well as brokers. Create and delete the test topic and check that every host shows up. Then install the Apache Kafka pack, save the ISR, controller and log directory searches and add the four alerts. If you're running your own OpenSearch cluster for Kafka logs, a Logit.io Logs stack gives you managed Logstash, OpenSearch and OpenSearch Dashboards with the Dashboard Library built in. For a dedicated cluster, see Hosted OpenSearch and OpenSearch pricing. To ask questions of the same broker logs from any MCP-compatible AI assistant, such as which broker started dropping out of ISRs before last night's incident, see Hosted OpenSearch MCP.

Get the latest Elastic Stack & logging resources when you subscribe

Want to see this in action?
Start a free trial and connect logs to your alert workflows.