Kubernetes dashboards for container logs and cluster events

The Dashboard Library on Logit.io Logs stacks has two Kubernetes packs: Kubernetes container logs (six dashboards for pod stdout and stderr shipped by Filebeat) and Kubernetes cluster events (six dashboards for the Event objects the API server records, such as BackOff, Unhealthy and FailedScheduling). The container logs pack fills as soon as your Filebeat DaemonSet adds Kubernetes metadata, but the cluster events pack needs an event exporter, because events never appear in container log files.

This guide is for teams who already ship Kubernetes logs to Logit.io and want the dashboards to work the first time. It covers which fields each pack reads, the Filebeat and exporter settings that produce them, the OpenSearch API checks that prove the data is there before you install and how to use the two packs together when a pod starts crash looping.

Two Kubernetes packs, twelve dashboards

Kubernetes gives you two different streams of evidence, and the Library ships one pack for each. Container logs are what your applications write to stdout and stderr. Cluster events are what Kubernetes itself records about those applications: the scheduler failing to place a pod, the kubelet backing off a crashing container, a liveness probe failing, a node running out of memory. An application that never starts writes no logs at all, so on its own a container log view can't tell you why a pod is stuck. Events can.

Every dashboard in both packs uses the same layout: four headline tiles, a full-width trend, two breakdowns and a detail table you can pivot from. The default time range is the last 24 hours with a 60-second refresh. Tiles that count something bad (stderr lines, HTTP 5xx, slow requests, Warning events, back-offs, probe failures, scheduling and volume failures) turn red above zero. Every board also has a Last event received tile, which is the first thing to read: a quiet dashboard with a stale last event means shipping has stopped, not that the cluster is healthy.

If you send Kubernetes data through Hosted OpenTelemetry instead of Filebeat, there is a third pack, Kubernetes Signals, which reads the otel.resource.k8s.* fields. The container logs pack deliberately ignores any line that carries otel.resource.k8s.pod.name, so the packs' filters shouldn't double-count the same document when both shippers run during a migration.

Container logs: the fields the pack reads

A line counts in the container logs pack when it has kubernetes.pod.name and no otel.resource.k8s.pod.name. The full base query is:

_exists_:kubernetes.pod.name AND NOT _exists_:otel.resource.k8s.pod.name

The required fields are @timestamp, kubernetes.namespace, kubernetes.pod.name, kubernetes.container.name, kubernetes.node.name, stream and message. log.level is optional and feeds a log-level breakdown on the investigation board, where apps that log no level show as Unknown.

The good news is that the Filebeat manifest in our Kubernetes integration guide already produces every required field. The container input sets stream to stdout or stderr, and the add_kubernetes_metadata processor adds the namespace, pod, container and node names by matching the log file path to the pod. The relevant part of the ConfigMap looks like this:

filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/*.log
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"

output.logstash:
  hosts: ['${LOGSTASH_HOST}:${BEATS_PORT}']
  loadbalance: true
  ssl.enabled: true

Beats data sent through your stack's Logstash lands in daily filebeat-* indices, which is the pattern the pack reads. If you have renamed the index in your Logstash pipeline, for example to aks-* per cluster, the Library will notice the data but the dashboards won't show it, because they only read filebeat-*. On AKS, index names containing aks are treated as a hint, not a match.

Request JSON for the two app boards

Two of the six dashboards, app requests and app errors and slow requests, go a step further. They read request logs that your application writes as one JSON object per line, in the style of Rails with lograge, decoded under a logjson prefix:

• logjson.controller and logjson.action name the handler. Library detection for the app boards also accepts logjson.route in place of logjson.controller.
• logjson.status is the HTTP status as a number. 5xx means logjson.status:[500 TO 599].
• logjson.duration is the request time in milliseconds. A slow request is one second or more.
• logjson.method and logjson.path are optional extras for the detail tables.

Decode the JSON in Filebeat with decode_json_fields, and scope it to the containers that actually write request JSON:

  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
    - decode_json_fields:
        when:
          equals:
            kubernetes.container.name: "web"
        fields: ["message"]
        target: "logjson"
        max_depth: 1
        overwrite_keys: false
        add_error_key: true

Put decode_json_fields after add_kubernetes_metadata so the container name exists when the condition is evaluated, and replace web with your own container name. Scoping matters more than it looks. If every container's JSON is decoded into logjson, one service that logs "status": "ok" and another that logs "status": 500 will fight over the field type in the same daily index, and OpenSearch will reject whichever arrives second. Our post on field type conflicts explains how that failure looks. The app boards stay empty, without errors, for pods that don't log request JSON, so leaving them unpopulated is fine.

Cluster events: ship Event objects, not log lines

Kubernetes events are API objects, not log files. kubectl get events reads them from the API server, and by default the API server keeps them for only an hour. Nothing writes them to /var/log/containers, so a Filebeat DaemonSet never sees them. To keep them, run an event exporter that watches the Events API and writes each event out as JSON. The pack was built against the document shape of kubernetes-event-exporter, and eventrouter writes the same core/v1 fields.

The pack needs @timestamp, involvedObject.kind, involvedObject.name, reason, type and message. involvedObject.namespace, source.component and count are optional. A document counts when it has both involvedObject.kind and reason.

Route A: stdout, your existing Filebeat and a Logstash filter

If you already run the Filebeat DaemonSet above, the simplest route needs no new credentials in the cluster. Deploy the exporter with its stdout receiver, which is also its default:

apiVersion: v1
kind: ConfigMap
metadata:
  name: event-exporter-cfg
  namespace: monitoring
data:
  config.yaml: |
    logLevel: warn
    logFormat: json
    maxEventAgeSeconds: 60
    route:
      routes:
        - match:
            - receiver: "dump"
    receivers:
      - name: "dump"
        stdout:
          deDot: true

Apply it together with the ServiceAccount, ClusterRole and Deployment from the exporter's deploy/ folder, whose container is named event-exporter. Filebeat now collects one JSON line per event from that container. Then, in your stack's Logstash filters, lift those lines to the top level and route them into their own index family:

filter {
  if [kubernetes][container][name] == "event-exporter" {
    json {
      source => "message"
      skip_on_invalid_json => true
    }
    if [involvedObject][kind] {
      mutate {
        replace => { "[@metadata][beat]" => "kubernetes-events" }
      }
    }
  }
}

Parsing into the root replaces message with the event's own message and adds reason, type, count, source.component and involvedObject.*. The second condition means only real events are moved into kubernetes-events-*; the exporter's own warnings stay in filebeat-*. @timestamp is the time Filebeat read the line, which is within seconds of the event because the exporter only emits events younger than maxEventAgeSeconds. deDot replaces dots in label and annotation keys with underscores, so a label like app.kubernetes.io/name doesn't turn into a nested object.

Route B: write straight to the OpenSearch API

The exporter also has an opensearch receiver that can index into your stack's OpenSearch API endpoint directly. It authenticates with a username and password, so the stack must be in Basic (Username and Password) authentication mode, where the username is your stack ID and the password is the stack API key. Switching modes applies to the whole stack endpoint, so check nothing else relies on the apikey query parameter first. The exporter's default document has no @timestamp, so set it in a layout:

receivers:
  - name: "logit"
    opensearch:
      hosts:
        - https://YOUR_STACK_ID-es.logit.io:443
      username: YOUR_STACK_ID
      password: ${LOGIT_STACK_API_KEY}
      indexFormat: "kubernetes-events-{2006.01.02}"
      layout:
        "@timestamp": '{{ if .LastTimestamp.IsZero }}{{ .GetTimestampISO8601 }}{{ else }}{{ .LastTimestamp.UTC.Format "2006-01-02T15:04:05Z" }}{{ end }}'
        type: "{{ .Type }}"
        reason: "{{ .Reason }}"
        message: "{{ .Message }}"
        count: "{{ .Count }}"
        involvedObject:
          kind: "{{ .InvolvedObject.Kind }}"
          name: "{{ .InvolvedObject.Name }}"
          namespace: "{{ .InvolvedObject.Namespace }}"
        source:
          component: "{{ .Source.Component }}"

Load the API key from a Secret into the LOGIT_STACK_API_KEY environment variable; the exporter expands ${...} references in its config. The @timestamp template uses the last time Kubernetes saw the event, and falls back to the first-seen or event time for events that have no last timestamp, so a crash loop that has been running for days still lands inside the 24-hour window. Two side effects of layouts are worth knowing: every value is rendered as a string, so count is indexed as text rather than a number, and cluster-scoped objects such as nodes get an empty namespace rather than none. Neither should break the boards — both routes should still populate them. Route A keeps count numeric, which is one reason we suggest it first.

Whichever route you choose, keep the index name starting kubernetes-events-. The pack reads the stack's catch-all *-* pattern, so any name with a hyphen would be read, but the Library's data check looks at kubernetes-events-* and *k8s-events* first when deciding whether to tick the pack for you.

Check your data before you install

The Library looks back 24 hours when it decides whether to mark a pack Data detected, using index mappings and per-dashboard detection profiles. A _count against each pack's base query is a quick sanity check with the OpenSearch API, not the same test the Library runs. These examples use the default API key mode; in Basic mode, drop the apikey parameter and add -u "YOUR_STACK_ID:$LOGIT_STACK_API_KEY".

ES="https://YOUR_STACK_ID-es.logit.io:443"

# Container logs pack
curl -sG "$ES/filebeat-*/_count" \
  --data-urlencode "apikey=$LOGIT_STACK_API_KEY" \
  --data-urlencode 'q=_exists_:kubernetes.pod.name AND NOT _exists_:otel.resource.k8s.pod.name AND @timestamp:[now-24h TO now]'

# App request boards
curl -sG "$ES/filebeat-*/_count" \
  --data-urlencode "apikey=$LOGIT_STACK_API_KEY" \
  --data-urlencode 'q=_exists_:kubernetes.pod.name AND _exists_:logjson.controller AND @timestamp:[now-24h TO now]'

# Cluster events pack
curl -sG "$ES/kubernetes-events-*/_count" \
  --data-urlencode "apikey=$LOGIT_STACK_API_KEY" \
  --data-urlencode 'q=_exists_:involvedObject.kind AND _exists_:reason AND @timestamp:[now-24h TO now]'

A count of zero means the fields aren't there, not that the cluster is quiet. A healthy cluster still writes a steady stream of Normal events (Scheduled, Pulled, Created, Started), so zero events in 24 hours almost always means the exporter isn't running or isn't reaching the stack.

Next, check that the fields the breakdowns group by are aggregatable. The events pack groups by the .keyword sub-fields that OpenSearch's default dynamic mapping adds to strings:

curl -sG "$ES/kubernetes-events-*/_search" \
  --data-urlencode "apikey=$LOGIT_STACK_API_KEY" \
  -H 'Content-Type: application/json' -d '{
  "size": 0,
  "query": { "range": { "@timestamp": { "gte": "now-24h" } } },
  "aggs": {
    "types":   { "terms": { "field": "type.keyword" } },
    "reasons": { "terms": { "field": "reason.keyword", "size": 15 } }
  }
}'

You want Normal and, on any real cluster, some Warning buckets, plus reasons such as Scheduled, Pulled, BackOff and Unhealthy. If the aggregation returns no buckets while _count was above zero, look at the mapping with GET kubernetes-events-*/_mapping/field/reason*. When reason is mapped as keyword with no .keyword sub-field, the donuts and distinct counts stay empty. For the container logs pack, the same kind of terms check on kubernetes.namespace and on stream should show your namespaces and both stdout and stderr.

If you use Hosted OpenSearch MCP, you can ask your AI assistant the same questions in plain English, for example "count documents in kubernetes-events-* over the last 24 hours grouped by reason". It works with any MCP-compatible client and is read-only, so it's a safe way to check fields without handing out the stack API key.

Install the packs from the Dashboard Library

1. In your Logit.io account, find your Logs stack and choose Dashboard Library from the links at the bottom of its card. Check the stack name at the top of the page.
2. Choose Detected on your stack. If you only just started shipping, choose Rescan stack first. The Kubernetes packs should show the Data detected badge and be ticked.
3. Use Select all beside each pack, or tick individual dashboards. If no pods log request JSON, you can leave the two app dashboards out.
4. Select Install dashboards. Allow a minute or two, then follow Open in OpenSearch Dashboards from the confirmation message.

You need permission to edit dashboards on the stack to install. Installed dashboards sit alongside the ones you built yourself and show an Installed badge in the Library. The Dashboard Library documentation covers the screens in more detail, and our post on Library data detection explains why a pack is or isn't ticked.

To verify the install, open Logit.io Kubernetes container logs and check that Last event received is a few seconds or minutes old and that the namespaces donut lists your namespaces. Then open Logit.io Kubernetes cluster events and check that its last event is recent and the reasons breakdown includes Normal reasons such as Scheduled and Started.

Worked example: a crash-looping pod

The packs are most useful together. Here is a typical sequence when a deploy goes wrong.

1. Pod restarts and probe failures (events) shows the crash-loop back-offs tile turn red. The pods breakdown points at checkout-7d9f… in the shop namespace, and the reasons breakdown shows BackOff and Unhealthy.
2. Event investigation, filtered with involvedObject.name:checkout*, puts the story in order: Pulled, Started, then Unhealthy with a message like Liveness probe failed: HTTP probe failed with statuscode: 500, then Killing and BackOff.
3. stderr output (container logs) for the same time window shows whether the container wrote anything before it was killed. Filter on kubernetes.pod.name:checkout* in the detail table and read the last lines before each restart.
4. App errors and slow requests, if the service logs request JSON, shows whether 5xx responses rose before the probe started failing, which separates "the app is broken" from "the probe is too strict".

The same pattern works for pods that never start. If the container logs boards show nothing for a pod at all, the scheduling and volume failures board usually has the answer in a FailedScheduling message (insufficient CPU or memory, an unsatisfied node selector) or a FailedMount for a missing Secret or volume.

When the dashboards stay empty

• Container logs installed but empty. The lines are reaching an index other than filebeat-*, or add_kubernetes_metadata isn't matching. Check one document in Discover: if kubernetes.pod.name is missing, the Filebeat ServiceAccount usually lacks get, watch and list on pods, namespaces and nodes, or NODE_NAME isn't set.
• Only some nodes appear. The DaemonSet isn't scheduled on every node, typically because of taints. Add tolerations for your tainted node pools; the nodes breakdown on the workloads board makes the gap obvious.
• App boards empty. No decoded logjson.controller. Either the container condition doesn't match, the app logs plain text, or the JSON keys are different. Look for error.message on the raw line, which add_error_key sets when decoding fails.
• Status or duration looks wrong. logjson.status was first indexed as text in today's index, so the [500 TO 599] range doesn't match. It usually corrects itself on the next daily index once only numeric values arrive.
• Events pack empty but the exporter is running. On Route A, the Logstash condition doesn't match the container name, or the filter hasn't been saved and applied. On Route B, look in the exporter's own logs for Indexing failed, which usually means a 401 from the wrong authentication mode.
• Events donuts empty while counts work. The strings were mapped as keyword with no .keyword sub-field. Check the mapping as shown above.
• Exporter logs "Events Discarded". On a busy cluster it is being throttled by the Kubernetes API. Raise kubeQPS and kubeBurst in the exporter config.

Trade-offs to know before you rely on them

• stderr is not an error. Many runtimes and frameworks log everything to stderr. Compare the stderr board against the previous period instead of reacting to the absolute number.
• Mean duration is not a percentile. The app requests board shows mean request time. It is good for spotting a shift, not for an SLO.
• One event can stand for many. Kubernetes folds repeats into one Event object and raises count. The boards count documents, so read message and count in the detail tables before treating a low number as a minor issue.
• Events add volume. Normal events are most of the stream. They are useful for freshness and for the scheduled-pods tile, so the pack expects them. If volume matters, drop noisy Normal reasons in the exporter's route rules rather than dropping all Normal events.
• Filebeat or OpenTelemetry. These two packs read Filebeat field names. If you are moving to the OpenTelemetry Collector, the Kubernetes Signals pack is the matching view, and the packs' filters shouldn't double-count the same document if both run during the move.

FAQ

Do I need an event exporter if I already ship container logs?

Yes, for the cluster events pack. Kubernetes events are stored in the API server, not in container log files, and expire after about an hour by default. Filebeat only reads log files, so it never sees them unless an exporter writes them out.

Which index should Kubernetes events go into?

Use a daily index starting kubernetes-events-, for example kubernetes-events-2026.11.09. The pack reads the catch-all *-* pattern, and the Library's data check looks at kubernetes-events-* first.

Do the packs work on AKS, EKS and GKE?

Yes, as long as the fields arrive. The container logs pack reads Filebeat's Kubernetes metadata, which is the same on any distribution. The one catch on managed clusters is index naming: if your pipeline writes AKS logs to an aks-* index, route them to filebeat-* for the dashboards to see them.

Why are the app request dashboards empty?

They only read request logs written as JSON and decoded under logjson, with controller, action, status and duration keys. Services that log plain text leave those two boards empty, and the other four container logs boards still work.

Can I try the Kubernetes packs on the free trial?

Yes. The 14-day free trial includes every feature, including the Dashboard Library and Hosted OpenSearch MCP. Point your Filebeat DaemonSet at a trial Logs stack, run the checks above and install the packs.

Next steps

If you already ship Kubernetes logs with Filebeat, run the three _count checks, open the Dashboard Library and install the container logs pack today, then add an event exporter so the next crash loop arrives with its cause attached. If you're still running your own OpenSearch cluster for Kubernetes logs, a Logit.io Logs stack gives you managed Logstash, OpenSearch and OpenSearch Dashboards with the Dashboard Library built in, so the patching, scaling and index housekeeping stop being your job. If you also need a dedicated cluster for other workloads, see Hosted OpenSearch and OpenSearch pricing. To query the same data from your AI assistant, see Hosted OpenSearch MCP.