Start your 14-day free trial today & Get 20% Off All Annual Managed ELK Plans

No Credit Card Required

Try Free

Already have an account? Sign In

Send data via Kubernetes Metrics to your Logstash instance provided by

Kubernetes Metrics

Collect and ship Kubernetes container service metrics to Logstash and Elasticsearch.

Metricbeat is a lightweight shipper that helps you monitor Kubernetes by collecting metrics from the containers running on the host system. Configure Metricbeat using the pre-defined examples below to collect and ship Kubernetes container service metrics and statistics to Logstash or Elasticsearch.

Step 1 - Download manifestCopy

Download the metricbeat kubernetes deploy manifest using the link below:

Click to download metricbeatbeat-kubernetes.yaml

Step 2 - Insert stack detailsCopy

Open the manifest and look for the environment variables controlling the logging destination. (Lines 132 and 261) We need to change environment variables on both lines 138 and 267 because Kubernetes runs a metricbeat instance on each node in the cluster. And also an additional single instance of Metricbeat to retrieve metrics for the cluster itself.

No input available! Your stack is missing the required input for this data source Talk to support to add the input

Step 3 - Configure moduleCopy

Each module has its own configuration file where different metricsets can be enabled / disabled. Locate the metricbeatbeat-kubernetes.yaml configuration file you downloaded in step 1.

These are the default metricsets available for the kubernetes module. To enable or disable a metric simply comment or uncomment the line out.

- module: kubernetes
    - node
    - system
    - pod
    - container
    - volume
  period: 10s
  host: ${NODE_NAME}
  hosts: ["localhost:10255"]
- module: kubernetes
  enabled: true
    - apiserver
- module: kubernetes
  enabled: true
    - proxy
  hosts: ["localhost:10249"]
  period: 10s
- module: kubernetes
  enabled: true
    - controllermanager
  hosts: ["http://localhost:10252"]
  period: 10s
- module: kubernetes
  enabled: true
    - scheduler
  hosts: ["localhost:10251"]
  period: 10s

There is also a system module enabled for the Kubernetes node, these are the default metricsets enabled. To enable or disable a metric simply comment or uncomment the line out.

- module: system
  period: 10s
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    #- core
    #- diskio
    #- socket
  processes: ['.*']
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

The Kubernetes cluster also has some default metricsets available. Metricbeat requires kube-state-metrics to be deployed and running to gather the cluster metrics. Follow the Kubernetes deployment docs to get this set-up.

- module: kubernetes
    - state_node
    - state_deployment
    - state_replicaset
    - state_pod
    - state_container
    # Uncomment this to get k8s events:
    #- event
  period: 10s
  host: ${NODE_NAME}
  hosts: ["kube-state-metrics:8080"]

Step 4 - Deploy podCopy

Now your deployment manifest is updated, you can deploy it using.

kubectl apply -f metricbeat-kubernetes.yaml

Step 5 - Confirm deployment completedCopy

Confirm that your pod has successfully been deployed using:

kubectl --namespace=kube-system get ds/metricbeat

kubectl --namespace=kube-system get pods

You should see a pod for each kubernetes node with a name similar to metricbeat-abcde listed. The pods should work though from Pending to Running within a couple of minutes as the containers are downloaded and started.

Step 6 - how to diagnose no data in StackCopy

If you don't see data appearing in your Stack after following the steps, visit the Help Centre guide for steps to diagnose no data appearing in your Stack or Chat to support now.

Step 7 - CompatibilityCopy

The Kubernetes module dashboards are tested with Kubernetes 1.13.0 & 1.14.0 Dashboards for controllermanager scheduler and proxy are not compatible with kibana versions below 7.2.0

Step 8 - Kubernetes module dashboardCopy

The Kubernetes module comes with predefined Kibana dashboards. To view your dashboards for any of your stacks, launch Kibana and choose Dashboards.

Prebuilt Kibana dashboards

Controller manager



Toggle View

Expand View

Return to Search

© 2022 Ltd, All rights reserved.