Kubernetes Metrics

Download the metricbeat kubernetes deploy manifest using the link below:

Click to download metricbeatbeat-kubernetes.yaml

Open the manifest and look for the environment variables controlling the logging destination. (Lines 132 and 261) We need to change environment variables on both lines 138 and 267 because Kubernetes runs a metricbeat instance on each node in the cluster. And also an additional single instance of Metricbeat to retrieve metrics for the cluster itself.

Each module has its own configuration file where different metricsets can be enabled / disabled. Locate the metricbeatbeat-kubernetes.yaml configuration file you downloaded in step 1.

These are the default metricsets available for the kubernetes module. To enable or disable a metric simply comment or uncomment the line out.

- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - container
    - volume
  period: 10s
  host: ${NODE_NAME}
  hosts: ["localhost:10255"]
- module: kubernetes
  enabled: true
  metricsets:
    - apiserver
  hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
- module: kubernetes
  enabled: true
  metricsets:
    - proxy
  hosts: ["localhost:10249"]
  period: 10s
- module: kubernetes
  enabled: true
  metricsets:
    - controllermanager
  hosts: ["http://localhost:10252"]
  period: 10s
- module: kubernetes
  enabled: true
  metricsets:
    - scheduler
  hosts: ["localhost:10251"]
  period: 10s

There is also a system module enabled for the Kubernetes node, these are the default metricsets enabled. To enable or disable a metric simply comment or uncomment the line out.

- module: system
  period: 10s
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    #- core
    #- diskio
    #- socket
  processes: ['.*']
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

The Kubernetes cluster also has some default metricsets available. Metricbeat requires kube-state-metrics to be deployed and running to gather the cluster metrics. Follow the Kubernetes deployment docs to get this set-up.

- module: kubernetes
  metricsets:
    - state_node
    - state_deployment
    - state_replicaset
    - state_pod
    - state_container
    # Uncomment this to get k8s events:
    #- event
  period: 10s
  host: ${NODE_NAME}
  hosts: ["kube-state-metrics:8080"]

Now your deployment manifest is updated, you can deploy it using.

kubectl apply -f metricbeat-kubernetes.yaml

Confirm that your pod has successfully been deployed using:

kubectl --namespace=kube-system get ds/metricbeat

kubectl --namespace=kube-system get pods

You should see a pod for each kubernetes node with a name similar to metricbeat-abcde listed. The pods should work though from Pending to Running within a couple of minutes as the containers are downloaded and started.

If you don't see data appearing in your Stack after following the steps, visit the Help Centre guide for steps to diagnose no data appearing in your Stack or Chat to support now.

The Kubernetes module dashboards are tested with Kubernetes 1.13.0 & 1.14.0 Dashboards for controllermanager scheduler and proxy are not compatible with kibana versions below 7.2.0

The Kubernetes module comes with predefined Kibana dashboards. To view your dashboards for any of your Logit.io stacks, launch Kibana and choose Dashboards.