Send data via Kubernetes Metrics to your Logstash instance provided by Logit.io

Kubernetes Metrics

Collect and ship Kubernetes container service metrics to Logstash and Elasticsearch.

Metricbeat is a lightweight shipper that helps you monitor Kubernetes by collecting metrics from the containers running on the host system. Configure Metricbeat using the pre-defined examples below to collect and ship Kubernetes container service metrics and statistics to Logstash or Elasticsearch.

Step 1 - Download manifest

Download the metricbeat kubernetes deploy manifest using the link below:

Click to download metricbeatbeat-kubernetes.yaml

Step 2 - Insert stack details

Open the manifest and look for the environment variables controlling the logging destination. (Lines 110 and 236) We need to change environment variables on both lines 110 and 236 because Kubernetes runs a metricbeat instance on each node in the cluster. And also an additional single instance of Metricbeat to retrieve metrics for the cluster itself.

env:
- name: LOGSTASH_HOST
  value: "your-logstash-host"
- name: BEATS_PORT
  value: "your-port"

Step 3 - Configure module

Each module has its own configuration file where different metricsets can be enabled / disabled. Locate the metricbeatbeat-kubernetes.yaml configuration file you downloaded in step 1.

These are the default metricsets available for the kubernetes module. To enable or disable a metric simply comment or uncomment the line out.

- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - container
    - volume
  period: 10s
  host: ${NODE_NAME}
  hosts: ["localhost:10255"]

There is also a system module enabled for the Kubernetes node, these are the default metricsets enabled. To enable or disable a metric simply comment or uncomment the line out.

- module: system
  period: 10s
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    #- core
    #- diskio
    #- socket
  processes: ['.*']
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

The Kubernetes cluster also has some default metricsets available. Metricbeat requires kube-state-metrics to be deployed and running to gather the cluster metrics. Follow the Kuberenetes deployment docs to get this set-up.

- module: kubernetes
  metricsets:
    - state_node
    - state_deployment
    - state_replicaset
    - state_pod
    - state_container
    # Uncomment this to get k8s events:
    #- event
  period: 10s
  host: ${NODE_NAME}
  hosts: ["kube-state-metrics:8080"]

Step 4 - Deploy pod

Now your deployment manifest is updated, you can deploy it using.

kubectl apply -f metricbeat-kubernetes.yaml

Step 5 - Confirm deployment completed

Confirm that your pod has successfully been deployed using:

kubectl --namespace=kube-system get ds/metricbeat

kubectl --namespace=kube-system get pods

You should see a pod for each kubernetes node with a name similar to metricbeat-abcde listed. The pods should work though from Pending to Running within a couple of minutes as the containers are downloaded and started.

Step 6 - Compatability

The Kubernetes module is tested with Kubernetes 1.8.0, 1.9.4 and 1.10.0.

Step 7 - Kubernetes module dashboard

The Kubernetes module comes with a predefined Kibana dashboard. To view your dashboards for any of your Logit stacks, launch Kibana and choose Dashboards.

expand view

Expand View

compact view

Compact View

Return to Search
Sign Up