AWS Elastic Kubernetes Service
Ship AWS EKS Metrics to Logstash
Metricbeat is an open source shipping agent that lets you ship AWS Elastic Kubernetes Service (EKS) Metrics to one or more destinations, including Logstash.
Follow this step by step guide to get 'logs' from your system to Logit.io:
Step 1 - Prerequisites
Firstly Make sure you have the aws cli, eksctl & kubectl installed on local machine using the following guide
Also make sure you have setup aws configure with your AWS credentials.
To do this run run the following command in your aws terminal.
aws configure
When you type this command, the AWS CLI prompts you for four pieces of information: access key, secret access key, AWS Region, and output format. This information is stored in a profile named default. This profile is used when you run commands, unless you specify another one.
Step 2 - Connecting to the cluster
Update your config by running the following command. Replace <enter_region>
and <enter_name>
with your AWS cluster region and name.
aws eks --region <enter_region> update-kubeconfig --name <enter_name>
Check you can connect to your cluster by running the following command:
kubectl get svc
Step 3 - kube-state-metrics
You will need an instance of kube-state-metrics running on your cluster before you install metricbeat.
Cloning the kube-state-metrics repository locally and applying the example is the easiest way to generate some metrics. You can tailor these to your needs at a later stage.
git clone https://github.com/kubernetes/kube-state-metrics.git
Change into the newly downloaded directory:
cd go/src/k8s.io/kube-state-metrics/
Then to deploy this project, you can simply run:
kubectl apply -f examples/standard
Step 4 - Deploy Metricbeat
You're going to need the metricbeat deployment manifest.
curl -L -O cdn.logit.io/metricbeat-kubernetes-7.10.0.yaml
Now you have the manifest we need to add your Stack Logstash endpoint details. Open the file in the terminal editor using the vim or nano command. Alternatively you can choose Open Editor from the console toolbar to make your changes.
vi metricbeat-kubernetes-7.10.0.yaml
or
nano metricbeat-kubernetes-7.10.0.yaml
Update the following lines in the yaml with your Stack Logstash endpoint and Beats-SSL port.
env:
- name: LOGSTASH_HOST
value: "guid-ls.logit.io"
- name: BEATS_PORT
value: "00000"
After updating the code should look as below.
env:
- name: LOGSTASH_HOST
value: ["your-logstash-host"]
- name: BEATS_PORT
value: ["your-ssl-port"]
Exit and save the file.
Step 5 - Apply your updates
Now we're going to apply the file to the cluster.
kubectl apply -f metricbeat-kubernetes-7.10.0.yaml
Step 6 - Confirm Deployment
Confirm your pod has deployed, you should see output similar to that below.
kubectl get po -A
or
kubectl logs ["podname"] --namespace=kube-system
Browse to Kibana and you should see Metrics arriving in your Stack.
Step 7 - Check Logit.io for your logs
Now you should view your data:
If you don't see logs take a look at How to diagnose no data in Stack below for how to diagnose common issues.
Step 8 - how to diagnose no data in Stack
If you don't see data appearing in your Stack after following the steps, visit the Help Centre guide for steps to diagnose no data appearing in your Stack or Chat to support now.