By Eleanor Bennett

Elasticsearch

12 min read

The ELK Stack has millions of users globally due to its effectiveness for log management, SIEM, alerting, data analytics, e-commerce site search and data visualisation.

In this extensive guide (updated for 2021) we cover all of the essential basics you need to know to get started with installing ELK, exploring its most popular use cases and the leading integrations you’ll want to start ingesting your logs and metrics data from.

What is the ELK Stack?

The ELK Stack stands for the previously open-source tools Elasticsearch, Logstash, and Kibana (which were open-sourced up to and including version 7.10). The ELK Stack is the most popular solution for log management and analysis and is also known as the Elastic Stack (as of their rebrand, formally announced in October of 2016).

In addition to log management, the ELK stack is used to solve an extensive array of cybersecurity, application performance and observability issues that arise from a lack of data visualisation and monitoring capabilities. ELK is also used for alert configuration and plays a key role in sending alerts to your existing incident management systems.

The first E of ELK is Elasticsearch. Elasticsearch is a powerful & fast search and analytics engine that processes data in real-time. Elasticsearch is well known for its features that allow the search engine to suggest intelligent results based on prior search data and its ability to return accurate results even for terms entered with typos.

The second tool that makes up ELK is Logstash. Logstash works as a powerful extract, transform & load (ETL) tool which serves as a flexible pipeline that collects, enriches and transports data. It works by collecting log messages and forwarding them onto Elasticsearch for visualisation and further processing within Kibana.

The last tool which makes up ELK is Kibana. Kibana is a highly popular visualisation and reporting user interface developed in 2013 by Rashid Khan of the Elastic Company.

Kibana enables its users to create reports and visualisations from a variety of data inputs. Users can create pie charts, heat maps, line graphs, scatter plots and so much more to reflect data in an easily digestible format by using Kibana to host their log and metrics data.

If you want to see what type of use cases Kibana (in association with Elasticsearch and Logstash) can be used for then why not check out our article on the best Kibana dashboard examples.

The Popularity of ELK

Interest in the ELK Stack has grown exponentially since its initial inception due to its ability to fulfil the needs of engineers that require effective and reliable log analysis and data visualisation amongst the other use cases we’ll expand on below.

Each month over 100,000 people across the globe search for the ELK Stack with the primary countries expressing the most interest being the US (17%), Japan (10%), Vietnam (7%), India (6%) and Russia (4%).

ELK Stack's creators Elastic BV have also experienced YoY growth for the past four years running due in part to the popularity of the platform with users for their commercial offering of ELK.

Who Uses ELK?

The ELK stack is used by leading businesses around the globe, some of the most notable include the following:

  • GDIT
  • Adobe
  • Udemy
  • Globant
  • Walmart
  • WebMD
  • SEMrush
  • NTT Data
  • Robinhood
  • Similarweb
  • ResearchGate
  • Procter & Gamble

The following famous companies have self-reported using Elasticsearch:

  • Slack
  • Github
  • CircleCI
  • Instacart
  • Stack Overflow

The following notable companies have stated that they use Logstash:

  • Fiverr
  • Ocado
  • Reddit
  • Indeed
  • TypeForm

And last but not least, the next selection of companies all use Kibana for visualisation purposes:

  • Trivago
  • Hubspot
  • Trust Pilot
  • Hello Fresh
  • DigitalOcean
  • Booking.com

Why Is Log Management Important?

Without centralised log management organisations are vulnerable as a result of not being able to pinpoint log events that signify potential security breaches, attempted hacks, abuse of privileges and errors.

Whilst many professionals that handle data in a variety of formats will find log management and its subsequent analysis useful, the leading roles that commonly find themselves conducting this task tend to be in one of the following departments: DevOps, Development, Data Analyst, System Administration, Cybersecurity and Business Analysts.

The global popularity of Log management is growing steadily and consistently with the most interest being expressed in this solution by companies operating in the US (46%), India (10%), the UK (6%), Vietnam (5%) and Indonesia (4%).

ELK Stack Installation Tutorial

You can run ELK in a variety of different ways, on your own hardware locally, in the cloud, using Docker and for popular configuration management tools like Puppet, Chef and Ansible.

The steps to install the ELK Stack can be quite complex and varied depending on your scenario but here we guide you on how to download, install and get started with Elasticsearch, Logstash, Kibana and Beats using the RPM package which is suitable for installation on Red Hat, Centos, Oracle enterprise and other RPM-based systems.

These steps can be performed on any AWS, Azure or GCP cloud instance ensuring that any security group firewall is configured to allow access from anywhere via SSH and TCP 5601.

Let’s get started

Installing Elasticsearch

Import the Elasticsearch Signing Key

In order to verify the downloaded package we first need to download and install the public signing key. Elasticsearch uses the PGP key D88E42B4, available from https://pgp.mit.edu with fingerprint:

4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4

Download and install the public signing key:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

For RedHat based distributions we need to create a file called elasticsearch.repo in the /etc/yum.repos.d/ directory containing:

[elasticsearch]
name=Elasticsearch repository for OSS 7.x packages
baseurl=https://artifacts.elastic.co/packages/oss-7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

Note: The above configuration installs the Apache 2.0 OSS licensed version of Elasticsearch, to install the version that contains commercially licensed features use the following repository

name=https://artifacts.elastic.co/packages/7.x/yum

You can now proceed to install Elasticsearch using the repository that we have configured above, using the following command:

sudo yum install --enablerepo=elasticsearch elasticsearch

Some of the commercial features enable automatic index creation in the non-OSS version of Elasticsearch, of course, if using the Logit.io log management platform we take care of index management for you.

To start and stop Elasticsearch with SysV init run the following command

sudo -i service elasticsearch start
sudo -i service elasticsearch stop

To start and stop Elasticsearch with systemd run the following command

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

To test that Elasticsearch is running we can send an HTTP request to port 9200 on localhost or point your browser to http://localhost:9200, we should see a similar output to that below:

{
  "name" : "****",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "****",
  "version" : {
"number" : "7.1.1",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "",
"build_date" : "****",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0"
  },
  "tagline" : "You Know, for Search"
}

Installation in a highly available Elasticsearch cluster architecture requires some additional configuration, of which we will cover in a future blog post.

Installing Logstash

Logstash has a dependency on Java and requires Java 8, Java 11 or Java 15. You can use the official Oracle distribution or an open-source version like OpenJDK.

Verify your Java version with the following command:

java -version

We’ve already verified the signing key above and set up the repository, so we can go ahead and install Logstash with the following command:

sudo yum install logstash

Now we can start Logstash.

To start and stop Logstash with SysV init run the following command

sudo -i service logstash start
sudo -i service logstash stop

To start and stop Logstash with systemd run the following command

sudo systemctl start logstash.service
sudo systemctl stop logstash.service

Installing Kibana

Run the following command to install Kibana using the repository that we configured earlier:

sudo yum install kibana

In order to tell Kibana which Elasticsearch to connect to, modify the Kibana configuration yaml file found which can be found at /etc/kibana/kibana.yml and make sure you have specified the server.port and elasticsearch.url as shown below:

server.port: 5601 elasticsearch.url: "http://localhost:9200"

We can now start Kibana.

To start and stop Kibana with SysV init run the following command

sudo -i service kibana start
sudo -i service kibana stop

To start and stop Kibana with systemd run the following command

sudo systemctl start kibana.service
sudo systemctl stop kibana.service

If you then browse to http://localhost:5601 you will be directed to the initial Kibana dashboard screen.

Installing Beats

Beats are a family of shippers that allow you to simply and quickly send data to Logstash and/or Elasticsearch.

We would recommend using Filebeat to get started shipping your logs and metrics to Logstash. Filebeat takes a file path to allow you to specify the location of your logs. Let’s start by installing Filebeat:

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-7.8.1-x86_64.rpm
sudo rpm -vi filebeat-oss-7.8.1-x86_64.rpm

Locate the configuration file:

/etc/filebeat/filebeat.yml

The default filebeat.inputs sections looks as below and will be looking for log files in /var/log/*.log, you can change this path to point to the location where your log files are generated:

filebeat.inputs:

- type: log
  # Change to true to enable this input configuration.
  enabled: false
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
  - /var/log/*.log

We can now validate any configuration changes by running the following command:

sudo filebeat -e -c /etc/filebeat/filebeat.yml

If you receive no configuration errors after running the check, let’s proceed and start Filebeat:

sudo systemctl enable filebeat
sudo systemctl start filebeat

If we have everything configured correctly a new Logstash index should have been created in Elasticsearch, let’s take a closer look.

Open Kibana and browse to Management > Kibana Index Patterns, we would expect to see a Logstash Index as well as a Filebeat index for the beat we are using.

Choose Create Index Pattern, entering logstash-* as the Index pattern and selecting @timestamp for the time filter field, then choose Create.

unnamed (4) (1)

You’re now ready to dive in and look for insights in your data! Click on the discover tab in Kibana and choose ‘Today’ from the date picker to ensure we are seeing all the data available.

For a more detailed guide on configuring Filebeat take a look at all of our Beats source integration guides.

The Leading Use Cases For ELK

The most popular use case for the ELK Stack is for log management and analysis. In addition to this, there are many more highly valuable reasons to use ELK for reporting, alerting and improving your observability which we have listed below:

  • SIEM
  • Alerting
  • Live tailing
  • Social listening
  • Data streaming
  • Troubleshooting
  • Tracking referrals
  • Website analytics
  • Server monitoring
  • Metrics management
  • Vulnerability scanning
  • Kubernetes monitoring
  • Compliance and auditing
  • Monitoring website uptime
  • Automated test monitoring
  • Measuring sales performance
  • Understanding user behaviour

This list, while extensive, will be expanded on in the future based on other uses we observe ELK being used to facilitate.

The Most Popular Integrations For ELK

An attribute that many users of ELK find useful is its high compatibility with processing logs and metrics from a wide variety of data sources.

This means that it doesn’t matter if your data source originates from an operating system, application, cloud service or programming language, ELK will be able to transform your unstructured and semi-structured data into data that can be easily reported upon and unlocked for the valuable insights it holds.

We’ve selected the integrations listed below based on their popularity with thousands of hosted ELK users who use the ELK stack on a regular basis.

Azure:

AWS:

Google Cloud:

Operating Systems:

Applications:

View all of the rest of the integrations for ELK.

Don’t have the time to build, configure and host your Elastic Stack locally? Then why not get started with hosted ELK from Logit.io.

With our no credit card required 14-day free trial you can launch Stacks within minutes and explore the full potential of Elasticsearch, Logstash, Kibana, Grafana & Open Distro for Elasticsearch.

If you enjoyed this guide and you wish to continue learning about ELK then why not check out our previous blog on Kibana vs Grafana?

backReturn to Blog