As a leading log management software provider, one of our highest priorities is to allow clients to browse the log data they collect so that they can ensure that the data points are consistent and contain all the attributes required to generate the dashboards and reports they need.
The ELK stack is the perfect tool for the job, and combines ElasticSearch, logstash and Kibana 4 to store and visualise huge amount of data in seconds.
Let's take a look at the three components behind the ELK stack:
This is a "schema-less" database, which is easy to scale horizontally and has powerful search capabilities. The "schema-less" tag just means that you just throw lightweight JSON at the database and it will update the schema as you go along.
ElasticSearch indexes every single field, so you can search anything (including full-text search) and it will aggregate and group the data. If you want to register a new node to a cluster then it is a matter of installing ElasticSearch on a machine and editing a config file, which is a pretty simple ask. ElasticSearch then takes care of spreading data around and splitting out requests over multiple servers.
logstash lets you pipeline data to and from anywhere via a tool known as an ETL (Extract, Transform, Load) pipeline in the Data warehousing and Business Analytics world. It also allows us to fetch, transform, and store events into ElasticSearch.
This is a web-based dashboarding and data analysis tool that leverages ElasticSearch's search capabilities so you can visualise your data in seconds. The recent launch of Kibana 4 has also introduced improved analytics tools and allowed for fast analytics as the tool now runs on its own server. For example, you can now export chunks of data easily for separate analysis, as well as share data and dashboards with your coworkers more efficiently.
The combination of these three tools is that you can store and visualise huge amounts of data in minutes. If you're looking for a hosted ELK solution, then we can help you work through the processes to get your ELK stack up and running. If you are adamant about setting up your own Stack then this ELK tutorial should help you get started.
If you enjoyed this post on big data and ELK then why not check out our post on what is DDoS?