ELK
2 min read
Digitalgov.gov is the Digital Government Division of the US government. One of the most important things that they do is provide the search technology for over 1,200 government websites. This also means that they are responsible for helping their client agencies analyse and visualise their search data, so that they can improve their service to users. Digitalgov effectively act as a massive SaaS provider to government agencies.
Since May 2014, Digitalgov has provided this service using the ELK stack (http://www.digitalgov.gov/2014/05/07/analyzing-search-data-in-real-time-to-drive-decisions). Their use of Kibana dashboards combined with Elasticsearch are allowing their users to see patterns in the huge volumes of data that their clients generate. They log and combine search terms, and also store analytics data on clicks and pageloads, along with more standard error logging.
There can't be many use cases of quite this sort of scale, so it's interesting to look at what Digitalgov have done, just to get an idea of how big a really big SaaS ELK solution can be. They posted a really interesting blog article at the start of this year going through their thinking about how they migrated from their previous system to ELK (http://www.digitalgov.gov/2015/01/07/elk). It's well worth a read.
Three things really stand out from the Digitalgov experience. First, they really did their homework in terms of making sure that Logstash and Elasticsearch were up to the job. The sites that they support really needed total reliability, and as a SaaS ELK provider, they were responsible for delivering it.
Second, the ELK stack replaced an extremely complicated legacy structure based on Hadoop, Solr and MySQL with a streamlined and integrated suite of programs. This goes to show how the ELK stack is the best way of handling your logging requirements; it's specialised to the task, so it doesn't need as much patchwork and infrastructure to get it to work.
Finally, you can see what a massive job it was and how hard Digitalgov had to work to get it done. If every single agency had to install their own ELK stack, it would have been chaos. The advantage of a SaaS ELK provider is that they can handle the tricky bits and the testing, and leave the users to work with their data. That's the reason we at logit.io believe that hosted ELK solutions are the way to go for users of all kinds, not just the US government!
If you liked this article about how DigitalGov uses ELK in their day to day operations then why not check out our guide to BABOK, or our guide to the best incident management tools.