By Eleanor Bennett
6 min read
In this post, we are going to cover some of the most common misconceptions and queries users have about AWS Elasticsearch, Open Distro & OpenSearch as these terms are often being used interchangeably to refer to any Amazon backed distribution of Elasticsearch and Kibana.
The Early Days
Three years ago the first iteration of AWS’s Open Distro for Elasticsearch looked like a combination of different repositories which included both Elasticsearch and Searchguard. At this early stage, the aim to combine Elasticsearch with out of the box security features was still in progress and was nowhere near production-ready.
The open-source version of the ELK Stack (where Elasticsearch originated from in the first place) did not come supplied with out of the box security features, to the dismay of many users at the time who wanted an alternative to having to use Elastic’s offering of Xpack.
Prior to Elastic choosing to release all future ELK versions under SSPL licensing (as of 2021), there were many users that found Xpack to not be as affordable as they would need to continue building their Stacks securely.
For users who wished to have access to something as simple as privileged user controls, they found that this was a paid-for feature when they saw this as something vital for the integrity of their data’s security.
Back in this AWS backed project's infancy, AWS didn't provide Kibana multi-tenancy or role-based access controls either. Fortunately, this is no longer the case and the AWS backed OpenSearch Dashboards includes these features as part and parcel of the current edition.
AWS Elasticsearch vs OpenSearch vs Open Distro for Elasticsearch (ODFE)
With all of these terms coined in the wake of Elastic’s decision to make the ELK Stack closed-source, it can be very confusing to know the difference between the new distributions of Elasticsearch and Kibana being offered by Amazon Web Services and other cloud/Saas platforms.
AWS Elasticsearch Service (Amazon ES) referred to the proprietary service Amazon had offered since 2015, this solution offered managed Elasticsearch as a service.
Amazon OpenSearch Service is the successor to Amazon Elasticsearch Service and offers the latest version of OpenSearch as a managed solution. This news was announced on September 8th 2021.
Open Distro for Elasticsearch (ODFE) was the previous name for OpenSearch, on the official website for ODFE an announcement was made on June 25 2021 that all users should refer to the new OpenSearch site for news and updates as the previous site sporting the old branding will be selectively decommissioned in time.
OpenSearch and OpenSearch dashboards are the current rebrand of Open Distro for Elasticsearch which was also announced on September 8th, 2021. This is the latest fully open-source version of the Elastic Stack.
All of the terms above are directly connected to AWS’s involvement with the distribution of Elasticsearch and Kibana, with Logstash having been excluded from the development process.
When was the Amazon Elasticsearch Service Renamed?
The Amazon Elasticsearch Service was renamed to Amazon OpenSearch Service on September 8th 2021 according to the official AWS open-source blog.
Why Are There Two Amazon Backed Projects Named OpenSearch?
Some users may be confused by the existence of two OpenSearch projects, a 2005 version versus the 2021 version. The 2005 edition of OpenSearch referred to a format for syndication and aggregation of web results within a browser.
Whilst both versions of OpenSearch are AWS led projects only the later version is the Lucene based distributed search engine inspired by a fork of Elastic.
There are quite a few interconnected threads between the two projects as technology from A9 (a former subsidiary of Amazon) was migrated into CloudSearch, which itself has engineers who work across both CloudSearch & OpenSearch, but nowadays both projects are largely independent from each other.
What are OpenSearch Dashboards?
OpenSearch Dashboards is the new name for Amazon’s fork of Kibana 7.10.2 which aims to offer improvements on the previous version of Kibana by offering additional fine-grained access controls.
OpenSearch dashboards serve as a suitable replacement for Kibana by offering a browser-based UI and the option to use visualisations and charts, just like you could do with its Elastic backed predecessor.
What is AWS ELK?
Another term that is increasingly used is AWS ELK. This term isn’t entirely accurate to say as the Amazon Web Services distribution of Elasticsearch and Kibana does not include the L of the ELK Stack, which is the extract, transform and load(ETL) tool, Logstash.
Is OpenSearch Production Ready?
OpenSearch became production-ready as of July 2021 when the release of OpenSearch version 1.0 was announced.
Will Beats Be Able To Ship Data To ODFE/OpenSearch?
According to Elastic's note regarding breaking changes in version 7.13, Beats may not be able to send data to Open Distro, OpenSearch or any versions of Elasticsearch released after version 7.10.2.
Is LogTrail Available For OpenSearch Dashboards?
Unfortunately as of 2021, the LogTrail live tailing plugin for Kibana has not been forked and adapted to be compatible with OpenSearch Dashboards. To keep an eye on if this is a feature being developed soon or if you wish to have your say then you may want to add your thoughts to the current roadmap.
How Can I Access AWS Elasticsearch/Open Distro/OpenSearch?
To access Amazon’s latest offering of Elasticsearch (OpenSearch) and Kibana (OpenSearch Dashboards) you will configure a Docker host to set up a two-node cluster to support both OpenSearch and OpenSearch dashboards.
If you instead want to experience the best features of OpenSearch and OpenSearch Dashboards easily with a platform that is ready to launch within minutes, then consider signing up for a free trial of Logit.io.
Logit.io saves engineers and technicians around the world from the tedious setup that is often associated and the subsequent maintenance and optimisation of Open-Source tools. The platform also provides managed Grafana and versions of the ELK Stack released prior to edition 7.10.2 for specific use cases.
What Will Happen To Clients Previously Used With Elasticsearch?
According to the August 4th update published by AWS, there are plans to fork the following clients in order to offer the same APIs and functionality they had in pre-existing versions;
Why Should You Consider A Managed Service?
If you prefer to have the option to use OpenSearch, Open Distro or another legacy version of the ELK Stack then you should consider using a managed service that hosts these solutions as ready to launch Stacks. Logit.io provides a highly affordable solution for hosting Open Distro, ELK and hosted OpenSearch side by side.
A managed service offered on top of open-source software can remedy many security and governance gaps left in the wake of the original providers wishing to monetise crucial elements essential for ensuring compliance.
Unless you have extensive experience with running enterprise-level ELK then it is highly likely that your team will become overwhelmed at the prospect of maintaining this complex and fast-evolving technology Stack.
Many teams that have tried to host their own deployment of either OpenSearch or the Elastic Stack soon find that they don't have the additional resources required to configure, maintain and manage multiple Elasticsearch instances on top of their existing workload.
Attempting to appropriately scale and configure shards without prior knowledge of Elastic software can be highly complex, this is where a managed solution such as Logit.io comes into its own. With Logit.io you can rapidly deploy ELK Stacks within minutes of registering and launch side by side Open Distro, OpenSearch and ELK Stacks that includes Kibana and Grafana.
If you enjoyed this article then why not find out what Elasticsearch is used for.