Opensearch vs Elasticsearch: Features and Comparison

The Early Days

Five years ago the first iteration of AWS’s Open Distro for Elasticsearch looked like a combination of different repositories which included both Elasticsearch and Searchguard. At this early stage, the aim to combine Elasticsearch with out of the box security features was still in progress and was nowhere near production-ready.

The open-source version of the ELK Stack (where Elasticsearch originated from in the first place) did not come supplied with out of the box security features, to the dismay of many users at the time who wanted an alternative to having to use Elastic’s offering of Xpack.

Prior to Elastic choosing to release all future ELK versions under SSPL licensing (2021), there were many users that found Xpack to not be as affordable as they would need to continue building their Stacks securely.

For users who wished to have access to something as simple as privileged user controls, they found that this was a paid-for feature when they saw this as something vital for the integrity of their data’s security.

Back in this AWS backed project's infancy, AWS didn't provide Kibana multi-tenancy or role-based access controls either. Fortunately, this is no longer the case and the AWS backed OpenSearch Dashboards includes these features as part and parcel of the current edition.

What is a Shard in Elasticsearch?

In Elasticsearch, a shard is a basic unit of data storage and distribution. Elasticsearch uses a distributed architecture to store and manage large volumes of data efficiently, and shards play a crucial role in this architecture.

AWS Elasticsearch vs OpenSearch vs Open Distro for Elasticsearch (ODFE)

With all of these terms coined in the wake of Elastic’s decision to make the ELK Stack closed-source, it can be very confusing to know the difference between the new distributions of Elasticsearch and Kibana being offered by Amazon Web Services and other cloud/Saas platforms.

In brief;

AWS Elasticsearch Service (Amazon ES) referred to the proprietary service Amazon had offered since 2015, this solution offered managed Elasticsearch as a service.

Amazon OpenSearch Service is the successor to Amazon Elasticsearch Service and offers the latest version of OpenSearch as a managed solution. This news was announced on September 8th 2021.

Open Distro for Elasticsearch (ODFE) was the previous name for OpenSearch, on the official website for ODFE an announcement was made on June 25 2021 that all users should refer to the new OpenSearch site for news and updates as the previous site sporting the old branding will be selectively decommissioned in time.

OpenSearch and OpenSearch dashboards are the current rebrand of Open Distro for Elasticsearch which was also announced on September 8th, 2021. This is the latest fully open-source version of the Elastic Stack.

All of the terms above are directly connected to AWS’s involvement with the distribution of Elasticsearch and Kibana, with Logstash having been excluded from the development process.

When was the Amazon Elasticsearch Service Renamed?

The Amazon Elasticsearch Service was renamed to Amazon OpenSearch Service on September 8th 2021 according to the official AWS open-source blog.

Why Are There Two Amazon Backed Projects Named OpenSearch?

Some users may be confused by the existence of two OpenSearch projects, a 2005 version versus the 2021 version. The 2005 edition of OpenSearch referred to a format for syndication and aggregation of web results within a browser.

Whilst both versions of OpenSearch are AWS led projects only the later version is the Lucene based distributed search engine inspired by a fork of Elastic.

There are quite a few interconnected threads between the two projects as technology from A9 (a former subsidiary of Amazon) was migrated into CloudSearch, which itself has engineers who work across both CloudSearch & OpenSearch, but nowadays both projects are largely independent from each other.

What are OpenSearch Dashboards?

OpenSearch Dashboards is the new name for Amazon’s fork of Kibana 7.10.2 which aims to offer improvements on the previous version of Kibana by offering additional fine-grained access controls.

OpenSearch dashboards serve as a suitable replacement for Kibana by offering a browser-based UI and the option to use visualisations and charts, just like you could do with its Elastic backed predecessor.

What is AWS ELK?

Another term that is increasingly used is AWS ELK. This term isn’t entirely accurate to say as the Amazon Web Services distribution of Elasticsearch and Kibana does not include the L of the ELK Stack, which is the extract, transform and load(ETL) tool, Logstash.

Is OpenSearch Production Ready?

OpenSearch became production-ready in July 2021 when the release of OpenSearch version 1.0 was announced.

Will Beats Be Able To Ship Data To ODFE/OpenSearch?

According to Elastic's note regarding breaking changes in version 7.13, Beats may not be able to send data to Open Distro, OpenSearch or any versions of Elasticsearch released after version 7.10.2.

Is LogTrail Available For OpenSearch Dashboards?

Unfortunately since 2021, the LogTrail live tailing plugin for Kibana has not been forked and adapted to be compatible with OpenSearch Dashboards. To keep an eye on if this is a feature being developed soon or if you wish to have your say then you may want to add your thoughts to the current roadmap.

How Can I Access AWS Elasticsearch/Open Distro/OpenSearch?

To access Amazon’s latest offering of Elasticsearch (OpenSearch) and Kibana (OpenSearch Dashboards) you will configure a Docker host to set up a two-node cluster to support both OpenSearch and OpenSearch dashboards.

If you instead want to experience the best features of OpenSearch and OpenSearch Dashboards easily with a platform that is ready to launch within minutes, then consider signing up for a free trial of Logit.io.

Logit.io saves engineers and technicians around the world from the tedious setup that is often associated and the subsequent maintenance and optimisation of Open-Source tools. The platform also provides managed Grafana and versions of the ELK Stack released prior to edition 7.10.2 for specific use cases.

What Will Happen To Clients Previously Used With Elasticsearch?

According to the August 4th update published by AWS, there are plans to fork the following clients in order to offer the same APIs and functionality they had in pre-existing versions;

• elasticsearch-py
• elasticsearch-java
• elasticsearch-net
• go-elasticsearch
• elasticsearch-js
• elasticsearch-ruby
• eland
• elasticsearch-php
• elasticsearch-rs
• elasticsearch-perl
• elasticsearch-specification
• elasticsearch-hadoop

OpenSearch: Latest Features and Updates

The latest version of OpenSearch is OpenSearch 2.10, released in September 2023. One of the new features is improved durability with remote-backed storage. With this update, integrated with OpenSearch segment replication, remote-backed storage supplies enhanced performance and durability, simultaneously decreasing the need for storage and computing resources.

Another improvement from this update is the changes to the OpenSearch dashboard UI. To modernize the visualizations and dashboard experience, a new theme has been added. The "Next (preview)" theme offers improvements to typography, colors, and actions, ensuring a more user-friendly environment for Dashboards. These enhancements seek to decrease cognitive load for users and apply to both light and dark mode designs within the theme.

A final feature from the latest OpenSearch update is the improvement of the Discover tool. Building on OpenSearch’s commitment to producing a more intuitive and cohesive tool, users now have the option to utilize either the updated version of the Discover tool or its previous iteration.

You can read the full list of updates in OpenSeearch 2.10 here.

Elasticsearch: Latest Features and Updates

The most up-to-date version of Elasticsearch was launched in November 2023 and is Elasticsearch 8.11. The main update of the new release is a new query language ES|QL. Elasticsearch Query Language (ES|QL) offers Elastic users a versatile, powerful, and robust query expression language for interrogating data. Also, the language improves the query user experience by incorporating post-ingest processing abilities. This transformation significantly expands the analytics and data processing capabilities of Elasticsearch.

Another feature of the latest Elasticsearch update is that the datastream lifecycle is now found in Technical Preview. Built-in streamlined and robust lifecycle implementation is now available for data streams, allowing them to automatically undergo rollover and tail merge when configured with a lifecycle.

You can read the full list of updates and changes to Elasticsearch in their latest release 8.11 here.

Side By Side Comparison of OpenSearch vs Elasticsearch

Origins and Community:

• Elasticsearch is an open-source search and analytics engine developed by Elastic. It was originally released in 2010 and gained significant popularity as a distributed search and data processing platform.
• OpenSearch is a community-driven, open-source search engine based on the code from Elasticsearch 7.10.2. It emerged as a fork of Elasticsearch following Elastic's decision to change the license of certain components.

Governance and Ownership:

• Elasticsearch is owned and maintained by Elastic, a company that provides commercial products and services built around Elasticsearch.
• OpenSearch is governed by the OpenSearch community, which includes various organizations and contributors. The project is developed in collaboration and aims to provide a transparent, vendor-neutral, and community-driven alternative to Elasticsearch.

Licensing:

• Elasticsearch changed its license to the Server Side Public License (SSPL) in January 2021. This license places restrictions on the use of Elasticsearch in certain scenarios, particularly for cloud service providers.
• OpenSearch is licensed under the Apache 2.0 license, which is more permissive and aligns with the open-source philosophy. It allows users to freely use, modify, and distribute the software without restrictive licensing conditions.

Features and Compatibility:

• Elasticsearch and OpenSearch share a common codebase from Elasticsearch 7.10.2. Therefore, they have similar features and functionalities.
• However, as time progresses, Elasticsearch and OpenSearch may diverge in terms of features and enhancements. Each project may develop and introduce new features independently based on the requirements and contributions of their respective communities.

Ecosystem and Compatibility:

• Elasticsearch has a well-established ecosystem with extensive documentation, official support, and a wide range of integrations and plugins. Many commercial products and services are built around Elasticsearch.
• OpenSearch aims to maintain compatibility with Elasticsearch, especially in terms of APIs and plugins. The project also encourages contributions from the community to expand the ecosystem and provide compatible tools and integrations.

Branding and Naming:

• Prior to the OpenSearch fork, the software released by Elastic was called Elasticsearch. It had gained significant popularity and recognition under this name.
• OpenSearch is the result of a community-driven fork of Elasticsearch. It was created to ensure the continued development and availability of an open-source search engine that is free from licensing restrictions. OpenSearch includes a suite of projects, including OpenSearch and OpenSearch Dashboards.

Community and Governance:

• Elasticsearch has a well-established and active community, with Elastic leading the development and direction of the project. While Elastic encourages community contributions, the decision-making and governance primarily rest with Elastic as the primary commercial entity.
• OpenSearch places a strong emphasis on community-driven development and governance. It aims to provide a more inclusive and collaborative environment for decision-making, with multiple organizations and contributors participating in the project's direction and development.

License Differences:

• In response to concerns about the licensing changes made by Elastic, OpenSearch emerged as a community-driven fork of Elasticsearch. The new license introduced by Elastic, the Server Side Public License (SSPL), raised questions and limitations, especially for cloud service providers.
• OpenSearch addresses the licensing concerns by using the Apache 2.0 license. This license is widely recognized and allows users the freedom to use, modify, and distribute the software without significant restrictions.

Compatibility and Interoperability:

• Elasticsearch and OpenSearch share a common codebase from Elasticsearch 7.10.2. As a result, they have similar APIs, query languages (e.g., Elasticsearch Query DSL), and data formats.
• OpenSearch aims to maintain compatibility with Elasticsearch to ensure a smooth transition for existing users. The project actively encourages developers to contribute compatible plugins, tools, and integrations to expand the OpenSearch ecosystem.

Long-Term Development and Divergence:

• While OpenSearch initially forked from Elasticsearch 7.10.2, both projects have evolved independently over time. Each project's development is driven by the respective communities and their unique goals and priorities.
• OpenSearch intends to follow a transparent and community-driven development process, taking into account the needs and feedback of its user base.

Why Should You Consider A Managed Service?

If you prefer to have the option to use OpenSearch, Open Distro or another legacy version of the ELK Stack then you should consider using a managed service that hosts these solutions as ready to launch Stacks. Logit.io provides a highly affordable solution for hosting Open Distro, ELK and hosted OpenSearch side by side.

A managed service offered on top of open-source software can remedy many security and governance gaps left in the wake of the original providers wishing to monetise crucial elements essential for ensuring compliance.

Unless you have extensive experience with running enterprise-level ELK then it is highly likely that your team will become overwhelmed at the prospect of maintaining this complex and fast-evolving technology Stack.

Many teams that have tried to host their own deployment of either OpenSearch or the Elastic Stack soon find that they don't have the additional resources required to configure, maintain and manage multiple Elasticsearch instances on top of their existing workload.

Attempting to appropriately scale and configure shards without prior knowledge of Elastic software can be highly complex, this is where a managed solution such as Logit.io comes into its own. With Logit.io you can rapidly deploy ELK Stacks within minutes of registering and launch side by side Open Distro, OpenSearch and ELK Stacks that includes Kibana and Grafana.

If you enjoyed this article then why not find out what Elasticsearch is used for.