What is OpenSearch?
OpenSearch is a distributed, community-driven, Apache 2.0-licensed, 100% open-source search and analytics suite used for a broad set of use cases like real-time application monitoring, log analytics, and website search. OpenSearch provides a highly scalable system for providing fast access and response to large volumes of data with an integrated visualization tool, OpenSearch Dashboards, that makes it easy for users to explore their data. OpenSearch is powered by the Apache Lucene search library, and it supports a number of search and analytics capabilities such as k-nearest neighbors (KNN) search, SQL, Anomaly Detection, Machine Learning Commons, Trace Analytics, full-text search, and more.
Why should I use OpenSearch?
OpenSearch enables you to easily ingest, secure, search, aggregate, view, and analyze data for a number of use cases such as log analytics, application search, enterprise search, and more. With OpenSearch, you benefit from having a 100% open source product you can use, modify, extend, monetize, and resell however you want. There are a growing number of OpenSearch Project partners that offer a variety of services such as professional support, enhanced features, and managed OpenSearch services. The OpenSearch Project continues to provide a secure, high-quality search and analytics suite with a rich roadmap of new and innovative functionality.
Why was OpenSearch created?
Developers embrace open source software for many reasons, one of the most important being the freedom to use that software where and how they wish. On January 21, 2021, Elastic NV announced that they would change their software licensing strategy and not release new versions of Elasticsearch and Kibana under the permissive Apache License, Version 2.0 (ALv2). Instead, Elastic is releasing Elasticsearch and Kibana with source code available under the Elastic License or Server Side Public License (SSPL). These licenses are not open source and do not offer users the same freedoms. Because some developers want their software to be open source and because they want it to avoid single vendor lock-in, we made the decision to create and maintain a fork from the last ALv2 version of Elasticsearch and Kibana. The fork is called OpenSearch and is available under ALv2.
How does OpenSearch relate to Amazon OpenSearch Service?
Amazon OpenSearch Service is an AWS-managed service that lets you run and scale OpenSearch clusters without having to worry about managing, monitoring, and maintaining your infrastructure, or having to build in-depth expertise in operating OpenSearch clusters. We launched support for OpenSearch in Amazon OpenSearch Service in Sep 2021, starting with version 1.0, and renamed the service from Amazon Elasticsearch Service to Amazon OpenSearch Service. Since then, the service has supported several new features that were added to OpenSearch across multiple versions. Some examples are support for cross-cluster replication, trace analytics, data streams, transforms, a new observability user interface, and notebooks in OpenSearch Dashboards. In addition to this, there have been significant improvements to k-NN, anomaly detection, PPL, SQL, and alerting. While Amazon OpenSearch Service continues to support legacy open-source Elasticsearch versions (until 7.10), we strongly recommend that you use OpenSearch on the service to benefit from new features that are being built as part of open-source OpenSearch versions.
Will Amazon OpenSearch Service support new Elasticsearch versions beyond 7.10?
No. Elasticsearch versions beyond 7.10 are not open source and are not released under the permissive ALv2 license. While Amazon OpenSearch Service will continue to support open source Elasticsearch versions until 7.10, moving forward, we will leverage new versions of open source OpenSearch to deliver features and innovations for customers using Amazon OpenSearch Service for their various use cases such as log analytics, search, and observability.
Will OpenSearch maintain compatibility and feature parity with new Elasticsearch versions?
The roadmap for OpenSearch is community-driven, and various organizations including AWS contribute new features to the open source code based. OpenSearch is used for a broad set of use cases like real-time application monitoring, log analytics, and website search. The roadmap for OpenSearch should be viewed as independent to that of Elasticsearch, and the focus for OpenSearch will be to provide new features and innovations that the community and customers ask for. While OpenSearch may include similar features to new features available in Elasticsearch (and vice versa), the implementation of all features are unique between the two projects. The primary goal for OpenSearch will be to build what best suits the needs of the OpenSearch community and our customers.
What are some features that OpenSearch provides?
| Feature |
Benefit |
| Offers encryption, authentication, authorization, and auditing features. They include integrations with Active Directory, LDAP, SAML, Kerberos, JSON web tokens, and more. OpenSearch also provides fine-grained role-based access control to indices, documents and fields. |
|
| Offers a number of features to help you customize your search experience such as Full-text querying, Autocomplete, Scroll Search, customizable scoring and ranking, and more. |
|
| Provides the familiar SQL query syntax. Use aggregations, group by, and where clauses to investigate your data. Read data as JSON documents or CSV tables so you have the flexibility to use the format that works best for you. |
|
| Enables you to use the familiar SQL query syntax while getting access to the rich set of search capabilities such as fuzzy matching, boosting, phrase matching, and more. |
|
| Data Prepper is a server side data collector capable of filtering, enriching, transforming, normalizing and aggregating data for downstream analytics and visualization. Data Prepper lets users build custom pipelines to improve the operational view of applications. |
|
| Trace Analytics provides a way to ingest and visualize OpenTelemetry data in OpenSearch. This data can help you find and fix performance problems in distributed applications. |
|
| Use application analytics to create custom observability applications to view the availability status of your systems, where you can combine log events with trace and metric data into a single view of overall system health. This lets you quickly pivot between logs, traces, and metrics to dig into the source of any issues. |
|
| The Piped Processing Language provides a familiar query syntax with a comprehensive set of commands delimited by pipes (|) to query data. |
|
| Build operational panels to organize Observability visualizations generated using the Piped Processing Language (PPL). |
|
| Use the Piped Processing Language (PPL) queries to interactively build and view different visualizations of your data including trace log correlations. |
|
| Use a range of machine learning algorithms like kmeans and anomaly detection to train models and predict trends in your data. ML Commons integrates directly with PPL and the REST API. |
|
| Schedule, export, and share reports from dashboards, saved searches, alerts and visualizations. |
|
| Leverage Machine Learning anomaly detection based on the Random Cut Forest (RCF) algorithm to automatically detect anomalies as your data is ingested. Combine with Alerting to monitor data in near real time and automatically send alert notifications. |
|
| Define custom policies to automate routine index management tasks, such as rollover and delete, and apply them to indices and index patterns. |
|
| Create a summarized view of your data centered around certain fields, so you can visualize or analyze the data in different ways. For example, suppose that you have airline data that’s scattered across multiple fields and categories, and you want to view a summary of the data that’s organized by airline, quarter, and then price. You can use a transform job to create a new, summarized index that’s organized by those specific categories. |
|
| Pick the fields that interest you and use index rollup to create a new index with only those fields aggregated into coarser time buckets. You can store months or years of historical data at a fraction of the cost with the same query performance. |
|
| Query numerous cluster performance metrics and aggregations. Use PerfTop, the command line interface (CLI) to quickly display and analyze those metrics. Use the root cause analysis (RCA) framework to investigate performance and reliability issues in clusters. |
|
| Run complex queries without worrying about the query timing out and with Asynchronous Search queries run in the background. Track query progress and retrieve partial results as they become available. |
|
| Ingest and visualize OpenTelemetry data for distributed applications. Visualize the flow of events between these applications to identify performance problems. |
|
| Automatically monitor data and send alert notifications automatically to stakeholders. With an intuitive interface and a powerful API, easily set up, manage, and monitor alerts. Craft highly specific alert conditions using OpenSearch’s full query language and scripting capabilities. |
|
| Create alerting policies that alert on grouped trends in your data. For example, you can alerting for each host that has an average CPU above your desired threshold. |
|
| Replicate indexes, mappings, and metadata from one OpenSearch cluster to another in order to create cross cluster redundancy or offload reporting querying to a secondary cluster. |
|
| Using Machine Learning, run the nearest neighbor search algorithm on billions of documents across thousands of dimensions with the same ease as running any regular OpenSearch query. Use aggregations and filter clauses to further refine similarity search operations. k-NN similarity search powers use cases such as product recommendations, fraud detection, image and video search, related document search, and more. |
|
| Combine dashboards, visualizations, text, and more to provide context and detailed explanations when analyzing data. |
|
| OpenSearch supports a range of language clients such as Go, JavaScript, Python, Java, and more. Use these clients to build applications that integrate directly with OpenSearch. |
Who sponsors and maintains OpenSearch?
Many organizations including AWS, SAP, CapitalOne, RedHat, Logz.io, Aiven.io, Bonsai, Logit.io, InstaCluster, and BAInsight have publicly backed OpenSearch.
We encourage and accept contributions from the community and do not require a Contributor License Agreement (CLA) in order to do so. Additionally, if you are an active contributor and would like to take on more responsibility in the project, we have defined a process for non-AWS employees to gain maintainer rights in OpenSearch project repos here.
What is the future of OpenSearch?
We launched the first Generally Available OpenSearch version 1.0 in July 2021, and since then have launched many new versions, detailed here. With great interest and support from the community, we introduced several new features across key use cases around log analytics, search, and observability. These areas continue to be key themes for future improvements for both OpenSearch and our visualization layer OpenSearch Dashboards. Please see the roadmap for a quick overview of the key areas OpenSearch is focusing on.
How is OpenSearch licensed?
All of the software in the OpenSearch project is released under the Apache License, Version 2.0 (ALv2). ALv2 grants well-understood and permissive usage rights that match the freedoms people expect with open source software: freedoms such as being able use, modify, extend, monetize, and resell the open source software where and how they want. For OpenSearch, we believe this license will enable broad adoption and contributions benefiting all members of the community. We have also published permissive usage guidelines for the OpenSearch trademark, so you can use the name to promote your offerings.
Where can I find more information about OpenSearch’s client and API compatibility?
Please see the FAQs on the open-source OpenSearch website. The FAQs are also a great source for information around tools and plugins, upgrade compatibility, the community in general, and how you can contribute and benefit from OpenSearch.
Additional Resources:
- OpenSearch FAQs »
- Blogs: Stepping up for a truly open source Elasticsearch and Introducing OpenSearch »
OpenSearch includes certain Apache-licensed Elasticsearch code from Elasticsearch B.V. and other source code. Elasticsearch B.V. is not the source of that other source code. ELASTICSEARCH is a registered trademark of Elasticsearch B.V.
