Mastering Centralized Logging with OpenSearch

Why Should You Use OpenSearch for Centralized Logging?

OpenSearch is an appropriate choice for centralized logging for numerous reasons. Firstly, OpenSearch is highly scalable as it's designed to manage extensive volumes of log data efficiently, making it suitable for centralized logging in environments with diverse sources and high log volumes. Also, the distributed architecture of OpenSearch enables organizations to scale horizontally by adding additional nodes or clusters as needed, guaranteeing that the logging infrastructure can accommodate growth and fluctuations in log data.

In addition to this OpenSearch provides robust security features, including encryption, authentication, and access controls, to guarantee the confidentiality, integrity, and availability of log data. Also, organizations can enforce fine-grained access controls, encrypt data in transit and at rest, and audit user activity to comply with regulatory requirements and industry standards.

Additionally, OpenSearch's flexible data model supports structured, semi-structured, and unstructured data, this allows it to be capable of ingesting and indexing a wide variety of log formats and sources. Whether logs are in text-based formats like JSON, CSV, or plain text, or structured formats like syslog or Apache logs, OpenSearch can efficiently index and analyze the data.

Lastly, OpenSearch supplies powerful querying and analysis capabilities, enabling users to execute complex searches, aggregations, and visualizations on log data. With features such as full-text search, field-level querying, and aggregations, organizations can attain insights into system performance, highlight trends, and troubleshoot issues more effectively.

Centralized Logging with OpenSearch

Below is a detailed configuration guide outlining how to set up OpenSearch for centralized logging. This configuration guide uses Filebeat as a log shipping tool.

Install and Configure Filebeat

• Install Filebeat: Download and install Filebeat on the systems where log files are generated.
• Configure Filebeat:

filebeat.inputs:

• type: log enabled: true paths:
  • /var/log/*.log # Path to log files on the local system (replace with your actual log file paths)
  fields_under_root: true fields: environment: production # Custom field to identify the environment (replace with your environment name)

output.elasticsearch: hosts: ["your-opensearch-host:9200"] # OpenSearch host and port (replace with your OpenSearch host) username: "your-opensearch-username" # OpenSearch username for authentication password: "your-opensearch-password" # OpenSearch password for authentication index: "logs-%{+yyyy.MM.dd}" # Index pattern for log entries with daily indices

Configure OpenSearch

• Define Index Template: Create an index template in OpenSearch to specify the mapping for log data.

  { "index_patterns": ["logs-*"],
  "mappings": { "properties": { "@timestamp": { "type": "date" },
  "message": { "type": "text" },
  "level": { "type": "keyword" },
  "source": { "type": "keyword" },
  "environment": { "type": "keyword" }

}

This JSON template defines the mapping of fields for log entries in OpenSearch. It specifies the data types and properties of each field, such as the timestamp, log message, log level, source, and environment.

• Configure Index Lifecycle Management (ILM): Define an ILM policy to manage the lifecycle of OpenSearch indices.

{ "policy": { "phases": { "hot": { "actions": { "rollover": { "max_size": "50gb",
"max_age": "1d"
} } }, "delete": { "min_age": "30d",
"actions": { "delete": {}
} } } } }

This JSON policy outlines the lifecycle phases for OpenSearch indices built by Filebeat. It specifies actions to be taken during the "hot" phase (e.g., index rollover based on size and age) and the "delete" phase (e.g., index deletion after a certain period).

Apply Index Template and ILM Policy

• Apply Index Template and ILM policy to OpenSearch using the appropriate API calls or management tools.

OpenSearch: Advanced Features

Among all the major features of OpenSearch in enhancing search capabilities, scalability, security, and making the operations efficient is a distributed search architecture. It allows the user to scale horizontally and distribute the search queries across different nodes while making the queries parallel. This distributed approach ensures high availability and fault tolerance since many nodes can concurrently process search requests.

In addition, advanced search and analytics are part of the advanced functionality of OpenSearch, which includes aggregations, faceted search, and geospatial queries. Aggregations enable analysis of metrics, statistics, and summaries on large datasets, hence providing advanced data exploration and analysis. The user can dynamically filter and drill down into search results with facets pre-defined by categories or attributes, allowing exploratory and interactive search experiences.

Lastly, an advanced feature of OpenSearch is visualization and dashboards. Incorporating dashboards into your OpenSearch logging configuration offers access to advanced log data visualization tools. These dashboards let you build dynamic displays that can give real-time insights into log data. Leverage the wealth of visualization options provided by dashboards, including line charts, pie charts, and more, for monitoring key log metrics and finding trends in the operations of your application.

OpenSearch: Use Cases

OpenSearch is a powerful search, querying, and analytics solution that effectively lends itself to a variety of use cases. An example of one of these use cases is anomaly detection. The tool's capabilities allow it to find deviations from normal patterns within very large datasets. OpenSearch supports real-time indexing and search, thus allowing the continuous processing of streaming data from log files, metrics, or events with the rapid identification of any anomalies or outliers.

OpenSearch can leverage statistical analysis, machine learning techniques, or custom rules for the detection of anomalous behavior indicative of security breaches, system failures, or performance fluctuations. The insights enable proactive intervention to mitigate risks, enhance security posture, and optimize operational efficiency.

Another common use case for OpenSearch is deriving metrics from traces as it puts to good use its capability for deriving effective insights from distributed tracing data. The ability of OpenSearch to ingest, index, and query large volumes of trace data allows an organization to analyze performance and behavior in distributed systems comprehensively.

OpenSearch allows the creation of meaningful metrics for visibility into system health and efficiency by aggregating trace data and extracting key performance indicators, such as latency, error rates, and throughput. Leaning on the flexible query capability in OpenSearch, organizations can tie trace data to other telemetry sources, like logs and metrics, for deeper insight into the root cause of performance issues and bottlenecks.

Lastly, log enrichment is another great use case for OpenSearch over competitors like Datadog, using flexible search and analytics to augment log data with additional context and insight. OpenSearch is able to ingest and index many different sources of data, enabling organizations to enrich log entries with relevant metadata, such as user attributes, geographic information, or application-specific context, hence enriching log data in terms of richness and utility for analysis and troubleshooting.

Additionally, OpenSearch is open source and extendable, so organizations can independently customize and extend log enrichment capabilities according to their unique needs, providing more flexibility and greater control over the log enrichment process compared with proprietary solutions like Datadog.

Enhanced Centralized Logging with Hosted OpenSearch

OpenSearch offers numerous benefits for conducting centralized logging but the time-consuming configuration and maintenance of the tool can be a deterrent to many users. With Hosted OpenSearch this challenging and tedious process is completely removed and carried out by the service provider, enabling you to begin utilizing OpenSearch for centralized logging almost immediately.

By opting for a Hosted OpenSearch from Logit.io’s solution, users can benefit from easily scalable logs management. Hosted OpenSearch solutions offer a managed environment where the infrastructure, maintenance, and scaling tasks are handled by the provider.

Lastly, Hosted OpenSearch solutions offer highly available infrastructure with built-in redundancy and failover capabilities. This guarantees that an organization's logging infrastructure can manage increasing log volumes, accommodate peak loads, and maintain uptime and availability, even during hardware failures or maintenance events.

Hosted OpenSearch from Logit.io

Logit.io offers a cost-effective and powerful Hosted OpenSearch solution, that provides users with the best features of OpenSearch without the time-consuming configuration and maintenance. With our solution, users can easily integrate the tool with existing cloud services, databases, or third-party applications.

As well as this, Logit.io’s Hosted OpenSearch offers node-to-node encryption and advanced RBAC (role-based access controls) to provide a complete data visualization, alerting, and monitoring platform that is suitable for any cybersecurity or data centralization use case.

If you’re interested in finding out more about the benefits of Hosted OpenSearch or the extensive capabilities of the Logit.io observability platform, feel free to contact us, or begin exploring the platform for yourself with a free 14-day trial.

If you've enjoyed this article why not read OpenSearch Dashboards vs Kibana or The Top 10 OpenSearch Plugins next?