Production OpenSearch high-ingestion settings on Logit.io

Most “high ingest” OpenSearch posts tell you to buy bigger hardware and hope. On Hosted OpenSearch the cluster is already managed — Logit.io provisions at least two nodes and runs upgrades — so the first bottleneck is usually index settings, not missing rack awareness. Before you bump a node tier on OpenSearch pricing, tune the knobs that decide whether bulk writes land or pile up: refresh_interval, number_of_shards / number_of_replicas, translog durability, bulk batch size, and the templates that freeze mappings at create time.

This is not a what-is-OpenSearch explainer. It is a production checklist grounded in Logit.io’s live docs: create and connect a cluster, call the REST API, create indexes with settings, and read health from CAT endpoints. No invented TB/day charts. When a number is not measured here, you get the setting name and the operational signal that tells you it is wrong.

Failure modes show up before you scale nodes

High ingest rarely dies as a polite latency graph. It shows up as rejected /_bulk responses, a mapping that grew a thousand dynamic fields overnight, search that feels “stuck” because every one-second refresh is competing with writes, or a stubborn yellow cluster that is actually bad replica math on a two-node footprint.

Provision the stack first if you have not already: create an OpenSearch cluster, then copy endpoint and credentials from Stack Settings → Endpoints (or follow connect to your cluster). Writes and settings changes go through the stack REST host at https://{stack-id}-es.logit.io as described in the OpenSearch REST API docs.

refresh_interval: cut the refresh storm

OpenSearch makes newly indexed documents searchable after a refresh. The default one-second cadence is fine for interactive dashboards on quiet indexes. Under sustained bulk load it becomes a refresh storm: the cluster spends CPU and heap opening new segments while your shipper is still pushing NDJSON.

For a heavy load window, set a slower index.refresh_interval on the write index (or in the template that creates it). Values like 30s or even -1 (manual refresh only) are normal during a backfill; restore a shorter interval when search freshness matters again. You do not need a vendor benchmark to justify this — watch write rejection and node CPU while refresh is aggressive, then widen the interval and watch whether bulk stops failing.

Put the setting on the index or template at create time via the Index & Template APIs. Changing it later is still an index settings update, not a reason to add nodes yet.

Shard and replica math on hosted clusters

Logit.io’s create flow assigns at least two cluster nodes and spreads them for stability. That minimum matters for replica math. Primaries must land somewhere; each replica copy needs a different node. Ask for more replicas than you have data-bearing capacity and you get yellow: primaries active, replicas unassigned. The Cluster & CAT APIs docs spell that out — yellow means replicas unassigned, not “cluster dying.”

Logit’s create-index example sets number_of_shards and number_of_replicas explicitly:

PUT my-custom-logs
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  }
}

Start small on write-heavy indexes. Extra primary shards multiply merge and refresh work; they do not magically multiply ingest throughput on a two-node developer tier. Prefer one primary (or a modest count you can justify with shard size over time) and a replica count that can actually allocate on your node count. Hosted stacks may adjust shard defaults — if create fails, the Index APIs guide says to contact support rather than inventing a self-hosted reroute.

Confirm allocation with GET /_cat/shards/{index}?v and overall status with GET /_cluster/health. Yellow after a replica bump is a settings mistake until proven otherwise.

Translog durability and bulk batch size

Every indexed document hits the translog. index.translog.durability set to request fsyncs on every request (safer, slower under load). async batches fsyncs (faster ingest, larger window of loss if a node dies before flush). For pure telemetry backfills where you can replay the source, async is a common temporary choice. For authoritative application search indexes, stay on request unless you have an explicit recovery story.

Pair durability with sane bulk sizing. Logit’s Document APIs guide is direct: send bulk payloads in batches — for example 1,000–5,000 documents — to avoid timeouts, and prefer Logstash for resilient production pipelines. The same ceiling shows up under managed stack limitations: very large /_bulk bodies time out; smaller chunks are the fix, not a mystery scale-out.

Operational signal: HTTP timeouts or partial bulk errors with "errors": true in the response. Shrink batch size, lengthen refresh, then revisit durability. Only after those stabilize should you shop a larger node size on OpenSearch pricing (published developer tiers from $45.52/node/mo annual on DEV-1-1-10).

Templates before the first write

Dynamic mapping feels convenient until a noisy field cardinality explodes the cluster state. Index templates apply settings and mappings when a matching index is created — they do not rewrite indexes that already exist. Logit documents both legacy /_template and composable /_index_template on stacks; see index templates and the Index & Template APIs page.

Before the first production bulk:

• Define explicit mappings for high-cardinality identifiers as keyword (not analyzed text).
• Disable or strictly limit dynamic mapping on untrusted payloads.
• Bake refresh_interval, shard, replica, and translog choices into the template so every rollover index inherits them.
• Verify with GET /_index_template or GET /_template from Dev Tools or curl using Endpoints credentials.

Mapping explosion shows up as runaway field counts in GET /{index}/_mapping and climbing cluster state size — not as a clean “please add nodes” alert.

ISM, retention, and hosted limits

Self-hosted runbooks love Index State Management (ISM) policies for rollover and delete. On Logit.io, the API overview lists ISM under plugin APIs and notes that retention is typically configured via your Logit.io plan. Snapshots and repository registration are platform-managed; some persistent /_cluster/settings changes may be rejected or overridden per managed stack limitations.

So the production move is: use templates and index settings you control for ingest shape; use plan retention (and support) for lifecycle and recovery; do not assume every upstream ISM example applies unchanged on a hosted stack. You still search, bulk index, create indexes, manage aliases, and read health through the documented APIs.

Signals to watch before you scale

• Rejected or timed-out bulk — shrink batches; check refresh and translog; confirm with Document APIs guidance.
• Refresh storm — high write CPU with frequent refreshes; widen refresh_interval.
• Yellow from replica math — /_cluster/health yellow plus unassigned replicas on /_cat/shards; lower replicas or add capacity that can actually hold them.
• Mapping explosion — template and mapping review before more hardware.
• 403/401 on admin APIs — may be a hosted restriction, not a bad password; see managed stack limitations, then support.

Dedicated Hosted OpenSearch is the right product when OpenSearch itself is the workload (custom indexes, application search, analytics). Shipper-first log pipelines still belong on log management (from $25/mo on the public tables; metrics from $12/mo; APM from $20/mo). Many teams run both — do not duplicate the same stream across two stacks just to “have OpenSearch.”

Get started

Create a Hosted OpenSearch stack, open Endpoints, and create a write index from a template that already sets refresh, shards, replicas, and translog. Bulk in 1,000–5,000 document chunks, watch /_cluster/health and /_cat/indices, and only then decide whether a larger node tier is warranted. Start from the OpenSearch overview, the Hosted OpenSearch overview, and the node table on OpenSearch pricing. Settings first. Nodes second.