Logs in Discover are useful for investigation. An alert is useful when the same pattern should wake someone up. This post assumes you already ship logs to a Logit.io Logs stack and walks the customer journey to one actionable rule — when to alert, a concrete error-spike / status-500 example, and where notifications go — with YAML details left in the docs.
Contents
When an alert is worth it
Alert on conditions a human should act on within minutes: sudden error volume, a burst of HTTP 500s, or a flatline where heartbeat logs disappear. Skip alerts for noisy debug chatter or one-off expected errors. A good first rule is narrow, named clearly, and tied to a destination your on-call already watches.
Logit.io runs ElastAlert 2 for you on managed infrastructure. You enable alerting on the stack, edit rule YAML in the dashboard, use Test only, then Update when ready. The product overview is in Alerting overview.
Enable alerting on the stack
Open your Logs stack → Alerting & Notifications → enable alerting. When the service is ready, you will see sample rules you can edit, or create a New Rule. Step-by-step UI guidance is in How do I create a new alerting rule?.
You do not put OpenSearch cluster endpoints inside the rule file — managed stacks query indexes you already use in Discover (often a pattern like *-*). Match the index pattern to documents you can already find by hand.
Concrete example: error spike or 500s
A practical first rule is a frequency alert: at least N matching events inside a timeframe. For example, 50 events in 15 minutes where the query matches error-level logs (log.level:error OR level:error), or status 500 fields your apps already emit. The overview docs include a managed-stack frequency example with type: frequency, num_events, timeframe, a query_string filter, and an alert destination.
Prefer spike when you care about a sudden jump versus a baseline window, or keep frequency when you simply want a hard count threshold. Browse types under the alerting rule-types guides linked from the overview — copy a full example from docs rather than inventing keys.
Before you page anyone, run Test only. If you see zero hits, simplify the filter and confirm the index pattern against Discover. If you see hits but no notification, check destination config, realert, and silence behaviour described in the overview FAQ.
Unlock complete visibility with hosted ELK, Grafana, and Prometheus-backed Observability
Destinations that page someone
Set alert: to one or more channels — email, Slack, PagerDuty, Microsoft Teams, webhooks (post / post2), and many others. Each destination has its own keys; use the Destinations index for the channel you need. Start with a destination you can verify immediately (email or Slack), then add on-call routing once the rule fires correctly.
Subjects, bodies, top counts, and Discover links are covered under Subject & body and Context & links in the same alerting section — useful once the basic page works.
Keep noise down
Use query_key to bucket by host or service, raise realert so the same condition does not spam, and reserve any (alert on every match) for rare, high-signal events. Rule execution logs land in your stack’s elastalert index so you can audit tests and failures alongside application logs.
Example shape without a full runbook
Conceptually, a first useful rule names the condition (“API 500 flood”), sets type: frequency (or spike), points index at the pattern that works in Discover, filters on status or level fields your apps already emit, and lists one destination under alert:. Copy the full YAML from the overview example or the frequency rule-type guide, then adapt the query string to fields you verified in Discover — do not invent field names.
After Update succeeds, force a safe test (generate synthetic errors in a non-production index or temporarily lower thresholds) and confirm the page arrives. Then restore production thresholds and realert so the rule stays useful on-call instead of chatty.
When you outgrow a single rule, add per-service query_key variants and separate destinations for warning vs page. Automate enable/disable through the Developer API only after the manual rule path is trusted.
Optional: OpenSearch Alerting
If your team prefers monitors inside OpenSearch Dashboards instead of YAML rules, the overview links to OpenSearch Alerting as an alternative. For most first alerts on a Logs stack, ElastAlert 2 YAML is the primary path documented for Logit.io.