Logit.io
OpenTelemetry Linux host logs: errors by host and service
โ† Back to blog
11/21/2026 ยท 27 min read

OpenTelemetry Linux host logs: errors by host and service

Logit.io Team
Logit.io Team
Technical Content Team

Last updated 11/21/2026

Log ManagementHow To GuidesOpensearch

The Linux hosts pack in the Logit.io Dashboard Library installs five OpenSearch Dashboards for host logs sent with the OpenTelemetry Collector: an overview, errors, hosts, services and a single-host investigation view. They read logstash-* and group everything by otel.resource.host.name, otel.resource.service.name and severity, so the counts are only right when the Collector sends a normalised severity text and a useful service name. A default file-tailing config sends neither, which is why the error panels on a fresh install often sit at zero while the volume charts look busy.

This guide is for teams who already ship, or are about to ship, Linux server logs to a Logit.io Logs stack through Hosted OpenTelemetry and want the dashboards to answer real questions: which host started throwing errors, which service is behind them and whether a quiet host is healthy or has stopped sending. It covers what each dashboard reads, what one log record turns into on your stack, a Collector config that fills every panel (checked against otelcol-contrib 0.161.0), how to prove each severity arrives before you install, an incident walk-through, saved searches, two alert rules and the problems you'll hit on real fleets.

Contents

Five dashboards, one scope clause

The pack is listed as Linux hosts in the Dashboard Library, under the OS category. Every panel in it is a visualisation over the stack's existing logstash-* index pattern with @timestamp as the time field, and every panel starts from the same filter:

_exists_:otel.resource.host.name AND NOT _exists_:otel.resource.k8s.pod.name

The first half keeps documents that came from a Collector which set a host name. The second half throws out Kubernetes pod logs, which also carry a host name but belong in the Kubernetes packs. Each dashboard repeats the clause as a pair of filters, so you can see it in the filter bar and you'll notice if someone removes it.

DashboardQuestion it answersWhat's on it
Linux Host SignalsWhich hosts are talking, and which are complaining?Message flow per host over time (top 10), error and fatal lines by host, and a host by severity table. Opens on whatever time range you already have selected.
Linux host errorsHow many errors, on how many hosts, and since when?Tiles for error lines, warning lines, hosts with errors and the newest error, an error histogram, error pies by host and by service, and a breakdown table by host, service and severity.
Linux hostsWhich hosts are sending, how much and at what severity?Total host lines, distinct hosts, error lines and Last event received, a volume histogram, top hosts, a severity pie and a host activity table.
Linux host servicesWhich services are noisiest, and which are warning?Distinct services, error and warning counts, a volume histogram, top services by lines and by warnings, and a service activity table.
Linux host investigationWhat did this one host or service do?The same tiles and charts with nothing narrowed down, built for you to add a host or service filter and read the timeline table.

The four drill-down dashboards open on the last 24 hours and refresh every 60 seconds. Errors mean severity:(ERROR OR FATAL) and warnings mean severity:(WARN OR WARNING). The pies show the top five values and the tables the top ten, which is plenty for a team with a dozen servers and a starting point for a fleet of hundreds, where you'll filter first and read second.

No panel shows message text. That's deliberate: logstash-* is a wide pattern on most stacks, and a saved search across it can ask for more docvalue fields than an index allows. The dashboards tell you where to look. Discover, with the same filter, shows you the lines.

What one log record becomes on logstash-*

Hosted OpenTelemetry takes OTLP over HTTPS on port 443 at https://{stackId}-otel.logit.io. It's switched on by default for new Logs stacks; on an existing stack, select Switch on in the Hosted OpenTelemetry card on the stack's Endpoints page. Your stack's Logstash splits each batch into one document per log record and writes it to the daily logstash-YYYY.MM.DD index, alongside your other non-Beats logs. The mapping from the OpenTelemetry log model is simple, and worth knowing before you change anything on the Collector side:

OpenTelemetry log recordField in OpenSearchUsed by the pack
Body (string)messageNo, read it in Discover
Body (map or list)message as a JSON stringNo
Severity textseverity, only when the text isn't emptyYes, every error and warning panel
Severity numberotel.severity_numberNo, useful for your own queries
Timestamp, or observed time when the timestamp is unset@timestampYes, every panel
Resource attributesotel.resource.*, keys kept with their dots, so host.name becomes otel.resource.host.nameYes, host and service grouping
Resource service.nameAlso copied to service.nameHandy for searching
Log record attributesotel.attributes.*No

Index templates are managed on Logit.io stacks, and string fields arrive as text with a .keyword sub-field. The pack's terms aggregations use the keyword sub-fields (otel.resource.host.name.keyword, otel.resource.service.name.keyword, severity.keyword), while the filters use the analysed base fields. Every document from this path is also tagged otlp_json, which is a quick way to separate it from other things in logstash-*.

Here's what an nginx failure from the journal looks like once it has landed. The values are illustrative, but the shape is what the config later in this guide produces:

{
  "@timestamp": "2026-11-21T09:14:03.000Z",
  "message": "nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)",
  "severity": "ERROR",
  "service": { "name": "nginx" },
  "otel": {
    "severity_number": 17,
    "resource": {
      "host.name": "web-01",
      "os.type": "linux",
      "service.name": "nginx"
    },
    "attributes": {}
  },
  "tags": ["otlp_json"]
}

Two lines in that document do all the work for the dashboards: "severity": "ERROR" puts it on the error panels, and "service.name": "nginx" under otel.resource puts it in the right slice of every service pie. Neither appears by magic.

A Linux host writes to the systemd journal and its application log files. The OpenTelemetry Collector reads both, sets severity, service name and host name, and exports OTLP over HTTPS to Hosted OpenTelemetry on your Logs stack. Logstash turns each log record into one document in the daily logstash indices, where the five Linux hosts dashboards and your alert rules read them

Why the error panels stay at zero on a default config

The quickest way to get host logs flowing is the Servers & log files config from onboarding, also in the guide to sending server log files, which tails the classic syslog files:

receivers:
  filelog:
    include: ["/var/log/syslog", "/var/log/messages"]
    start_at: end

That's the right first step. Lines arrive within seconds and the volume panels fill. But a plain file tail reads each line as an opaque string. Nothing in Nov 21 09:14:03 web-01 nginx[812]: bind() to 0.0.0.0:443 failed says "error" in a way the receiver understands, and the traditional syslog file format doesn't record the priority at all. The record goes out with no severity text, the document has no severity field, and every error and warning tile reads zero. The service pies show one slice per host, because the only service.name is the single value from the resource/logit processor.

Switching to a receiver that does know the priority only gets you halfway. We replayed the same lines through otelcol-contrib 0.161.0, with journal entries in journalctl's JSON format standing in for the journald receiver, with and without a normalising step, and the severity text the Collector sends looks like this:

Source line                                    severity_text   severity_text
                                               (no transform)  (with transform)
journald PRIORITY=3  nginx bind() failed       3               ERROR
journald PRIORITY=4  timesyncd timed out       4               WARN
journald PRIORITY=3  kernel OOM kill           3               ERROR
syslog <10> crit     EXT4-fs error             crit            ERROR
syslog <11> err      CRON command failed       err             ERROR
syslog <12> warning  sshd connection reset     warning         WARN
app file "ERROR"     payment batch failed      ERROR           ERROR

The journal's priority comes through as a digit unless you map it, and the syslog parser uses the syslog names err, crit and warning. The pack's filters are analysed, so case doesn't matter, but err isn't error and 3 isn't anything. The severity pie would also split ERROR and error into two slices, because it groups on the exact keyword. The fix is the same whatever the source: parse a severity number wherever the line has one, then let one processor set the text from the number.

A Collector config that fills every panel

This replaces /etc/otelcol-contrib/config.yaml on a systemd host. It reads the journal rather than /var/log/syslog, so it works the same on hosts with and without rsyslog, adds one application log file as an example, and keeps the extension and exporter from onboarding. It needs the contrib build; 0.161.0 is the version onboarding installs. We checked the operators and processors on 0.161.0 by replaying journal entries in journalctl's JSON format through them, so run the logger test below on one host before rolling it out.

extensions:
  basicauth/logit:
    client_auth:
      username: "{stackId}-otel"
      password: "${env:LOGIT_API_KEY}"
  file_storage:
    directory: /var/lib/otelcol/file_storage

receivers:
  journald:
    directory: /var/log/journal
    priority: info
    storage: file_storage
    operators:
      - type: severity_parser
        parse_from: body.PRIORITY
        mapping:
          fatal: "0"
          error: ["1", "2", "3"]
          warn: "4"
          info: ["5", "6"]
          debug: "7"
      - type: move
        if: 'body.SYSLOG_IDENTIFIER != nil'
        from: body.SYSLOG_IDENTIFIER
        to: resource["service.name"]
      - type: move
        if: 'resource["service.name"] == nil and body._SYSTEMD_UNIT != nil'
        from: body._SYSTEMD_UNIT
        to: resource["service.name"]
      - type: move
        from: body.MESSAGE
        to: body

  file_log/billing:
    include: [/var/log/billing-worker/*.log]
    start_at: end
    storage: file_storage
    multiline:
      line_start_pattern: '^\d{4}-\d{2}-\d{2}T'
    operators:
      - type: regex_parser
        regex: '^(?P<time>\S+)\s+(?P<level>[A-Za-z]+)\s'
        timestamp:
          parse_from: attributes.time
          layout_type: gotime
          layout: '2006-01-02T15:04:05Z07:00'
        severity:
          parse_from: attributes.level
      - type: remove
        field: attributes.time
      - type: add
        field: resource["service.name"]
        value: billing-worker

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  resource_detection:
    detectors: [system]
    system:
      hostname_sources: [os]
  transform/severity:
    error_mode: ignore
    log_statements:
      - set(log.severity_text, "FATAL") where log.severity_number >= SEVERITY_NUMBER_FATAL
      - set(log.severity_text, "ERROR") where log.severity_number >= SEVERITY_NUMBER_ERROR and log.severity_number < SEVERITY_NUMBER_FATAL
      - set(log.severity_text, "WARN") where log.severity_number >= SEVERITY_NUMBER_WARN and log.severity_number < SEVERITY_NUMBER_ERROR
      - set(log.severity_text, "INFO") where log.severity_number >= SEVERITY_NUMBER_INFO and log.severity_number < SEVERITY_NUMBER_WARN
      - set(log.severity_text, "DEBUG") where log.severity_number >= SEVERITY_NUMBER_TRACE and log.severity_number < SEVERITY_NUMBER_INFO
  resource/logit:
    attributes:
      - { key: service.name, value: "web-host", action: insert }
  batch: {}

exporters:
  otlp_http/logit:
    endpoint: "https://{stackId}-otel.logit.io"
    auth: { authenticator: basicauth/logit }

service:
  extensions: [basicauth/logit, file_storage]
  pipelines:
    logs:
      receivers: [journald, file_log/billing]
      processors: [memory_limiter, resource_detection, transform/severity, resource/logit, batch]
      exporters: [otlp_http/logit]

What each part does, in the order a record meets it:

  • journald reads entries through journalctl, starting at the end of the journal. priority: info drops debug entries, which on a busy host can be most of the volume. The receiver hands each entry over as a map of journal fields, such as PRIORITY, SYSLOG_IDENTIFIER, _SYSTEMD_UNIT and MESSAGE.
  • The severity_parser operator maps the journal's 0 to 7 priority to OpenTelemetry levels: 0 is fatal, 1 to 3 are errors, 4 is a warning, 5 and 6 are info and 7 is debug. That's the same split the syslog parser uses, so the two agree if you ever mix them.
  • The two move operators set the resource service.name. The syslog identifier comes first because it's the name people recognise: sshd, nginx, CRON, kernel. Entries without one fall back to the systemd unit, such as backup.service. The second move checks that the resource name is still empty, because the first move has already removed the identifier from the map by the time it runs.
  • The last move makes MESSAGE the whole body. Without it the body stays a map, and the stack stores it as a JSON string in message, which is hard to read and harder to search. The other journal fields are dropped at this point. If you want one of them, move it into attributes first and it will land under otel.attributes.
  • file_log/billing is the pattern for an application that writes its own file. The multiline setting starts a new record only on a line that begins with a date, so a stack trace stays attached to the error above it instead of arriving as a dozen records with no severity. The regex picks out the time and level, the timestamp parser makes @timestamp the time the application wrote, and the add operator names the service. Repeat the block once per application, with its own name and its own service.name.
  • memory_limiter stops the Collector growing without bound if the exporter falls behind. It comes first in the pipeline.
  • resource_detection adds host.name and os.type. hostname_sources: [os] uses the kernel's host name. The default tries DNS first, which gives a fully qualified name on some hosts and a short one on others, and that turns one server into two rows on the hosts dashboard.
  • transform/severity is the line that fills the error panels. Every source has a severity number by now, and these five statements set the text to FATAL, ERROR, WARN, INFO or DEBUG from it, whatever the receiver put there. Records with no number at all keep whatever text they had, usually none.
  • resource/logit is the onboarding fallback. It only inserts web-host when nothing earlier has set a service name. Change web-host to the host's role.
  • file_storage remembers the journal cursor and file offsets across restarts. Without it, a restart starts from the end again and anything written while the Collector was down is skipped.

The component names follow 0.161.0. Older configs, including the ones on onboarding and in our earlier guides, use filelog, resourcedetection and otlphttp. Those still load, but 0.161.0 logs a deprecation warning for each, and the old names will eventually be removed. You can rename them when you next touch the file.

If the host still runs rsyslog and you'd rather forward to the Collector than read the journal, the syslog receiver works too. It sets the severity number from the priority, and the same transform/severity processor turns err and crit into ERROR. Don't run both on one host: rsyslog copies the journal into /var/log/syslog, so every line would arrive twice.

Roll it out and restart

The packaged service runs as the otelcol-contrib user, which can't read the journal or create a state directory on its own. Three commands fix that, one check finds the journal and the validate step catches typos before systemd does:

# Let the Collector read the journal and keep its read positions
sudo usermod -aG systemd-journal otelcol-contrib
sudo install -d -o otelcol-contrib -g otelcol-contrib -m 0750 /var/lib/otelcol/file_storage

# Check which journal directory this host actually writes
ls -d /var/log/journal /run/log/journal 2>/dev/null

# Check the config before the service tries to load it
sudo -u otelcol-contrib env LOGIT_API_KEY=check otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml

sudo systemctl restart otelcol-contrib
sudo journalctl -u otelcol-contrib -n 30 --no-pager

If /var/log/journal doesn't exist, the journal on that host is volatile and lives under /run/log/journal; change directory to match. The validate command reads the same file the service will. If the state directory is missing it fails with directory must exist, which is better there than in a restart loop.

Keep the API key in the service's environment file rather than the config, as the guide to installing the OpenTelemetry Collector on Linux and Windows servers shows. In the restart output, a clean start ends with Everything is ready. Begin running and processing data. and no exporter errors after it. A 401 means the username or key is wrong; permission denied from journalctl means the group change hasn't reached the service. Group membership is read when the process starts, so a restart after usermod is enough, but a restart before it isn't.

Send one line per severity and find each one

Before installing anything, prove that each severity arrives with the right text. logger writes straight to the journal with the priority you choose:

logger -t otel-test -p user.info    "otel-test info line"
logger -t otel-test -p user.warning "otel-test warning line"
logger -t otel-test -p user.err     "otel-test error line"
logger -t otel-test -p user.crit    "otel-test critical line"

Then open Discover on your Logs stack, pick the logstash-* index pattern and search for the test lines. Use your own host name:

service.name:"otel-test" AND otel.resource.host.name:"web-01"

You should see four documents. Add severity, otel.severity_number and otel.resource.service.name as columns and check them against what you sent:

Test lineseverityotel.severity_number
info lineINFO9
warning lineWARN13
error lineERROR17
critical lineERROR17

The critical line counts as an error because the severity parser maps journal priorities 1 to 3 to the same level. If it says 2 instead, the transform processor isn't in the pipeline. If none of the four appear, check the time picker first, then the Collector's own log. Once the test lines look right, run the search that finds anything the pack can't classify:

_exists_:otel.resource.host.name AND NOT _exists_:otel.resource.k8s.pod.name
  AND NOT severity:(FATAL OR ERROR OR WARN OR WARNING OR INFO OR DEBUG)

On a host running the config above this should return nothing. On a host still running the default file tail, it returns everything, which is a good way to find the servers you haven't migrated yet.

Install the Linux hosts pack

Open your Logs stack's card in your Logit.io account and choose Dashboard Library. Search for Linux hosts. The Library checks your stack for the fields a pack needs, looking back over the last 24 hours, and shows a Data detected badge when otel.resource.host.name is there. If you only just started the Collector, press Rescan stack. If the same stack also receives Kubernetes pod logs through OpenTelemetry, the Kubernetes pack shows Data detected too. The two don't overlap, because Linux hosts leaves out any record with a pod name, and you can tick either pack by hand whether or not it's detected. Use Select all for the five dashboards, choose Install dashboards and open them in OpenSearch Dashboards a minute or two later.

The install adds the dashboards and their visualisations alongside anything you've built yourself, and reuses the existing logstash-* index pattern. The Dashboard Library guide covers permissions and what gets installed, and the post on how Dashboard Library detection works explains the badge.

A sensible way to use the five once they're in:

  1. Start of the day, Linux hosts. Hosts (count) should match the number of servers you think are shipping. Last event received should be minutes old. One host missing from the top-hosts pie is easy to overlook, so compare the count rather than the slices.
  2. When the error count moves, Linux host errors. Hosts with errors tells you whether it's one machine or a pattern. The by-service pie tells you whether it's one daemon or several.
  3. Weekly, Linux host services. The warnings-by-service pie is where slow-burning problems show up first: certificate renewals, NTP timeouts, a disk that keeps logging retries.
  4. During an incident, Linux host investigation. Filter to one host, then read the timeline table to see which services were active and when they last logged.

Worked example: a backup that fails every night

Here's how the pack plays out on a small fleet. The scenario is illustrative, but every query and field in it is real.

On Monday morning the Linux host errors dashboard shows 46 error lines over the last 24 hours against a usual handful, and Hosts with errors reads 1. The histogram has a single tall bar just after 02:00 rather than a steady drip, so something ran and failed rather than something being broken all day. The by-host pie names db-02 and the by-service pie says kernel.

On its own a kernel error at 02:00 could be anything. Linux host services adds the context: the warnings-by-service pie has a new systemd slice, and the volume histogram has a small bump a minute after the kernel errors. Something systemd was running failed straight after the kernel complained.

Open Linux host investigation, add a filter for otel.resource.host.name is db-02 and set the range to 01:50 to 02:20. The timeline table has three rows that matter: kernel ERROR, last seen at 02:03, systemd WARN with a count of one at 02:04, and backup.sh INFO with a few dozen lines in between. To read the lines, open Discover with the same filter and narrow to the kernel:

_exists_:otel.resource.host.name AND service.name:kernel
  AND message:("Out of memory" OR "oom-kill" OR "I/O error" OR "EXT4-fs error")

That returns EXT4-fs error (device sdb1) lines from 02:03. Then the unit:

_exists_:otel.resource.host.name AND service.name:systemd
  AND message:("Failed with result" OR "Main process exited")

The answer is backup.service: Main process exited, code=exited, status=23/n/a at notice level, then backup.service: Failed with result 'exit-code' at warning level, which is the WARN row. The backup.sh lines say rsync error: some files/attrs were not transferred (see previous errors) (code 23), and they're INFO. That's not a config mistake: systemd records whatever a service prints on standard output or standard error at info priority, unless the line starts with a priority prefix such as <3> or the script logs through logger -p user.err. Rsync's code 23 is a partial transfer, which fits a filesystem error on the target. The fix is a disk, not the job.

Two lessons come out of it. Without the normalising processor the kernel lines would have carried a bare priority digit, 2 or 3, as their text and the night would have shown zero errors. And scripts that run under systemd report their own failures as info unless you make them say otherwise, so the systemd warning and the unit-failure search below are what catch them.

Searches worth saving next to the dashboards

The dashboards answer "where" and "how many". These saved searches answer "what", and they're worth saving in Discover under names your team will recognise.

Errors that aren't the usual noise. The kernel and systemd are chatty about things that rarely need action on a healthy host. Excluding them leaves application and daemon errors, which is usually what people mean by "are there errors?":

_exists_:otel.resource.host.name AND NOT _exists_:otel.resource.k8s.pod.name
  AND severity:(ERROR OR FATAL) AND NOT service.name:(kernel OR systemd)

Kernel trouble. The OOM killer and filesystem errors are the kernel messages that most often explain an application failure elsewhere. The query from the worked example is a good one to keep.

Units that failed. The systemd query from the worked example finds every service that exited badly, including timers and one-shot jobs whose own output arrives as info or not at all.

Errors by host and service, as a table you can export. For a weekly review, the SQL and PPL Query Workbench in OpenSearch Dashboards gives you a sorted summary that's easier to paste into a ticket than a pie chart. The guide to querying with the Query Workbench covers where it is:

source = logstash-*
| where isnotnull(`otel.resource.host.name`) and `otel.severity_number` >= 17
| stats count() as errors by `otel.resource.host.name`, `service.name`
| sort - errors
| head 20

Backticks are needed because the field names contain dots. If logstash-* on your stack holds fields with conflicting types from other sources, PPL may refuse the wildcard; point it at a single day's index, such as logstash-2026.11.21, instead.

Alerts for error bursts and silent hosts

Logs stacks include alerting with ElastAlert 2 rules, which you enable from Alerting & Notifications in your stack settings; the guide to creating alerting rules walks through it. Two rules cover most of what the dashboards would otherwise need someone watching for.

The first uses the frequency rule type to catch a burst of errors on any single host:

name: Linux host error burst
type: frequency
index: "logstash-*"
num_events: 25
timeframe:
  minutes: 10
query_key: "otel.resource.host.name.keyword"
realert:
  minutes: 30
filter:
  - query:
      query_string:
        query: "_exists_:otel.resource.host.name AND NOT _exists_:otel.resource.k8s.pod.name AND severity:(ERROR OR FATAL)"
alert:
  - "email"
email:
  - "[email protected]"

query_key makes the count per host, so a noisy fleet doesn't hide one bad server, and realert stops it paging every ten minutes for the same burst. Set num_events from your own baseline: look at the error histogram for a normal week, take the busiest ten minutes and go comfortably above it.

The second uses the flatline rule type to catch a host that stops sending anything at all, which the dashboards can only show you if you happen to look:

name: Linux host stopped sending logs
type: flatline
index: "logstash-*"
threshold: 1
timeframe:
  minutes: 30
query_key: "otel.resource.host.name.keyword"
forget_keys: false
filter:
  - query:
      query_string:
        query: "_exists_:otel.resource.host.name AND NOT _exists_:otel.resource.k8s.pod.name"
alert:
  - "email"
email:
  - "[email protected]"

A healthy systemd host logs something every few minutes, so 30 minutes of silence usually means the Collector stopped, the network path broke or the host is down. The rule tracks each host it has seen, and forget_keys: false keeps alerting about a host until it comes back rather than dropping it after the first alert. Run both rules with Test only before enabling them. Stopping the Collector on a test server for 30 minutes is a safe way to watch the flatline rule fire.

When host panels stay empty or look wrong

What you seeLikely causeWhat to do
Volume panels busy, error and warning tiles at zeroNo severity text, or unnormalised text such as 3 or errAdd the severity parser and transform/severity, then run the unclassified search above
Every dashboard emptyNo otel.resource.host.name on the documents, often because resource_detection isn't in the logs pipelineSearch tags:otlp_json and check the resource fields on one document
Service pies have one slice per hostOnly the resource/logit fallback sets service.nameAdd the identifier and unit moves, or an add operator per application file
One server appears twice under different namesHost name resolved through DNS on some runs and the kernel on othersSet hostname_sources: [os]
Several servers merged into one rowCloned images that all report the same host name, often localhostFix the host name, or set host.name explicitly in a resource processor per host
Every line arrives twiceJournal and /var/log/syslog both collected on a host running rsyslogCollect one or the other
Messages look like {"MESSAGE":"...","PRIORITY":"6",...}Journal body sent as a mapAdd the final move from body.MESSAGE to body
Stack traces show up as many short records with no severityNo multiline setting on the application fileSet line_start_pattern to match the first line of each record
Kubernetes nodes show up in the Linux hosts packNode journal collected without pod metadata, which is correctLeave it, or filter by host name if you only want VMs
Some records missing after a restartNo file_storage, so the Collector restarted from the endAdd the storage extension and a writable directory
A few documents never arrive while the rest doA field changes type between sources, for example an attribute that is a number on one host and text on another, and the index rejects itRename the attribute on one side; if the dead letter queue is enabled on your stack, rejected events are kept in the Logstash dead letter queue, and the post on fixing field type conflicts explains the error

Trade-offs of collecting host logs this way

The journal gives you priority for free; files usually don't. Daemons that log through syslog or straight to the journal get a correct severity with no parsing. Applications that write their own files need a regex per format, and a format change in a new release quietly turns their errors back into unclassified lines. The unclassified search is the cheapest way to notice.

A service name per identifier is a choice. Using the syslog identifier makes the service dashboards readable, but it mixes daemons and applications in one list and it means service.name on a host isn't the same kind of thing as service.name from an instrumented application. If you also send traces, keep application files on the application's real service name so logs and traces line up.

Severity is only as good as the source. Plenty of software logs real failures at info, including anything a systemd service prints to standard error, and some logs routine events at error. The pack counts what the line says. Exclusions in saved searches, like the kernel and systemd example, are how you tune that without changing the dashboards.

The pack reads logs only. It can show that a disk is failing because the kernel says so, but not that a disk is 95 percent full. Host metrics from the Collector's hostmetrics receiver go to a Metrics stack and are read in Grafana; the guide to a first Grafana dashboard is a good place to start with them.

The Collector buffers in memory by default. If the exporter can't reach the endpoint for long, the sending queue fills and new batches are dropped until the endpoint is reachable again. For hosts where losing an hour of logs during a network outage matters, the exporter's sending queue can be backed by the same file_storage extension. That's a Collector-side setting; it doesn't change anything on the stack.

Top 10 is a starting view. On a fleet of hundreds of hosts, the tables and pies show the loudest few. Filter by a naming convention or an environment attribute first, or save copies of the dashboards with a filter pinned per environment.

Start Free Trial

Unlock complete visibility with hosted ELK, Grafana, and Prometheus-backed Observability

Start Free Trial

FAQ

We ship host logs with Filebeat today. Will the Linux hosts pack read them?

No. The pack filters on otel.resource.host.name, which only the OpenTelemetry path sets. Filebeat documents land in filebeat-* with ECS fields such as host.name. For SSH and sudo activity from Filebeat, the Linux auth log dashboards are the matching pack.

Can I make the dashboards use the severity number instead of the text?

Not without editing them, and you don't need to. The transform/severity processor derives the text from the number on the Collector, so the number is still the source of truth and the dashboards keep working as installed. For your own searches, otel.severity_number:[17 TO *] finds errors and worse regardless of the text.

Do I have to change every host before installing the pack?

No. The dashboards fill from whatever hosts already send a host name, so volume panels work straight away. Hosts still on the default config simply won't count towards errors and warnings until they get the new config, which is what the unclassified search helps you track.

Will reading the journal send much more than /var/log/syslog did?

Usually about the same, because rsyslog writes most of the journal to that file anyway. The priority: info setting drops debug entries. If a single unit is noisy, add units or matches to the receiver to narrow what it reads, rather than filtering after it has been sent.

Can we change the installed dashboards?

Yes. They are ordinary OpenSearch Dashboards objects on your stack. Save a copy before you change one, so you can see what you've added and compare it with the original.

Can we try this on a free trial?

Yes. Free trials have full access to every feature, including Hosted OpenTelemetry and the Dashboard Library, so you can point one test server at a trial stack, run the four logger lines and install the pack the same day.

Next steps

Swap the file tail for the journal receiver on one server, add the severity parser, the service name moves and transform/severity, and restart. Send the four test lines, check their severity in Discover, then install the Linux hosts pack and add the error burst and silent host alerts. Run the unclassified search once a week until it comes back empty. If you're running your own OpenSearch cluster for host logs, a Logit.io Logs stack gives you managed Logstash, OpenSearch and OpenSearch Dashboards with Hosted OpenTelemetry and the Dashboard Library built in. For a dedicated cluster, see Hosted OpenSearch and OpenSearch pricing. To ask the same questions in plain language, such as which hosts logged kernel errors last night, from any MCP-compatible AI assistant, see Hosted OpenSearch MCP.

Get the latest Elastic Stack & logging resources when you subscribe

Want to see this in action?
Start a free trial and connect logs to your alert workflows.