OpenTelemetry Collector on Linux and Windows servers: system logs and host metrics

On a fleet of servers, the OpenTelemetry Collector can replace two agents at once. One otelcol-contrib process tails the system log, scrapes CPU, memory, disk and network, and sends both to Logit.io through a single exporter with a single credential. It runs the same way on Debian, RHEL and Windows Server, which is the main reason to pick it over a Beats-plus-Telegraf setup when you're starting fresh.

This is the setup behind the Servers & log files and Host metrics options in Logit.io onboarding, written out so you can read it before you run it, or put it into Ansible, Salt or a golden image. Onboarding gives you the same commands with your stack's values filled in.

What you need first

• A Logs stack with Hosted OpenTelemetry switched on. New Logs stacks have it on already, on every plan. For an older stack, open the stack's Endpoints page. An account owner or administrator, or a stack administrator, presses Switch on on the Hosted OpenTelemetry card. Setup takes a few minutes, and then the connection details appear.
• The endpoint, username and API key from that card. The endpoint is https://your-stack-id-otel.logit.io and the username is your stack ID followed by -otel.
• A Metrics stack, if you want the host metrics. The endpoint accepts metrics without one, but drops them and shows a count in the dashboard, because there's nowhere to store them.
• Root on Linux, or an elevated PowerShell on Windows.

Everything below uses the contrib distribution, pinned to 0.161.0, the version onboarding installs. Contrib is the build that includes the basicauth extension and the filelog and windowseventlog receivers. The core build doesn't.

Install the Collector

Debian and Ubuntu

curl -fsSLO https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.161.0/otelcol-contrib_0.161.0_linux_amd64.deb
sudo dpkg -i otelcol-contrib_0.161.0_linux_amd64.deb
sudo usermod -aG adm otelcol-contrib

The package creates an otelcol-contrib systemd service that runs as an otelcol-contrib user and reads /etc/otelcol-contrib/config.yaml. Adding that user to adm lets it read /var/log/syslog.

RHEL and Amazon Linux

curl -fsSLO https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.161.0/otelcol-contrib_0.161.0_linux_amd64.rpm
sudo rpm -ivh otelcol-contrib_0.161.0_linux_amd64.rpm
sudo setfacl -m u:otelcol-contrib:r /var/log/messages

Group membership doesn't help here. /var/log/messages is usually root:root with mode 0600, so the ACL is what grants read access, and logrotate creates a new file without it. Either reapply the ACL after rotation (a postrotate line works) or read from journald with the journald receiver instead.

Windows Server

Invoke-WebRequest -Uri https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.161.0/otelcol-contrib_0.161.0_windows_x64.msi -OutFile otelcol-contrib_0.161.0_windows_x64.msi
Start-Process msiexec.exe -ArgumentList "/i otelcol-contrib_0.161.0_windows_x64.msi /qn" -Wait

The MSI installs to C:\Program Files\OpenTelemetry Collector and registers an otelcol-contrib service that reads config.yaml from that folder. It ships a default config, so the service is already running when the installer finishes. It runs as LocalSystem, which is why it can read the Security event log.

System logs

On Linux, replace /etc/otelcol-contrib/config.yaml with this:

extensions:
  basicauth/logit:
    client_auth:
      username: "your-stack-id-otel"
      password: "${env:LOGIT_API_KEY}"

receivers:
  filelog:
    include: ["/var/log/syslog", "/var/log/messages"]
    start_at: end
  otlp:
    protocols:
      http: { endpoint: "127.0.0.1:4318" }
      grpc: { endpoint: "127.0.0.1:4317" }

processors:
  resourcedetection:
    detectors: [system]
  resource/logit:
    attributes:
      - { key: service.name, value: "my-server", action: insert }
  batch: {}

exporters:
  otlphttp/logit:
    endpoint: "https://your-stack-id-otel.logit.io"
    auth: { authenticator: basicauth/logit }

service:
  extensions: [basicauth/logit]
  pipelines:
    logs:
      receivers: [filelog, otlp]
      processors: [resourcedetection, resource/logit, batch]
      exporters: [otlphttp/logit]

Both paths are listed so the same file works on Debian-family and RHEL-family hosts; whichever doesn't exist is ignored. Debian 12 and later and Amazon Linux 2023 don't install rsyslog by default, so neither file exists there; on those journald-only hosts, use the journald receiver instead (and skip the setfacl line) or install rsyslog. start_at: end means the Collector ships new lines from the moment it starts rather than replaying the whole file. Add your application's log files to include, as long as the otelcol-contrib user can read them.

The otlp receiver listens on loopback only. It's there so processes on the same box can send OTLP to localhost. The Hosted endpoint itself only takes OTLP/HTTP on 443, which is why the exporter is otlphttp. resourcedetection adds the hostname and OS, and the resource/logit processor sets service.name if nothing else has.

Change my-server to something that describes the box's role, such as web or db-primary. Because the processor uses insert, it only fills in service.name when nothing upstream has set one, and the hostname from resourcedetection still tells individual servers apart. That gives you a role to filter on in OpenSearch Dashboards and a host to drill into.

On Windows, the receivers change from files to event logs, and the rest of the file stays the same. Save it as C:\Program Files\OpenTelemetry Collector\config.yaml:

receivers:
  windowseventlog/application:
    channel: application
    start_at: end
  windowseventlog/system:
    channel: system
    start_at: end
  windowseventlog/security:
    channel: security
    start_at: end
  otlp:
    protocols:
      http: { endpoint: "127.0.0.1:4318" }
      grpc: { endpoint: "127.0.0.1:4317" }

# extensions, processors and exporters as above

service:
  extensions: [basicauth/logit]
  pipelines:
    logs:
      receivers: [windowseventlog/application, windowseventlog/system, windowseventlog/security, otlp]
      processors: [resourcedetection, resource/logit, batch]
      exporters: [otlphttp/logit]

Host metrics

The Host metrics config uses the same extension, exporter and processors, with the hostmetrics receiver and a metrics pipeline:

receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers: { cpu: {}, memory: {}, disk: {}, filesystem: {}, network: {}, load: {} }

service:
  extensions: [basicauth/logit]
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection, resource/logit, batch]
      exporters: [otlphttp/logit]

It's identical on Windows. Every scraper in that list runs there, with load emulated. Onboarding gives you logs and metrics as two separate files that each replace config.yaml. For both on one server, keep a single extensions, processors and exporters section, put all the receivers together, and list both pipelines under service. One exporter and one credential serve both.

Keep the key out of the config file

Onboarding's config has your API key written into the password field, which is the quickest way to see data. For anything managed by configuration management, the ${env:LOGIT_API_KEY} form above is better, but the variable has to reach the service, not just your shell.

On Linux, the packaged unit reads /etc/otelcol-contrib/otelcol-contrib.conf as its environment file:

sudo sed -i '/^LOGIT_API_KEY=/d' /etc/otelcol-contrib/otelcol-contrib.conf
echo "LOGIT_API_KEY=$LOGIT_API_KEY" | sudo tee -a /etc/otelcol-contrib/otelcol-contrib.conf >/dev/null
sudo chmod 600 /etc/otelcol-contrib/otelcol-contrib.conf

On Windows, a machine-level environment variable isn't visible to a service started by the Service Control Manager until a reboot. Set it on the service itself:

Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\otelcol-contrib -Name Environment -Type MultiString -Value "LOGIT_API_KEY=$env:LOGIT_API_KEY"

Restart and check

# Linux
sudo systemctl restart otelcol-contrib
sudo journalctl -u otelcol-contrib -n 20 --no-pager

# Windows
Restart-Service otelcol-contrib
Get-WinEvent -FilterHashtable @{ LogName = 'Application'; ProviderName = 'otelcol-contrib' } -MaxEvents 20 | Format-List TimeCreated, Message

A clean start shows no exporter errors. A 401 in that output means the username or key is wrong, or the variable didn't reach the service. Then check the data: logs appear in OpenSearch Dashboards on your Logs stack, so search for your hostname or service.name. With start_at: end, nothing appears until something new is written, so logger "otel test" on Linux is a quick nudge. Metrics arrive in your Metrics stack every 30 seconds.

From there, building a first Grafana dashboard is a good use of the host metrics, and Five ways Windows Event Logs reach OpenSearch compares the Collector with Winlogbeat and the other Windows options. The Collector install docs and our OpenTelemetry getting started docs cover the rest.