Filebeat filestream configs that pass test config but still fail

A Filebeat config can print Config OK and still exit on start, ship your multiline settings as event data, or send a whole log file to your stack twice. We found most of these while testing the Filebeat configs on our integration pages, then rebuilt each mistake, plus the migration traps around them, on Filebeat 8.19.23 and 9.5.5 to see exactly what happens.

This post shows each broken config, the log line or event it produces, the corrected config and how to prove the fix in OpenSearch Dashboards on logstash-*. If you are planning the move from the log input to filestream, or an upgrade to Filebeat 9, read the sections on ids and take_over before you change anything.

The short answer

filebeat test config only checks that your YAML parses and that each section has a shape Filebeat recognises. It does not check that a setting sits where the filestream input will read it, so a config can print Config OK and still exit on start, ship multiline settings as event data, or send a whole file twice. Put input settings directly on the input, put multiline under parsers, give every filestream input a unique id, and use take_over when you change an input's type or id.

We tested every case below on Filebeat 8.19.23 and 9.5.5 with the same 302-byte application log, and ran filebeat test config before each run. Every config passed that check, including three mistakes that then stopped Filebeat from starting on at least one of those versions.

Scroll sideways to see the full table.

Build a test rig you can throw away

You can reproduce all of this on a laptop in a few minutes without touching production. Unpack the Filebeat tarball, point each test at its own data directory so the registry starts clean, and write events to a local file instead of your stack. The output.file output lets you read exactly what Filebeat would have sent to Logstash.

curl -sLO https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.5.5-linux-x86_64.tar.gz
tar xzf filebeat-9.5.5-linux-x86_64.tar.gz
FB=$PWD/filebeat-9.5.5-linux-x86_64

mkdir -p rig/logs
printf '%s\n' \
  '2026-10-31 08:00:01 INFO  Order 1001 accepted' \
  '2026-10-31 08:00:02 ERROR Payment failed for order 1002' \
  'java.lang.IllegalStateException: card declined' \
  '    at com.example.pay.Gateway.charge(Gateway.java:88)' \
  '    at com.example.pay.Service.run(Service.java:41)' \
  '2026-10-31 08:00:03 INFO  Order 1003 accepted' > rig/logs/app.log

Then run any test config like this. Filebeat refuses a config file that other users can write to, hence the chmod.

chmod go-w rig/test.yml
$FB/filebeat test config -c $PWD/rig/test.yml --path.home $FB
timeout 15 $FB/filebeat -e -c $PWD/rig/test.yml --path.home $FB \
  --path.data $PWD/rig/data --path.logs $PWD/rig/fb-logs
cat rig/out/*.ndjson | jq -c '{message, fields, flags: .log.flags}'

Each test config ends with an output block like this one instead of output.logstash:

output.file:
  path: "/tmp/rig/out"
  filename: out

Change path to match where you created the rig. Delete rig/data between tests unless the test is about the registry.

Mistake 1: input_type instead of type

input_type is a very old spelling of type. It still turns up in copied snippets and in configs that have been upgraded in place for years.

filebeat.inputs:
- input_type: filestream
  paths: ["/var/log/payments/*.log"]

Both versions pass filebeat test config, log a deprecation warning, then exit. On 9.5.5 the error reads:

DEPRECATED: input_type input config is deprecated. Use type instead.
Exiting: Failed to start crawler: starting input failed: error while initializing input:
Error creating input. No such input type exists: 'filestream'

The message names filestream, which sends people looking for a missing plugin. The fix is one word: replace input_type with type and add an id. If Filebeat runs under systemd, this shows up as a restart loop in journalctl -u filebeat rather than as a failed deploy, so check the unit after every config change.

Mistake 2: input settings nested under fields

fields is a free-form map. Filebeat copies whatever you put there into every event and never interprets it. When an indentation slip moves real settings under fields, Filebeat accepts them as data.

filebeat.inputs:
- type: filestream
  id: payments-app
  paths: ["/var/log/payments/*.log"]
  fields:
    type: payments-app
    fields_under_root: true
    encoding: utf-8
    multiline.type: pattern
    multiline.pattern: '^\d{4}-\d{2}-\d{2}'
    multiline.negate: true
    multiline.match: after

Both versions start cleanly and log nothing unusual. The output shows what happened. The stack trace arrives as three separate events, and every event carries the settings as data:

{"message":"java.lang.IllegalStateException: card declined",
 "fields":{"type":"payments-app","fields_under_root":true,"encoding":"utf-8",
   "multiline":{"type":"pattern","pattern":"^\\d{4}-\\d{2}-\\d{2}","negate":true,"match":"after"}}}

So fields_under_root did not lift type to the top level, encoding was not applied and no lines were joined. Any Logstash conditional that expects a top-level [type] also stops matching, so events fall through to whatever your pipeline does with unmatched data.

The rule is simple. If Filebeat documents a key as an input option, it goes at the same indentation as paths. Only your own labels go under fields.

Mistake 3: multiline at input level on filestream

This one comes from migrating a type: log input by changing only the type. The old log input read multiline.* at input level. The filestream input does not, and it does not warn you.

filebeat.inputs:
- type: filestream
  id: payments-app
  paths: ["/var/log/payments/*.log"]
  multiline.type: pattern
  multiline.pattern: '^\d{4}-\d{2}-\d{2}'
  multiline.negate: true
  multiline.match: after

On both versions this runs with no warning and ships six events: the stack trace is split into three. Filestream reads multiline only as a parser:

filebeat.inputs:
- type: filestream
  id: payments-app
  paths: ["/var/log/payments/*.log"]
  parsers:
    - multiline:
        type: pattern
        pattern: '^\d{4}-\d{2}-\d{2}'
        negate: true
        match: after

With the parser in place the same file ships three events. The error event carries the whole trace in message and gets "log":{"flags":["multiline"]}, which is the field you search for later to confirm the fix.

Parsers run in the order you list them. If your application writes JSON lines, the stack trace is usually already inside one JSON string, so an ndjson parser on its own is enough and multiline is not needed. Keep multiline for plain-text logs where a trace spans raw lines. The filestream input reference lists every parser.

Mistake 4: missing or duplicate filestream ids

Filestream keeps each file's read position in the registry under a key that includes the input's id. Two inputs with no id share the empty id, which means they can overwrite each other's state.

filebeat.inputs:
- type: filestream
  paths: ["/var/log/payments/*.log"]
- type: filestream
  paths: ["/var/log/checkout/*.log"]

On 8.19.23 Filebeat keeps running and logs errors like these:

filestream input ID without ID might lead to data duplication,
please add an ID and restart Filebeat
filestream input with ID '' already exists, this will lead to data duplication,
please use a different ID. Metrics collection has been disabled on this input.

Because data still flows, that line is easy to miss. On 9.5.5 the same config does not start:

Exiting: filestream inputs validation error: filestream inputs with duplicated IDs: ""

Two inputs that both say id: app-logs behave the same way on each version. A single filestream input with no id still runs on 9.5.5, with a warning that it is discouraged. That is why an upgrade can fail on one host and pass on another: the hosts with one input are fine, and the hosts with two unnamed inputs stop.

Pick ids that describe the source, such as payments-app or nginx-access, and keep them stable. The id is part of the registry key, so treat it like a primary key rather than a label.

Mistake 5: changing the type or id without take_over

This is the expensive one, because it does not fail. It duplicates. When you change type: log to type: filestream, or rename a filestream id, Filebeat looks up the file under a new registry key, finds nothing and starts from offset zero.

We measured it on the test file. A type: log input shipped six events. Switching the same data directory to a filestream input shipped the same six events again. Renaming a filestream id from app-logs to app-logs-v2 on 9.5.5 did the same. On a host with weeks of logs on disk, that is weeks of duplicates landing in today's index.

The fix is take_over, which copies the old offset to the new input before it starts reading. On 8.19 and 9.0 the option is a boolean, and it takes over only from log inputs:

filebeat.inputs:
- type: filestream
  id: payments-app
  take_over: true
  paths: ["/var/log/payments/*.log"]

From 9.1 it is a block, and it can also take over from a filestream input with a different id, which is the only safe way to rename one:

filebeat.inputs:
- type: filestream
  id: payments-app-v2
  take_over:
    enabled: true
    from_ids: ["payments-app"]
  paths: ["/var/log/payments/*.log"]

With take_over set, both tests shipped zero duplicate events. On 8.19.23 Filebeat backed up the registry and logged filestream inputs took over 1 file(s) from loginputs. On 9.5.5 it logged Taking over state and migrated entry in registry for each file instead. We also ran a type: log input on 8.19.23, then started 9.5.5 on the same data directory with a take_over.enabled filestream input. It shipped nothing twice, so an upgrade and a migration can happen in one restart.

Three caveats from the 9.5.5 reference config. take_over is still marked beta. You must remove the old input in the same change, because taking over from an input that is still running is unsupported and can duplicate or lose data. And from_ids disables take-over from log inputs, so migrate from log first and rename later.

Mistake 6: leaving the log input in place on Filebeat 9

On 8.19.23 the old log input still works, including input-level multiline. On 9.5.5 it stops Filebeat by default:

Exiting: Failed to start crawler: starting input failed: error while initializing input:
cannot create log input: Found log input configuration: Log input is deprecated.
Use Filestream input instead.

Adding allow_deprecated_use: true to the log input brings it back on 9.5.5; we tested that too. Treat it as a stopgap for the upgrade window, not a fix, and plan the log to filestream migration with take_over before you upgrade the package. Elastic's migration guide covers the full option mapping. The Winlogbeat 8 vs 9 post covers the equivalent jump for Windows event logs.

A related trap on Windows: pointing Filebeat at C:\Windows\System32\winevt\Logs\*. Those are binary .evtx files, and Filebeat reads text lines. Use the Winlogbeat integration for Windows event logs and keep Filebeat for text logs.

The same bug class in Telegraf

If you also ship metrics with Telegraf, the same kind of drift happens there, but Telegraf is stricter. The Elasticsearch input dropped http_timeout in v1.35.0. On Telegraf 1.40.2 a config that still sets it fails to load:

E! loading config file t1.conf failed: plugin inputs.elasticsearch: line 3:
configuration specified the fields ["http_timeout"], but they were not used;
this is either a typo or this config option does not exist in this version

Replace it with timeout = "5s". telegraf --config telegraf.conf --test catches this before you restart the service, which is more than filebeat test config does for the Filebeat cases above.

A known-good filestream config for a Logit.io stack

This is the shape we recommend for a text log with stack traces, shipping to your stack's Logstash endpoint over TLS. Your Logstash endpoint and Beats SSL port are on the stack's Logstash inputs settings, and the Filebeat integration page pre-fills them when you are signed in.

filebeat.inputs:
- type: filestream
  id: payments-app
  enabled: true
  paths:
    - /var/log/payments/*.log
  fields:
    type: payments-app
  parsers:
    - multiline:
        type: pattern
        pattern: '^\d{4}-\d{2}-\d{2}'
        negate: true
        match: after

processors:
  - add_host_metadata: ~

output.logstash:
  hosts: ["your-logstash-endpoint-ls.logit.io:your-beats-ssl-port"]
  loadbalance: true
  ssl.enabled: true

Logstash on your stack handles parsing, and OpenSearch index templates and mappings are managed for you, so the Filebeat side only has to get whole events out with the right labels. If you need to parse the joined message further, the Grok parse failures post shows how to test patterns against real lines first.

Roll out a config change safely

1. Copy the new config to the host next to the old one, for example /etc/filebeat/filebeat.yml.new.
2. Run filebeat test config -c /etc/filebeat/filebeat.yml.new. Treat Config OK as a syntax check only.
3. Run filebeat test output -c /etc/filebeat/filebeat.yml.new to confirm the host can reach your Logstash endpoint over TLS.
4. Stop Filebeat and back up the registry directory (/var/lib/filebeat/registry on package installs). If you are changing a type or id, this is your way back.
5. Swap the config in and start Filebeat in the foreground for a minute with filebeat -e -c /etc/filebeat/filebeat.yml.
6. Watch for Exiting:, duplicated IDs, without ID and DEPRECATED. If you set take_over, look for took over on 8.19 or Taking over state on 9.x. Stop with Ctrl+C, then start the service as normal.
7. Verify on the stack, as below, before you roll the same change to the next host.

Verify the fix on your stack

In OpenSearch Dashboards, open Discover on the logstash-* index pattern and set the time range to cover the rollout. Three searches tell you most of what you need:

• fields.multiline.pattern:* should return nothing after the fix. Any hit is a host still shipping multiline settings as data.
• log.flags:multiline should start returning events from the hosts you changed once a multi-line event, such as a stack trace, has been written. If it stays empty after you know one was logged, the parser is not applied.
• fields.type:payments-app should match the events you expect. If you set fields_under_root: true at input level, search type:payments-app instead.

For a quick duplicate check after a type or id change, look for the same file offset shipped more than once from one host. Treat it as a signal rather than proof: offsets repeat after log rotation or truncation, and a terms aggregation can miss buckets on large data. Use the OpenSearch API with your stack credentials from the Endpoints settings:

curl -s -u "$LOGIT_OS_USER:$LOGIT_OS_PASS" \
  -H 'Content-Type: application/json' \
  "https://your-stack-id-es.logit.io/logstash-*/_search?size=0" -d '{
  "query": {"bool": {"filter": [
    {"match_phrase": {"host.name": "payments-01"}},
    {"term": {"log.file.path.keyword": "/var/log/payments/app.log"}},
    {"range": {"@timestamp": {"gte": "now-2h"}}}
  ]}},
  "aggs": {"repeated_offsets": {"terms": {
    "field": "log.offset", "min_doc_count": 2, "size": 20}}}
}'

Set the time range to cover both the original read and the re-send. A long repeated_offsets list that starts at offset 0 is the classic sign of a whole file sent again. On a default Logit.io stack, string fields such as log.file.path are indexed as text with a .keyword subfield, which is what the term filter needs. Paths longer than 256 characters are not stored in that subfield, so use a shorter path or a match_phrase on log.file.path for those. The connection guide shows both authentication modes, and the no data troubleshooting guide covers the case where nothing arrives at all.

If duplicates did land, do not delete by offset alone, because the original and the copy share the same log.offset. Narrow the copies down by host, path and the time window of the re-send, run that query as a _count first, and only then consider _delete_by_query, which the managed stack limitations page lists as available on your stack. For most teams, letting the duplicated window age out under normal retention is the safer choice.

Trade-offs worth knowing

Stricter 9.x validation is a good thing, but it moves failures to upgrade day. A config that has logged duplicate-id errors on 8.x for months will stop on 9.x. Search your Filebeat logs for without ID and already exists across the fleet before you upgrade.

File identity changed in 9.x. Filestream on 9.x identifies files by a fingerprint of their first 1,024 bytes by default. Current 9.x releases enable a growing fingerprint by default (file_identity.fingerprint.growing), so small files are read straight away, which is why our 302-byte test file shipped on 9.5.5. Some earlier 9.x releases waited until a file reached 1,024 bytes. If you pinned an older 9.x release, small new files may appear to be ignored until they grow.

Multiline patterns are cheap until they are not. A pattern that never matches keeps joining lines until the 5 second timeout flushes the event, and any lines beyond max_lines (500 by default) are discarded, not sent. You get huge events and silent data loss. Test the pattern against a real file in the rig before rollout.

Pin versions in config management. A few of our integration pages still installed Filebeat 8.11.3 or 8.15.2, both end of life, and several had the mistakes above. Our docs team ran each corrected config end to end, from Filebeat through Logstash over TLS into OpenSearch, and moved the pages to 8.19.23, the supported 8.x line. The tests in this post write to a local file so you can reproduce them without a stack. Those fixes are merged and will show on the integration pages once the docs site next publishes. Pin the version you tested in your own config management too, so a package update never changes the input behaviour under you.

FAQ

Why does filebeat test config say Config OK when Filebeat then exits?

It validates YAML and the overall config structure. Input-specific checks, such as which input type exists, whether ids are unique and whether the log input is allowed, run when the input starts. Run filebeat -e for a minute after every change. If it still will not start, our guide to diagnosing issues with your Filebeat configuration walks through the next checks.

Do I need an id on a single filestream input?

It runs without one on 8.19.23 and 9.5.5, with an error or warning in the log. Add one anyway, because adding a second input later turns the missing id into a duplicate, and adding an id later without take_over re-sends the file.

Will take_over work if I upgrade from 8.19 to 9.x and migrate in one go?

In our test, yes. A registry written by a log input on 8.19.23 was taken over by a 9.5.5 filestream input with take_over.enabled: true and no lines were re-sent. Back up the registry first, because the feature is still marked beta.

Where should fields_under_root go?

At input level, next to paths. Under fields it becomes a field called fields.fields_under_root and has no effect.

How do I find hosts still sending the broken config?

Search fields.multiline.pattern:* in Discover on logstash-* and add host.name as a column, or run a terms aggregation on host.name with the same query through the API.

Get whole events into Hosted OpenSearch

Every Logit.io Logs stack includes managed Logstash and OpenSearch Dashboards, so once Filebeat ships whole, correctly labelled events, search and alerting work on them straight away. See the Hosted OpenSearch overview for what is managed for you, compare plans on the Hosted OpenSearch pricing page, or ask your AI assistant about the results with Hosted OpenSearch MCP.