Apache access and error logs: 4xx, 5xx and busy clients

The Apache HTTP Server pack in the Logit.io Dashboard Library installs six OpenSearch Dashboards built from the two logs Apache already writes: an overview, HTTP 4xx errors, HTTP 5xx errors, high request rate clients, Apache error logs and a request investigation view. Every panel reads filebeat-* and filters on event.module:apache with event.dataset of apache.access or apache.error, so it fills when Filebeat ships your access and error logs through its apache module. Your Logs stack parses those lines into status codes, paths, client addresses and error-log modules, and within a few minutes you can see whether errors are rising, which URLs are failing and who is sending the traffic.

This guide is for teams running Apache in front of PHP, WordPress, Java or proxied applications on Linux or Windows. They want the 5xx spike, the scanner hammering /wp-login.php and the broken backend all in one place, instead of tailing access.log on each server. It covers which Apache log formats the parser understands and which it quietly gets wrong, where each distribution keeps the files, how to log the real client address behind a load balancer, how to prove the fields with two test requests, what each dashboard is for, three investigations worth knowing and how to turn the 5xx share into an alert.

Six dashboards from two Apache logs

The pack is called Apache HTTP Server in the Dashboard Library. Its six dashboards all read the filebeat-* index pattern with @timestamp as the time field, open on the last 24 hours and refresh every 60 seconds. Five of them are built on the access log and one on the error log, and each is built around a question:

The overview is the one to keep open. Across the top are four tiles: HTTP requests, HTTP 4xx responses, Last event received and Response bytes. Below them, a full-width chart stacks requests by status class over time, with a data volume line next to it. Then come donuts for the top paths and the request methods, an access map that plots source.geo.location, and a table of the latest 4xx and 5xx requests with the time, method, status, path, client address and host. The other five dashboards share a header row (requests, 4xx, 5xx and freshness) so you can move between them without losing your bearings.

Every panel starts from the same base query, written in Lucene:

event.module:apache AND agent.type:filebeat AND event.dataset:(apache.access OR apache.error)

The access panels narrow that to event.dataset:apache.access and add a status range such as http.response.status_code:[500 TO 599]. The error-log dashboard narrows it to event.dataset:apache.error. event.module and event.dataset come from Filebeat. Everything else, including the status code, the path and the client address, is set when your stack parses the log line. So what the dashboards count depends on whether your log format is one the parser recognises, and that's worth ten minutes before you trust a number.

Which Apache log formats the parser understands

When Filebeat's apache module sends an access or error log line, your Logs stack routes it through an Apache access or Apache error ingest pipeline before it's stored. Those pipelines use fixed patterns. A line that matches gets ECS fields. A line that doesn't match is still stored, but with an error.message saying the pattern didn't match and none of the HTTP fields. It still counts towards the HTTP requests tile, but not towards any status, path or client panel.

We ran a set of real-world Apache lines through the same Apache access and error pipeline definitions that Logit.io stacks use, on a local OpenSearch node, to see what comes out. This is what to expect:

A few details from those tests are worth knowing before you read the dashboards:

• The path and query are split. A request for /api/orders?id=42 gives url.path of /api/orders and url.query of id=42. The Top paths donut groups on url.path, so query strings don't fragment it.
• The user is often a dash. Without HTTP authentication, Apache writes - for the user and the pipeline keeps it, so user.name is the literal - on most lines. Search for NOT user.name:"-" when you want authenticated requests only.
• Requests with no request line still count. A client that connects and sends nothing gets a line such as "-" 408 -. It's parsed with a status of 408 and no method or path, so it shows in the 4xx dashboard as a request without a path.
• TLS sent to a plain HTTP port doesn't parse. Scanners that open a TLS handshake against port 80 leave lines like "\x16\x03\x01..." 400. Those don't match any pattern, so they're stored with error.message and don't reach the 4xx panels.
• Location is added from the client address. Public addresses get source.geo fields when the stack can run the ingest GeoIP processor (country, city and a source.geo.location point for the map) and source.as fields with the network owner. Private addresses, such as a load balancer's, get neither.
• Outcome follows the status. Below 400 is event.outcome:success, 400 and above is failure.

The error log is simpler. Apache 2.4's default error format, [time] [module:level] [pid N:tid N] [client address:port] message, gives you apache.error.module (such as core, proxy, ssl or php), log.level, process.pid, source.ip when there's a client and the message itself. The older 2.2-style format, [time] [level] [client address] message, still parses, but without a module. Error lines with no client part, such as start-up notices and MaxRequestWorkers warnings, have no source.ip, which is expected.

The error log has no time zone in its timestamps. The Filebeat module sends the server's time zone with each event, and the pipeline uses it to set @timestamp. If a server's clock or zone is wrong, error-log events will be shifted by whole hours relative to the access log, which does carry its offset.

Get Apache logs onto your stack

First, find your log files

Apache's log locations depend on the distribution and on whether you have TLS or per-site virtual hosts. Filebeat's module looks in the default locations only, and on Red Hat family servers that misses all your HTTPS traffic:

On Red Hat family servers with mod_ssl, HTTPS requests go to ssl_access_log and TLS-side errors go to ssl_error_log, and neither matches the module's default access_log* and error_log* globs. If your site is HTTPS only, the dashboards will show a trickle of port 80 redirects and nothing else. Leave ssl_request_log out: it repeats the same requests in a format without a status code.

On Debian and Ubuntu, check for per-site files with grep -R "CustomLog\|ErrorLog" /etc/apache2/sites-enabled/, and on Red Hat family with grep -R "CustomLog\|TransferLog\|ErrorLog" /etc/httpd/conf /etc/httpd/conf.d. Anything that writes to its own file needs a path in the module config below.

Enable the apache module

Start with the Apache HTTP Server integration guide in your Logit.io account. Choose Install Integration first, which prepares your stack for this source, then install Filebeat for your OS as the guide shows. The guide's examples use Filebeat 8.x. Then enable the module:

sudo filebeat modules enable apache

Edit /etc/filebeat/modules.d/apache.yml. Listing the paths explicitly means the same file works on Debian and Red Hat family servers and picks up the TLS logs:

- module: apache
  access:
    enabled: true
    var.paths:
      - /var/log/apache2/access.log*
      - /var/log/apache2/other_vhosts_access.log*
      - /var/log/apache2/*-access.log*        # per-site files, if you use them
      - /var/log/httpd/access_log*
      - /var/log/httpd/ssl_access_log*
  error:
    enabled: true
    var.paths:
      - /var/log/apache2/error.log*
      - /var/log/apache2/*-error.log*
      - /var/log/httpd/error_log*
      - /var/log/httpd/ssl_error_log*

Paths that don't exist on a given server are simply skipped, and rotated .gz files are excluded by the module. Adjust the per-site globs to match your own naming. If a per-site file uses vhost_combined or a custom format, check it against the table above first.

Set the output in /etc/filebeat/filebeat.yml. Copy the block from the integration guide while signed in, because it fills in your stack's Logstash host and port:

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.logstash:
  hosts: ["your-logstash-host:your-ssl-port"]
  loadbalance: true
  ssl.enabled: true

Then test and start:

sudo filebeat test config
sudo filebeat test output
sudo systemctl enable --now filebeat
sudo journalctl -u filebeat -f

test output should finish with talk to server... OK. The packaged Filebeat runs as root, so it can read /var/log/apache2 and /var/log/httpd. Beats data sent through your stack's Logstash lands in daily filebeat-* indices, which is the pattern the pack reads. If you have renamed the index in your Logs stack, the pack won't find your data.

Keep the module rather than a plain filestream input pointed at the same files. A plain input sends the lines without event.module:apache or the pipeline hint, so they're never parsed into HTTP fields and the pack never sees them.

Log the real client behind a load balancer

If Apache sits behind a load balancer, a CDN or a reverse proxy, the default combined format logs the address of the thing in front of it. Every request then appears to come from two or three private addresses, the high request rate clients dashboard shows your load balancer as the busiest client, and the access map is empty because private addresses have no location.

Apache's mod_remoteip fixes this at the source. It replaces the client address with the one in a forwarding header, but only when the request comes from a proxy you trust. On Debian and Ubuntu enable it with sudo a2enmod remoteip; on Red Hat family it's loaded by default. Then add a config file, for example /etc/apache2/conf-available/remoteip.conf (enable it with a2enconf remoteip) or /etc/httpd/conf.d/remoteip.conf:

RemoteIPHeader X-Forwarded-For
# Your load balancer's private range. Only these addresses are trusted to set the header.
RemoteIPInternalProxy 10.0.0.0/8

# Same as combined, with %a (the client address mod_remoteip sets) in place of %h
LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

Reload with sudo apachectl configtest && sudo systemctl reload apache2 (or httpd). The format is still combined as far as the parser is concerned, so nothing changes on your stack. Two cautions. Only trust the addresses of your own proxies: if you trust everything, any client can choose the address you log by sending its own header. And if you're behind a CDN whose addresses are public, use RemoteIPTrustedProxy with its published ranges, and its own header if it has one, such as CF-Connecting-IP for Cloudflare.

Prove it with two test requests

You don't need to wait for real traffic. Two requests from the server itself prove both halves of the path, one for each log:

curl -s -o /dev/null -w '%{http_code}\n' "http://localhost/logit-test-$(date +%Y%m%d)"
curl -s -o /dev/null -w '%{http_code}\n' "http://localhost/.htaccess"
sudo tail -n 2 /var/log/apache2/access.log /var/log/apache2/error.log   # or /var/log/httpd/access_log error_log

The first should print 404 and write an access log line for a path nothing real uses. The second should print 403, because both Debian and Red Hat family configs deny .ht files by default. It writes an access log line and an error log line that says AH01630: client denied by server configuration. If the server still uses Order/Allow/Deny, the same 403 is logged as AH01797. If your site redirects everything to HTTPS, use https://localhost/ with -k instead, and check ssl_access_log on Red Hat family.

Now open Discover in OpenSearch Dashboards, choose filebeat-*, set the query language to Lucene and the time picker to the last 15 minutes, and search:

event.dataset:apache.access AND url.path:\/logit-test-*

Open the document and check that http.response.status_code is 404, http.request.method is GET, event.outcome is failure and source.ip is 127.0.0.1 or ::1. Then check the error half:

event.dataset:apache.error AND message:AH01630

That should show apache.error.module as authz_core and log.level as error. If either document has error.message and no HTTP or module fields, the events arrived but weren't parsed. Check the format against the table above, and check that you chose Install Integration on the Apache HTTP Server page for this stack.

Next, check the field types with the OpenSearch API on your stack. The endpoint and stack API key are on the Endpoints page in your stack settings, and the guide to connecting to your stack's OpenSearch API covers both authentication modes. In API key mode:

curl -s "https://<stack-id>-es.logit.io/filebeat-*/_field_caps?fields=event.module,event.dataset,http.response.status_code,http.response.body.bytes,url.path,source.ip,source.geo.location,apache.error.module,log.level,host.name&apikey=<stack-api-key>&pretty"

You want http.response.status_code and http.response.body.bytes as long, source.ip as ip, source.geo.location as geo_point and the rest as keyword, all aggregatable. The numeric status code is what makes range filters such as [500 TO 599] work. Unlike logstash-*, where string fields are text with a .keyword sub-field, the Beats mapping on filebeat-* maps these fields as keyword directly, which is why the panels group on url.path rather than url.path.keyword. Index templates and mappings on Logit.io stacks are managed for you (the index templates guide shows how to read them). The default Filebeat template is applied when an index is created, so if a type looks wrong, fix it in the pipeline or the shipper rather than editing the live mapping; existing daily indices keep their first type until they age out.

Finally, count what the Library will see. Detection looks for recent documents in filebeat-* with event.module of apache, and needs event.dataset and http.response.status_code to be aggregatable. This query counts the last 24 hours:

curl -s -H 'Content-Type: application/json' \
  "https://<stack-id>-es.logit.io/filebeat-*/_search?apikey=<stack-api-key>" -d '{
  "size": 0,
  "query": { "bool": { "filter": [
    { "range": { "@timestamp": { "gte": "now-24h" } } },
    { "term": { "event.module": "apache" } }
  ] } },
  "aggs": {
    "datasets": { "terms": { "field": "event.dataset" } },
    "hosts":    { "terms": { "field": "host.name", "size": 50 } },
    "status":   { "terms": { "field": "http.response.status_code", "size": 10 } },
    "unparsed": { "filter": { "exists": { "field": "error.message" } } }
  }
}'

Expect an apache.access bucket and, once something has gone wrong, an apache.error bucket. Expect one hosts bucket per web server and a status list that looks like your traffic. If unparsed is more than a handful of documents, look at a few of them in Discover: it's almost always one server or virtual host with a custom format.

Install and know which dashboard to open

Open your Logs stack's card in your Logit.io account and choose Dashboard Library. Search for Apache HTTP Server. Once matching events have arrived the pack shows a Data detected badge. If you've only just started shipping, press Rescan stack. Some dashboards may already be ticked, based on what the Library found: access log records point it at the 4xx, 5xx, clients and investigation dashboards, and error log records at the error log dashboard. Use Select all to take all six, choose Install dashboards and open them in OpenSearch Dashboards a minute or two later. The Dashboard Library guide covers the permissions you need and what gets installed. The post on how Dashboard Library detection works explains the badge in more depth, using the sister Nginx pack as its example.

A sensible routine for the six dashboards:

1. Daily, the overview. Check Last event received first. On a site with steady traffic, a timestamp more than a few minutes old means Filebeat has stopped or the log moved, not that your visitors went home. Then look at the status chart. A 5xx band that appears where there wasn't one yesterday matters far more than the absolute number of 404s.
2. When 5xx appear, HTTP 5xx errors. Status codes and paths tell you whether it's one endpoint or everything. 503 across every path usually means a backend or the worker pool. 500 on one path usually means that application code.
3. When 4xx jump, HTTP 4xx errors. Same layout for client errors. A burst of 404s on paths you don't serve is scanning. A burst of 404s on paths you do serve is a broken link or a bad deploy.
4. When one address dominates, high request rate clients. Two donuts: all requests by source.ip, and 4xx responses by source.ip. An address that's big in the second and small in the first is failing at almost everything it asks for.
5. When the backend misbehaves, Apache error logs. Volume over time, by host and by level, with the module in the detail table. This is where proxy, PHP and TLS problems show up before users report them.
6. When you need the story, web request investigation. Add a filter for one source.ip, one url.path or one status at the top and read the requests in order alongside any error-log entries.

Three investigations worth knowing

A 5xx spike after a deploy

The overview's status chart grows a red band at 14:05, five minutes after a release. Open HTTP 5xx errors and look at the two donuts. If it's mostly 503 across many paths, Apache couldn't reach the application behind it. If it's 500 on one or two paths, the application answered with an error. Then open Apache error logs for the same minutes. With a proxied backend, connection failures look like this:

[proxy:error] [pid 4121:tid 140230] (111)Connection refused: AH00957: http: attempt to connect to 127.0.0.1:8080 (127.0.0.1:8080) failed

That parses with apache.error.module:proxy and log.level:error, so a search for event.dataset:apache.error AND apache.error.module:proxy* shows exactly when the backend stopped answering and on which hosts. For PHP-FPM, look for apache.error.module:proxy_fcgi. If the error log is quiet and the 500s are on one path, the cause is in your application's own logs, not Apache's.

If the 5xx affect every path and the error log shows AH00484: server reached MaxRequestWorkers setting, Apache ran out of workers. Requests queued, then timed out at the load balancer. That one is a capacity problem, and it's worth its own alert (below).

One client hammering the site

Requests double overnight and nothing else changed. Open high request rate clients. If one address is a quarter of all traffic and most of its requests are 4xx, it's probably a scanner. Confirm in Discover:

event.dataset:apache.access AND source.ip:"203.0.113.42" AND http.response.status_code:[400 TO 499]

Add url.path, user_agent.original and source.as.organization.name as columns. Paths such as /.env, /wp-login.php, /.git/config and /phpmyadmin/ on a site that has none of them are the classic pattern, and the network owner tells you whether it's a hosting provider or a real ISP. To see every scanner at once rather than one address, search across all clients:

event.dataset:apache.access AND url.path:(\/.env OR \/wp-login.php OR \/.git* OR \/phpmyadmin*)

The pack shows you the traffic. It doesn't block it. Use your firewall, your CDN or a tool like fail2ban for that, and come back to this dashboard to check that the blocks worked. If the busy client turns out to be one of your own monitoring checks or a load balancer health check, filter it out of the dashboard with a pinned negative filter on its url.path or user_agent.original.

A 404 wave that's your own broken link

The 4xx tile is three times its normal level, but no single address stands out. Open HTTP 4xx errors and look at the paths donut. When one real-looking path, such as /assets/app.3f9c1.js or /pricing/, is most of the 404s, something on your own site is linking to it. In Discover, search for that path and add http.request.referrer as a column:

event.dataset:apache.access AND http.response.status_code:404 AND url.path:"/assets/app.3f9c1.js"

The referrer column usually names the page carrying the broken link, or shows that it's your own HTML, cached by a CDN, referencing an asset from the previous release. That's a fix in minutes rather than a security investigation.

Turn the 5xx share into an alert

A dashboard only helps when someone is looking at it. Logs stacks include alerting with ElastAlert 2 rules, which you enable from Alerting & Notifications in your stack settings (the guide to creating alerting rules walks through it). Counting 5xx responses alone gives false alarms when traffic rises, so the percentage match rule type is a better fit: it alerts on the share of requests that failed.

name: Apache 5xx share above 5%
type: percentage_match
index: "filebeat-*"
buffer_time:
  minutes: 15
filter:
  - query:
      query_string:
        query: "event.module:apache AND event.dataset:apache.access"
match_bucket_filter:
  - range:
      http.response.status_code:
        gte: 500
max_percentage: 5
min_denominator: 200
realert:
  minutes: 30
alert:
  - "email"
email:
  - "[email protected]"

That fires when more than 5% of the last 15 minutes of Apache requests were 5xx, as long as there were at least 200 requests, so a quiet night with three requests and one error doesn't wake anyone. Add query_key: host.name if you'd rather get a separate alert per web server.

Two more rules are worth having. A flatline rule tells you when the access log goes quiet, which catches a stopped Filebeat as well as a stopped site:

name: Apache access log has gone quiet
type: flatline
index: "filebeat-*"
threshold: 1
timeframe:
  minutes: 15
query_key: host.name
filter:
  - query:
      query_string:
        query: "event.module:apache AND event.dataset:apache.access"
alert:
  - "email"
email:
  - "[email protected]"

Raise the threshold to match your traffic. A site that normally serves thousands of requests in 15 minutes is in trouble well before it reaches one. And a frequency rule catches worker exhaustion, which is rare enough that one occurrence is worth a message:

name: Apache reached MaxRequestWorkers
type: frequency
index: "filebeat-*"
num_events: 1
timeframe:
  minutes: 10
query_key: host.name
filter:
  - query:
      query_string:
        query: "event.dataset:apache.error AND message:AH00484"
alert:
  - "email"
email:
  - "[email protected]"

Run each with Test only before you enable it, so you can see how often it would have fired over recent data. If scanners are a recurring problem, a spike rule on 4xx responses, with spike_type: up and a threshold_cur above your normal background, is a good next step.

When the numbers look wrong

• The pack isn't detected. No recent document in filebeat-* has event.module:apache. Either Filebeat is using a plain input rather than the module, both filesets are disabled, the files aren't where the module looks or the events are going to a renamed index. The detection count above tells you which.
• HTTPS traffic is missing on Red Hat family servers. It's in ssl_access_log, which the module doesn't read by default. Add it to var.paths as shown above.
• Every request comes from two or three addresses. You're logging your load balancer. Set up mod_remoteip and log %a.
• The access map is empty. Same cause, or your traffic is internal. Private addresses aren't geolocated, so an intranet site will never fill the map.
• HTTP requests is higher than the status panels add up to. Some lines weren't parsed. Search event.dataset:apache.access AND error.message:* and look at event.original to see the format. TLS handshakes sent to port 80 and custom formats are the usual causes.
• url.domain shows 443 or 80. Those lines are from other_vhosts_access.log in vhost_combined format. Status, path and client are still right. If you need the site name, give each virtual host its own CustomLog in combined format, or log %v without the port.
• Error-log events are hours away from the matching requests. The error log has no time zone, so the server's zone is used. Check it with timedatectl.
• The error log dashboard is empty. Either the error fileset is disabled, the file isn't in var.paths, or Apache is genuinely quiet. Note that in Apache 2.4, a plain 404 for a missing file is logged at info level, which the default LogLevel warn doesn't write. Use the .htaccess test above to prove the path.
• Health checks dominate the top paths. Load balancer checks every few seconds can outnumber real requests. Pin a negative filter, or stop Apache logging them with a SetEnvIf on the health check path and CustomLog ... combined env=!healthcheck.
• Last event received is empty. No Apache events in the selected range. On a site with traffic, collection has stopped. Check journalctl -u filebeat for TLS or permission errors.

What the pack doesn't cover

• No response times. The parser reads the combined and common fields only, so %D or %T appended to your format is ignored and there's no latency panel. For latency, use the load balancer's own logs or application traces.
• No server status metrics. Busy and idle workers come from mod_status, not from logs. The same Apache integration guide shows how to send them to a Metrics stack with Telegraf.
• It reads logs, it doesn't block anything. Keep your firewall, CDN and web application firewall rules in front of Apache. Use the dashboards to see whether they're working.
• Filebeat only. Every query includes agent.type:filebeat and event.module:apache. Apache logs shipped with the OpenTelemetry Collector, Fluent Bit or a syslog forwarder still reach your stack, but not these dashboards. Apache running in containers and logging to stdout is covered by the container log dashboards instead.
• Access logs contain personal data. Client addresses, user names and full query strings are all in there, and query strings sometimes carry tokens or email addresses. Check who can view filebeat-* on your stack, keep retention to what you need and don't put secrets in URLs.

FAQ

Which Apache versions does the pack work with?

Apache 2.4 with the default access and error log formats is what the pack was built and checked against. Apache 2.2-style error logs also parse, but without the module name, so the module column in the error table stays empty.

We run Nginx as well. Do we need a different pack?

Yes. Nginx logs arrive with event.module:nginx and are covered by the separate Nginx pack, which has a similar layout. If you run both on one server, enable both Filebeat modules and install both packs: each one only counts its own module's events, so the numbers don't overlap.

Can I get response times into the dashboards?

Not from this pack's parser, which ignores fields added after the combined format. If you need request duration per path, the cleanest options are your load balancer's access logs or tracing in the application. Changing Apache's format to JSON will stop these dashboards from working, because JSON lines don't match the access patterns.

Does it work on Windows?

Yes. The module's Windows defaults are the Chocolatey path (C:/tools/Apache/httpd-2.*/Apache24/logs/) and the older ASF Program Files path (C:/Program Files/Apache Software Foundation/Apache2.*/logs/). Apache Lounge usually installs to C:/Apache24/logs/, which is not in those defaults, so set var.paths to your logs folder using forward slashes, for example C:/Apache24/logs/access.log*.

Can I change the panels or add my own?

Yes. The installed dashboards are ordinary OpenSearch Dashboards saved objects. Save a copy under a new name before you edit, so that reinstalling the pack later doesn't overwrite your changes. Adding url.domain or user_agent.name donuts to a copy of the overview is a common first change.

Can I try it on the free trial?

Yes. The 14-day free trial includes every feature, including the Dashboard Library, alerting and Hosted OpenSearch MCP. Point Filebeat at one web server, make the two test requests and install the pack.

Next steps

Find every access and error log your servers write, including ssl_access_log on Red Hat family, list them in the Filebeat apache module, set up mod_remoteip if you're behind a load balancer, make the two test requests and find them in Discover. Then install all six dashboards and add the 5xx share alert. Look at the high request rate clients dashboard once a week: it's the one that shows you traffic you didn't know you had. If you're running your own OpenSearch cluster for web logs, a Logit.io Logs stack gives you managed Logstash, OpenSearch and OpenSearch Dashboards with the Dashboard Library built in. For a dedicated cluster, see Hosted OpenSearch and OpenSearch pricing. To ask questions of the same access logs from any MCP-compatible AI assistant, such as which paths returned the most 5xx after yesterday's release, see Hosted OpenSearch MCP.