Filebeat: Registry log.json grows constantly even with filebeat.registry.flush: 60s

Question

Filebeat 7.17.1

I'm looking into decreasing the amount of IO for the filebeat registry in our deployment. I found the filebeat.registry.flush setting after some research and thought that it would do what I wanted. I set it to 60s redeployed and monitored the registry directory. I didn't see what I expected as I found that the log.json constantly grows at least per second if not faster. I was expecting to see a gap of 60 seconds between the file modifications. I'm guessing my understanding is incorrect though?

I also tried changing the setting to 5s as a test to see if it was a problem with the amount of time. No difference...

Here is my filebeat.yml

filebeat.config.inputs:
  enabled: true
  path: /etc/filebeat/inputs.yml
  reload.enabled: true
  reload.period: 10s

filebeat.registry.path: /var/resin/logs/${REALM_CMID}/registry
filebeat.registry.flush: 5s
#================================ Outputs =====================================
output.logstash:
  hosts: ["${FILEBEAT_LOGSTASH_HOST}:5044"]
  ssl.certificate_authorities: ["/etc/ssl/certs/${RESIN_SSL_CERTIFICATE_NAME}"]
  ssl.verification_mode: none

I know the changes that I made are being read as I also changed the registry path and that did take effect.

We have approx 30 files being monitored. The input config for each is almost identical:

- type: filestream
  id: pok-account
  enabled: true
  fields:
    log_type: account
    product: pok
    kingdom: ${KINGDOM_CMID}
    realm: ${REALM_CMID}
  parsers:
    - multiline:
        type: pattern
        pattern: '^\['
        negate: true
        match: after
  paths:
    - /var/resin/logs/*/pok/log/*account.log

Any assistance or clarification would be greatly appreciated.

score 0 · Answer 1 · answered Aug 30 '23 at 16:22

This looks like a bug. The filebeat.registry.flush setting works when using the log input, but not filestream. So I would stick to the log input if you need to control the flush interval. This is my testing method.

filebeat.inputs:
- type: filestream # Or 'log'.
  paths:
    - ./test.log

filebeat.registry.flush: 10s

logging:
  level: debug
  selectors: [registrar]

http:
  host: 127.0.0.1
  port: 5066

output.console.enabled: true

Filebeat exposes a metric that it increments when the registry flushes. You can view it with:

curl --silent "http://127.0.0.1:5066/stats" | jq .registrar.writes.total

When using the log input the metric increments on a 10s interval as expected. It also logs a message for each flush.

When using filestream the metric is never incremented and there is no log message, but if I watch the underlying data/registry/filebeat/log.json file its modified time is continuously changing. So I think the filestream input is bypassing the registrar code that throttles flushes and is writing directly to the file-based data store. I recommend opening a bug with the product.

Filebeat: Registry log.json grows constantly even with filebeat.registry.flush: 60s

1 Answers1