Promtail: How to remove timestamps from filenames?

Question

I have a simple problem:

My logfiles have timestamps in their name, i.e.:

/var/log/html/access-2021-11-27.log
/var/log/html/access-2021-11-28.log
/var/log/html/access-2021-11-29.log

Promtail is scraping this but does not "see" that access-2021-11-28.log is a continuation of access-2021-11-27.log. So it will "detect" a log file access-2021-11-28.log on the 28th and not show the access-2021-11-27.log anymore. I would want to see just "access.log" with data for several days.

I would assume this should be a well-known scenario, but I cannot find anything on this on the Internet.

score 1 · Accepted Answer · answered Nov 29 '21 at 14:11

The only way is to change log configuration of the application which is generating the logs, to use a unique access.log instead of the schema of the access-xxxx-xx-xx.log files. Unfortunately, this is not always possible.

But...

The old files can still be shown, it only depends on the time range used. Here is an example:

You can use regular expressions to perform the query, like in this example:

{filename=~".*JIRA_INSTALL/logs/access_log\\..*"}

score 1 · Answer 2 · answered Aug 01 '22 at 12:50

If you want to statically override the filename field you can so something as simple as this:

scrape_configs:
  - job_name: system
    static_configs:
      - labels:
          job: remotevarlogs
          __path__: /var/log/html/access-*.log
    pipeline_stages:
      - match:
          selector: '{job="remotevarlogs"}'
          stages:
          - static_labels:
              filename: '/var/log/html/access.log'

For those of you searching how to dynamically change the filepath prefix. For example, I'm using FreeBSD jails to nullfs mount my logs from other jails into a promtail jail. I don't want the local mount location (/mnt/logs/<hostname>) to show up as part of the path. Mounting shared folder could similarly be done with NFS or Docker.

scrape_configs:
  - job_name: system
    static_configs:
      - labels:
          job: remotevarlogs
          __path__: /mnt/logs/*/**/*.log
    pipeline_stages:
      - match:
          selector: '{job="remotevarlogs"}'
          stages:
          - regex:
              source: filename
              expression: "/mnt/logs/(?P<host>\\S+?)/(?P<relativepath>\\S+)"
          - template:
              source: host
              template: '{{ .Value }}.mylocaldomain.com'
          - template:
              source: relativepath
              template: '/var/log/{{ .Value }}'
          - labels:
              host:
              filename: relativepath
          - labeldrop:
              - job
              - relativepath

/etc/fstab for loki jail to pass-in /var/log/ directory from the grafana jail:

# Device                       Mountpoint                         FStype  Options          Dump Pass#
...
/jails/grafana/root/var/log/   /jails/loki/root/mnt/logs/grafana  nullfs  ro,nosuid,noexec 0    0
...

Now when I browse the logs, instead of seeing /mnt/logs/grafana/nginx/access.log, I see /var/log/nginx/access.log from grafana.mylocaldomain.com.

Promtail: How to remove timestamps from filenames?

2 Answers2