I am using Promtail to ship some Jenkins logs to Loki and I want to truncate the logs which are longer than e.g. 12k characters. This is the current configuration that I'm using:
clients:
- url: my-loki-instance
external_labels:
cluster: "clustername"
app: "jenkins-test-promtail"
namespace: ${JENKINS_NAMESPACE}
pod: ${JENKINS_POD_NAME}
container: ${JENKINS_CONTAINER}
node: ${JENKINS_NODE_NAME}
positions:
filename: "/var/log/jenkins/promtail/positions.yaml"
limits_config:
readline_rate_enabled: true
readline_rate_drop: false
options:
stream_lag_labels: namespace,pod,container
scrape_configs:
- job_name: master_logs
static_configs:
- labels:
component: master
__path__: /var/log/jenkins/master.0.log
pipeline_stages:
- regex:
expression: "(?P<truncated_log>.{0,12000}).*"
- output:
source: truncated_log
As expected, most of the logs are shorter than 12k characters and I get this error:
level=error ts=2022-10-06T13:48:37.260110981Z caller=main.go:117 msg="error creating promtail" error="failed to make file target manager: invalid regex stage config: could not compile regular expression: error parsing regexp: invalid repeat count: `{0,12000}`"
I have tried truncating to smaller values, like 20 characters and it works as expected because the lines are longer than 20 characters. Any idea how to overcome this or if there's another solution for it?
Promtail version: 2.6.1
UPDATE: I ended up using go templates to do the truncating, like this:
...
pipeline_stages
- match:
selector: '{component="master"}'
stages:
- template:
source: log_truncated
template: |
{{- if le (len .log) 12000 -}}
{{ .log }}
{{- end -}}
{{- if gt (len .log) 12000 -}}
{{ trunc 12000 .log }} TRUNCATED
{{- end -}}
- output:
source: log_truncated
...
