Promtail: how to trim not JSON part from log

Question

I have multiline log that consists correct json part (one or more lines), and after it - stack trace. Is it possile to parse first part of the log as json, and for stack-trace make new label ("stackTrace" for example) and put there all the lines after first part?

Unfortunately, logs can contain a different number of fields in json format, and therefore it is unlikely to parse them using regex.

{ "timestamp" : "2022-03-28 14:33:00,000", "logger" : "appLog", "level" : "ERROR", "thread" : "ktor-8080", "url" : "/path","method" : "POST","httpStatusCode" : 400,"callId" : "f7a22bfb1466","errorMessage" : "Unexpected JSON token at offset 184: Encountered an unknown key 'a'. Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys. JSON input: {     \"entityId\" : \"TGT-8c8d950036bf\",     \"processCode\" : \"test\",     \"tokenType\" : \"SSO_CCOM\",     \"ttlMills\" : 600000,     \"a\" : \"a\" }" }
    com.example.info.core.WebApplicationException: Unexpected JSON token at offset 184: Encountered an unknown key 'a'.
    Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys.
    JSON input: {
        "entityId" : "TGT-8c8d950036bf",
        "processCode" : "test",
        "tokenType" : "SSO_CCOM",
        "ttlMills" : 600000,
        "a" : "a"
    }
       at com.example.info.signtoken.SignTokenApi$signTokenModule$2$1$1.invokeSuspend(SignTokenApi.kt:94)
       at com.example.info.signtoken.SignTokenApi$signTokenModule$2$1$1.invoke(SignTokenApi.kt)
       at com.example.info.signtoken.SignTokenApi$signTokenModule$2$1$1.invoke(SignTokenApi.kt)
       at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:248)
       at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:116)
       at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:136)
       at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:78)
       at io.ktor.routing.Routing.executeResult(Routing.kt:155)
       at io.ktor.routing.Routing.interceptor(Routing.kt:39)
       at io.ktor.routing.Routing$Feature$install$1.invokeSuspend(Routing.kt:107)
       at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)
       at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)

UPD. I've made promtail pipeline like so

scrape_configs:
  - job_name: Test_AppLog
    static_configs:
    - targets:
        - ${HOSTNAME}
      labels:
        job: INFO-Test_AppLog
        host: ${HOSTNAME}
        __path__: /home/adm_web/app.log
    pipeline_stages:
    - multiline:
        firstline: ^\{\s?\"timestamp\"
        max_lines: 128
        max_wait_time: 1s
    - match:
        selector: '{job="INFO-Test_AppLog"}'
        stages:
        - regex:
            expression: '(?P<log>^\{ ?\"timestamp\".*\}[\s])(?s)(?P<stacktrace>.*)'
        - labels:
            log:
            stacktrace:
        - json:
            expressions:
              logger: logger
              url: url
              method: method
              statusCode: httpStatusCode
              sla: sla
            source: log

But in fact, json config block does not work, the result in Grafana is only two fields - log and stacktrace.

Any help would be appreciated

score 0 · Answer 1 · answered Mar 28 '22 at 09:09

0

if the style is constantly like this maybe the easiest way is to analyze whole log string find index of last symbol "}" - then split the string using its index+1 and result should be in the first part of output array

answered Mar 28 '22 at 09:09

Lutonet

1
1

The style is similar to the one shown in the example, but the number of fields in the json log may differ, as well as the number of rows in the stack-trace. According to your suggestion, I don't quite understand how this can be implemented in the promtail configuration – Vitaly Motov Mar 28 '22 at 09:17
As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Mar 29 '22 at 16:03
I am sorry I was not completely clear. I am not sure which programming language you are using so I can't give you sample code but I am assuming you have this output as some sort of string variable - so you can just iterate through this string and memorize locations of the } symbol - when found one just record its position to variable and rewrite it each time when this } symbol is found - when this iteration finishes you should have stored index of the last accurance of the } symbol - so you know everything in front of this is JSON and can use string split method. – Lutonet Mar 30 '22 at 18:35
Your answer is reasonable, but unfortunately not applicable, since another json may be nested inside the json – Vitaly Motov Apr 05 '22 at 03:54

Promtail: how to trim not JSON part from log

1 Answers1