3

I've just started exploring Loki to aggregate my logs and I'm assuming I've missed something obvious.

I have a bunch of machines/vms/containers with promtail forwarding systemd's journal to a single loki server (just the default example config) and I can explore them just fine. I can filter the logs by any of the labels as required.

However, on a dashboard panel I have tried both the "table" and "logs" panel types and can't seem to do what seems extremely basic to me. I want filter by some of my labels and then extract one or more of the remaining labels as a column to be displayed. The closest I have found is to turn on "Unique labels" in the "logs" panel type which just throws all of the values in one column together.

What have I missed?

Richard Huxton
  • 21,516
  • 3
  • 39
  • 51

3 Answers3

1

Just starting out also, so please don't take this as authoritative.

I made some progress by:

  1. processing the log "Line" using a regexp to create capture groups as referable labels, and

  2. using the line_format to re-assemble the log line in the desired format

    ... | regexp ^(?P<thread><.*>)\s+(?P<channel>\[.*\])\s+(?P<message>.*)$ | line_format "{{.env}} {{.node}} {{.thread}} {{.channel}} {{.message}}"

Hope it helps.

Update:

You can also prepend information using the template stage when sending, something like as follows (in the agent config):

- template:                  
    source: message                  
    template: '{{ .node }} {{ .message }}'
paulkmoore
  • 3,253
  • 2
  • 21
  • 20
  • Thanks for the suggestion - that's mangling the log-line itself though, which wasn't what I was after. I suppose I could keep the original line after the newly regexp-extracted repetition of the label. – Richard Huxton Nov 02 '22 at 13:50
  • You can also mangle it at the sender (updated answer), but I agree that both feel a bit hacky – paulkmoore Nov 03 '22 at 01:23
  • While I can get labels/fields I want in the data by configuring `promtail` with a regex, I'm still confused about the (non-)possibilities to have **columns** in a **logs panel**, i.e. some *table* functionality in a *logs panel*. It seems, surprisingly, that there is no solution for that yet? (https://github.com/grafana/grafana/discussions/42315) – ppenguin Nov 07 '22 at 09:50
1

Use line_format and the line function in the following form:

... | line_format "{{(printf \"%-7s\" .node)}} {{__line__}}"

assuming that node is the label, and you want those printf format specifiers (adjust to taste).

line_format allows any go template argument, so should be able to achieve the need I think.

Note: for performance reasons it is preferred to put such processing after any filtering steps.

paulkmoore
  • 3,253
  • 2
  • 21
  • 20
1

You can use the table panel type. Then go to the Transform tab which is located next to the Query tab at the bottom. Add an Extract fields transformation and for the Source dropdown select Labels. Then press Add path button and enter the name of the label you want as a column.

Note that there is also an Organize Fields transform which is useful to control the visibility and ordering of columns.

Jonas Kello
  • 1,178
  • 2
  • 13
  • 25