Graylog: Fields from Pipeline rule not showing up in search data

Question

I'm trying out Graylog for system logs and Snort alerts. I've followed the example here to get my snort alerts into Graylog and then proceeded to add another Stream, Pipeline and Rule for a separate IDS log source. I basically copied the Snort example and changed the Regex to extract the pertinent fields from the new log source. The new Regex does test OK when I select a message from the search tab and then select "Test against stream". Messages are also showing up under my new stream when selected from the Streams menu item at the top of Graylog.

The thing is, in the Pipeline rule, I use the set_field() function to assign the value of the matching group from the regex. I've got nine fields in the Rule using set_field() however none of them are showing up in the search data. Why is this? Do I need to use add_field() first like in the GELF examples? I was assuming set_field() automatically did that as that is how the snort example at the link above works. Attached my Pipeline rule below which is attached to my IDrops stream in Graylog.

rule "Extract IDropS fields"
when
  has_field("message")
then
  let m = regex("^([a-z]+)\\s.*(TCP|UDP|ICMP)\\s([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}):(\\d{1,5})\\s([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}):(\\d{1,5}).*SnortSid.*:([0-9]+):([0-9]+)\\s(.*)$", to_string($message.message));

  set_field("snort_alert", false);

  set_field("sd_host", m["0"]);
  set_field("sd_proto", m["1"]);
  set_field("sd_src", m["2"]);
  set_field("sd_sport", m["3"]);

  set_field("sd_dst", m["4"]);
  set_field("sd_dport", m["5"]);
  set_field("sd_sid", (m["6"]));
  set_field("sd_rev", m["7"]);

  set_field("sd_desc", m["8"]);
end

I've got the same issue; if I do set_field in one stage of the pipeline then subsequent stages can see it and act on it OK (e.g. for routing to different streams), but when the message actually gets to the stream the new field isn't there. — Vince Bowdren, Feb 03 '17 at 12:08
By contrast, the pipeline simulator runs through fine and under 'Added fields' it shows the fields and values I've added with `set_field`. — Vince Bowdren, Feb 03 '17 at 12:10

score 2 · Accepted Answer · edited Feb 24 '17 at 12:54

The problem might be you need to change the Message Processors Configuration order under Configurations screen.

By default it's GeoIP Processor > Pipeline Processor > Message Filter Chain.

As documented on Greylog official docs about pipelines' usage under "Configure the message processor", you need to change this to Message Filter Chain > Pipeline Processor > GeoIP Processor.

I forgot to do this on a newly installed 2.2 Graylog after having it work for some weeks under 2.1. Symptoms were the same as yours, the snort stream rules and pipeline rules matched, simulations worked, but no fields were actually set and no sign of errors, yet curiously it would work if I connected the pipeline to the default "All messages" stream, which I don't want. Now I've changed this message processor order the snort pipeline works as expected.

I'm changing the selected answer to yours. Mine solved the problem with 2.1, but as you say, upgrading to 2.2 was a problem. For me, all my custom fields disappeared from all my custom streams after upgrading to 2.2. I'm assuming this has something to do with the addition of the 'All Messages' stream and the "Default" stream has apparently been removed. After following your answer, I have all my custom fields back. *whew* thanks. — Server Fault, Feb 16 '17 at 21:45

score 0 · Answer 2 · answered Feb 04 '17 at 21:31

I'm not sure if it's the right way to do it, but I added all my Pipeline rules to the Default stream (System/Pipelines > Pipelines > Default Stream > Edit connections .. then add your custom pipeline) and then all of my fields started showing up in the search data.

Graylog: Fields from Pipeline rule not showing up in search data

2 Answers2