graylog pipeline rules to skip the non-existing keys in the lookup table

Question

I am new to Graylog pipeline rules. I want to write a pipeline to skip the non-existing keys in the look up table from indexing.

we have 20 Ip addresses as Key and 4 regions as values. we are receiving more than 100 IP addresses from source. I want to index only 10 existing IP addresses in the look up table.

CSV file:

"IP_Address","region"
"IP1","region1"
"IP2","region4"
"IP3","region3"
"IP4","region1"
"IP5","region2"
"IP6","region4"
"IP7","region1"

I have tried by using many functions as below.

• Option 1: rule "look up IP address"

  when
    is_null(lookup_value("IP_Addr_table", IP_Address))
  then
    drop_message();
  end
  

• Option 2: Tried with two stages

  stage: 0

  rule "lookup region"

  when
    has_field("region")
  then
   let device_region = lookup_value("IP_addr_table", to_string($message.IP_Address));
  end
  

  Stage: 1

  rule "look up region"

  when
    has_field("device_region")    -- I want to check the variable value here.
  then
  end
  

Also, can anyone provide a good place to learn pipelines and how to use rules other than Graylog documentation

Answer 1

Just for reference - the same question is asked in the Graylog Community:

https://community.graylog.org/t/pipeline-rule-to-stop-indexing-for-non-lookup-values/9580

My answer will be given in that place.