1

I am upgrading my ELK+Redis from a very old version to the latest one.

ES 1.6 to 7.4
LS 1.4 to 7.4
Redis 2.8 to 5.0

I freshly installed everything in a fresh machine. Things went smooth except one thing: logstash-filter-redis plugin

Here is my logstash conf file 

input
  {
    redis {
      host
      port => 6380
      data_type
      key
    }
  }
filter {
  grok {
    match => [ "message", "" ]
  }
  redis {
    host => ""
    port => 6379
    key => ""
    }
output {
  elasticsearch {
    host => ""
    index => ""
  }

Here, the first redis:6380 inside input { } is being used to fetch logs (working as logstash input source)

while second redis:6379 is being called inside filter { } to get data associated with those logs.

I cannot see any logstash-filter-redis plugin at https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

Can someone please help me with that?

Thanks

Val
  • 207,596
  • 13
  • 358
  • 360
Rohail Abbas
  • 551
  • 1
  • 7
  • 20
  • 1
    The `logstash-filter-redis` is not bundled with logstash and it seems that the developer didn't update it in years, you can try to install it, but it probably won't work with the newer versions. – leandrojmp Oct 15 '19 at 15:43

1 Answers1

5

There are indeed no supported Redis filters in Logstash. You can find two community filters by synlay and meulop but I'm unsure how well they are supported and maintained.

I just learned recently another way of enriching your data from Redis by accessing Redis through a ruby filter like this :

filter {
   ...
   ruby {
      init => 'require "redis"; $rc = Redis.new(path: "/var/run/redis/redis.sock", db: 0)'
      code => 'event.set("enriched_field", $rc.get(event.get("key_field")))'
   }
}
Val
  • 207,596
  • 13
  • 358
  • 360
  • 3
    thanks for sharing this - the suggested approach above worked for me - a couple of additional things - you will need to add the redis gem first (https://stackoverflow.com/a/54224889/150878) and if you have a remote server, use this: $rc = Redis.new(host: "", port: ) – sami Jan 31 '20 at 06:25
  • Thank you @Val. It surely helped and we used the same to serve the purpose. No other filter worked. – Rohail Abbas Feb 13 '20 at 06:51
  • How should we handle possible errors in `Redis.new()` if Redis is unreachable or set a timeout in `$rc.get()` if Redis is very slow to respond? – Mohammadreza Sahelgozin Aug 14 '22 at 11:21
  • 1
    @MohammadrezaSahelgozin please ask a new question, possibly referencing this one – Val Aug 14 '22 at 13:09