Redis Sentinel for monitoring purposes? Notification script fires off too many times

Question

For my purposes, I only want one instance of redis running and one redis sentinel. I'm running Redis 3.0.6. My sentinel.conf is all defaults except quorum is 1 and the notification script line is uncommented: sentinel notificication-script mymaster /etc/redis/notify_me.sh. Inside notify_me.sh I execute a python script that for testing purposes just says print "HEY SOMETHING IS UP WITH REDIS".

I want to use redis sentinel for monitoring purposes only. Later, I will write something in the python script that will email/text me when redis goes down. However, as it is right now, it is firing off way too often. I only want to receive the message one time when sentinel as determined that redis has died. Right now when I start it off, the statement is printed out once at the beginning, and then several more times after failover-state-select-slave

23863:X 06 Jan 15:26:18.422 # Sentinel runid is db267af1b9257ced70eee9cbd076291db31f9335
23863:X 06 Jan 15:26:18.422 # +monitor master mymaster 127.0.0.1 6380 quorum 1
HEY SOMETHING IS UP WITH REDIS
23863:X 06 Jan 15:27:07.602 # +sdown master mymaster 127.0.0.1 6380
23863:X 06 Jan 15:27:07.602 # +odown master mymaster 127.0.0.1 6380 #quorum 1/1
23863:X 06 Jan 15:27:07.602 # +new-epoch 1
23863:X 06 Jan 15:27:07.602 # +try-failover master mymaster 127.0.0.1 6380
23863:X 06 Jan 15:27:07.604 # +vote-for-leader db267af1b9257ced70eee9cbd076291db31f9335 1
23863:X 06 Jan 15:27:07.604 # +elected-leader master mymaster 127.0.0.1 6380
23863:X 06 Jan 15:27:07.604 # +failover-state-select-slave master mymaster 127.0.0.1 6380
HEY SOMETHING IS UP WITH REDIS
HEY SOMETHING IS UP WITH REDIS
HEY SOMETHING IS UP WITH REDIS
HEY SOMETHING IS UP WITH REDIS
HEY SOMETHING IS UP WITH REDIS
HEY SOMETHING IS UP WITH REDIS
HEY SOMETHING IS UP WITH REDIS
23863:X 06 Jan 15:27:07.682 # -failover-abort-no-good-slave master mymaster 127.0.0.1 6380

I don't want it to print at the beginning and I only want it to print once when the server dies, so that I only get one email/text later. Anyone, got any tips on what I can do? Thanks!

pyramidface · Accepted Answer · 2016-01-07T18:23:48.433

Okay I've figured it out with the help of #redis on freenode. Inside my notify_me.sh, echo $* will show you some stuff like:

+odown master mymaster 127.0.0.1 6379 #quorum 1/1

The first thing is a pubsub message like the ones listed here: http://redis.io/topics/sentinel#pubsub-messages. +odown is when sentinel deems the server is objectively down, and this is when I wanted to do my python stuff. The notify_me.sh fires every time there's a message which is why I got so many HEY SOMETHING IS UP WITH REDIS, so I just wrote this:

In notify_me.sh,

#!/bin/sh
python notify_redis.py $*

And then in notify_redis.py,

import sys

def main(args):
    for arg in args:
        if arg == "+odown":
            print "HEY SOMETHING IS UP WITH REDIS"
            email_text_or_whatever_thing_you_wanna_do()

main(sys.argv)

Hope this helps someone!

score 0 · Answer 2 · answered Jan 07 '16 at 05:16

Not sure, but probably has something to do with the retry rule mentioned in sentinel.conf comments:

The scripts are executed with the following rules for error handling:

If script exits with "1" the execution is retried later (up to a maximum number of times currently set to 10).

If script exits with "2" (or an higher value) the script execution is not retried.

If script terminates because it receives a signal the behavior is the same as exit code 1.

A script has a maximum running time of 60 seconds. After this limit is reached the script is terminated with a SIGKILL and the execution retried.

Redis Sentinel for monitoring purposes? Notification script fires off too many times

2 Answers2