I am writing an app that uses RQ for jobs. The Redis instance for RQ is in Heroku, but the RQ workers are on external machines. They make a Redis connection when they start.
If Heroku moves the Redis instance (which it can and does), then the workers get stranded, i.e. with a connection to a Redis instance that no longer exists, and without a connection to the new one.
I can get the redis URL from heroku using the API, but how do I get the workers to reconnect?
I'm thinking of getting the Redis URL on startup, then having a background thread that pings the Redis server every N minutes, and exits when it can't talk to Redis.
Since I'll have a process monitor (circus, supervisor, something), it would just restart the worker.
Is this reasonable? Is there a more common best practice for running RQ workers outside of Heroku, but the Redis instance inside Heroku?
For that matter, it seems like RQ workers on Heroku would have to deal with this same issue.
