Predis + Elasticache automated failover config

Question

I am running Predis against an Elasticache cluster on AWS which has a write master and two read replicas. Predis is configured for master/slave replication roughly as follows..

self::$client = new Predis\Client(
[
    'tcp://' . REDIS_MASTER  . '?alias=master',
    'tcp://' . REDIS_SLAVE01 . '?alias=slave-01',
    'tcp://' . REDIS_SLAVE02 . '?alias=slave-02'
],
['replication' => true]
);

I am in the process of configuring automated failover recovery. Elasticache supports master node failure recovery by promoting a read slave and updating the dns record for the master node's hostname. In the case of a failure, Predis should be none-the-wiser with the above config, since the hostname of the master will not change.

With the above config, however, I would have a problem. I would effectively going from a three node cluster to a two node cluster until a human intervened (or some heroic code was authored.)

To make my point clear.. Before failure..

REDIS_MASTER -> node1
REDIS_SLAVE01 -> node2
REDIS_SLAVE01 -> node3

After failure... (node1 fails and node2 gets promoted)

REDIS_MASTER -> node2
REDIS_SLAVE01 -> node2
REDIS_SLAVE01 -> node3

This is fine for a limited time, but ideally I would like this to solve itself.

When recovery is complete elasticache will restore node1 as a read replica. I would like it to start doing work as a read replica as soon as its available.

I was thinking that I could work around this by configuring Predis like so..

self::$client = new Predis\Client(
[
    'tcp://' . REDIS_MASTER  . '?alias=master',
    'tcp://' . REDIS_SLAVE01 . '?alias=slave-01',
    'tcp://' . REDIS_SLAVE02 . '?alias=slave-02',
    'tcp://' . REDIS_SLAVE03 . '?alias=slave-03'
],
['replication' => true]
);

...where REDIS_SLAVE03 points to the same underlying instance as REDIS_MASTER, but by a hostname that will not change in a failure event. Effectively, all nodes behave at all times as read slaves, and the 'pointer' to master for the writes is shifted without Predis's knowledge.

So a few questions...

1) what is Predis's behavior when a slave becomes unresponsive? Will it ignore that configuration and route reads to other, responsive slaves?

2) will the master redis instance get double the read operations? (presumably the answer is yes)

3) Are there flaws in this approach that I am missing?

4) Better idea?

Any and all advice is appreciated.

Are you sure that you will remain with 2 nodes after the automatic failover? Because I know a new read replica will be created in the place of the one that was promoted as master. And if you use the URLs (and not IPs) everything will be handled by AWS at the DNS level. You can check this: https://aws.amazon.com/blogs/aws/elasticache-redis-multi-az/ — Liviu Costea, Jul 15 '15 at 19:33
Liviu, you are right. If you reference the read replica's by their 'read endpoints' this all works as you describe. — rshepherd, Jul 16 '15 at 16:34

score 0 · Answer 1 · answered Jul 16 '15 at 16:35

0

This whole idea is flawed. Referencing the replicas by their read endpoints solves the issues, as aws will update the dns for those names as well during the recovery process.

answered Jul 16 '15 at 16:35

rshepherd

1,396
3
12
21

Predis + Elasticache automated failover config

1 Answers1