6

According to the git commit messages, ServiceStack has recently added failover support. I initially assumed this meant that I could pull one of my Redis instances down, and my pooled client manager would handle the failover elegantly and try to connect with one of my alternate Redis instances. Unfortunately, my code just bugs out and says that it can't connect with the initial Redis instance.

I am currently running instances of Redis 2.6.12 on a Windows, with the master at port 6379 and a slave at 6380, with sentinels set up to automatically promote the slave to a master if the master goes down. I am currently instantiating my client manager like this:

PooledRedisClientManager pooledClientManager =
    new PooledRedisClientManager(new string[1] { "localhost:6379"},
        new string[1] {"localhost:6380"});

where the first array is read-write hosts (for the master), and the second array is read-only hosts (for the slave).

When I terminate the master at port 6379, the sentinels promote the slave to a master. Now, when I try to run my C# code, instead of failing over to port 6380, it simply breaks and returns the error "could not connect to redis Instance at localhost:6379".

Is there a way around this, or will failover simply not work the way I want it to?

Vliu
  • 911
  • 8
  • 6
  • 1
    Have you found a way @Vliu – jaxxbo Jul 09 '13 at 21:26
  • No, from most of the research I've done, most people don't rely on the code to handle failover. They use a (linux) proxy layer over the redis database like HAProxy @jaxxbo – Vliu Jul 11 '13 at 21:45

1 Answers1

4

PooledRedisClientManager.FailoverTo allows you to reset which are the read/write hosts, vs readonly hosts, and restart the factory. This allows for a quick transition without needing to recreate clients.

TheDruidsKeeper
  • 325
  • 3
  • 12
  • 1
    Like a try{//redis operation} catch{ PooledRedisClientManager.FailoverTo(slave port as read-write port) }? – Vliu Sep 10 '13 at 02:37
  • 1
    That would be one way to do it, yes. I set up a special service to handle incoming requests from the Redis Sentinel, that way if it detected a failed node it would negotiate & promote the proper slave, then notify my webservices of the new address & port. – TheDruidsKeeper Sep 11 '13 at 16:26
  • Hi @TheDruidsKeeper, would you elaborate more on your approach with some samples, for ex, what service do you mean, and how exactly does it detect a failed node, greatly appreciate it. – wliao Jan 14 '14 at 21:47
  • @wliao I never got this fully up and running due to a bug (that has since been fixed) with the MS port of the sentinel. However, I have a webservice using ServiceStack that I had set up to use Redis as a message bus. I set up a route to handle a failover request (that would have been generated by the sentinel), to trigger the web service to call the `PooledRedisClientManager.FailoverTo`. Here is the specific liine of code: `((PooledRedisClientManager)this.ClientsManager).FailoverTo(new string[] { string.Format("{0}:{1}", request.IP, request.Port) });` – TheDruidsKeeper Mar 12 '14 at 19:38
  • I should also mention that the purpose of [Redis Sentinel](http://redis.io/topics/sentinel) is to detect the failed nodes, and automatically promote a slave to become a master. I was setting it up so that after the promotion, it would execute a custom script to notify my web service stack, but that wasn't possible at the time due to that previously mentioned [bug](https://github.com/MSOpenTech/redis/issues/60) - although this was a bug with the Windows port, not the *nix-native sentinel. – TheDruidsKeeper Mar 12 '14 at 19:48