Retry Pattern Vs fall back pattern in rest client

Question

We have multiple resources exposed as REST service and running in to design argument whether client needs to implement re-try logic if service in unavailable due to network and/or application level failures. Is it worth it? One group argues, if service is unavailable then there is no point of re-trying but other group argues there may be network busy issues and re-try could help. There are no statistics to defend either arguments at this time. How about implementing a fall back URL (a replica of original http resource) and use fall back service during failures.

Any suggestions based on your prior experience?

Answer 1

Keep in mind that when a service request is failing, it can be due to network overload or service degradation. In several cases the best option is to just fail immediately. Regarding using a fall back URL, it probably will not solve your problem, since it could keep network under high load.

Suggestion is to take a look in patterns such as:

• Backpressure: http://mechanical-sympathy.blogspot.com.br/2012/05/apply-back-pressure-when-overloaded.html , http://engineering.voxer.com/2013/09/16/backpressure-in-nodejs/
• Circuit breaker, Load Shedding: http://martinfowler.com/bliki/CircuitBreaker.html , http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html
• Bulkhead pattern: http://skife.org/architecture/fault-tolerance/2009/12/31/bulkheads.html

Answer 2

It's generally a good idea to retry failed requests, but take care to always set a reasonable retry limit with proportion to the timeout. A very good way to avoid bringing down a server with retry requests is to use an exponential backoff. For example the first retry after 30 seconds, the next after 300 seconds, etc.

It's also common to have certain server responses that signal the client to not retry. These are used when the server experiences problems that won't be resolved by trying again later, like a DB failure.

A fallback URL seems unRESTful- there should be a single endpoint for a resource. It shouldn't matter to the client whether that endpoint is backed by your primary stack or a backup. Typically a dispatcher would be used to failover to a different server pool such that if the primary fails, it can divert traffic to the standbys until the problem is fixed.