0

I need to get a connection from pool to connect to the upstream but facing 503 issues occassionally. I would like to know the root cause for this

https://github.com/envoyproxy/envoy/blob/master/source/common/router/router.cc

Http::ConnectionPool::Instance* conn_pool = getConnPool();
  if (!conn_pool) {
    sendNoHealthyUpstreamResponse();
    return Http::FilterHeadersStatus::StopIteration;
  }

This is the part of the code that is causing 503. I tried retrying with a while loop for fixed number of times but still facing the error. Can someone guide to get to the root cause or how I can handle this?

EDIT: One possible reasons I suspect would be that draining is occuring too often on envoy side. So is there a way I can control the drain_type of the duration for which draining occurs?

I have followed this documentation (https://www.envoyproxy.io/docs/envoy/latest/api-v1/listeners/listeners.html?highlight=draining%20modify_only) for changing the drain_type from default to modify_only.

But facing this error:

error initializing configuration '/etc/envoy-config.json': JSON at lines 3-57 does not conform to schema.
 Invalid schema: #
 Schema violation: additionalProperties
 Offending document key: #/drain_type
Nagireddy Hanisha
  • 1,290
  • 4
  • 17
  • 39

1 Answers1

0

One solution that I tried was increasing the drain_timeout to a larger value like 20 seconds rather than using the default value set to 5 seconds. This has considerably reduced the number of 503's that I am hitting right now by a factor of 2. The other side of this is still unknown.. What are the possible side effects that would arise with this setting is yet to be explored even though it is solving the purpose at the moment. Any suggestions/comments/solutions are welcome :)

Nagireddy Hanisha
  • 1,290
  • 4
  • 17
  • 39