How can I config envoy proxy to failover when some upstream being unreachable

Question

I am new to envoy proxy, what I need is using envoy as sidecar proxy between grpc client and server.

So far, I have connected a grpc client and two server, with lb_policy set as ROUND_ROBIN. But when I close one of the servers, grpc client call would fail.

So, How can I config envoy to deal with this scenario?

This is my envoy config:

admin:
  access_log_path: "/tmp/admin_access.log"
  address:
    socket_address:
      address: "10.19.17.188"
      port_value: 12000
static_resources:
  listeners:
    -
      name: "grpc-listener"
      address:
        socket_address:
          address: "10.19.17.188"
          port_value: 12001
      filter_chains:
        -
          filters:
            -
              name: "envoy.http_connection_manager"
              config:
                stat_prefix: "ingress"
                codec_type: "AUTO"
                route_config:
                  name: "grpc-route"
                  virtual_hosts:
                    -
                      name: "grpc-route"
                      domains:
                        - "*"
                      routes:
                        -
                          match:
                            prefix: "/"
                          route:
                            cluster: "grpc-service"
                http_filters:
                  -
                    name: "envoy.router"

  clusters:
      -
        name: "grpc-service"
        connect_timeout: "0.25s"
        type: "static"
        lb_policy: "ROUND_ROBIN"
        http2_protocol_options: {}
        hosts:
          -
            socket_address:
              address: "10.19.17.188"
              port_value: 12011
          -
            socket_address:
              address: "10.19.17.188"
              port_value: 12012

python grpc client error message:

Traceback (most recent call last):
  File "greeter_client.py", line 40, in <module>
    run()
  File "greeter_client.py", line 32, in run
    response = stub.SayHello(helloworld_pb2.HelloRequest(name='%03d'%i))
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "upstream connect error or disconnect/reset before headers. reset reason: connection failure"
    debug_error_string = "{"created":"@1568810074.217216860","description":"Error received from peer ipv4:10.19.17.188:12001","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"upstream connect error or disconnect/reset before headers. reset reason: connection failure","grpc_status":14}"

envoy log:

[2019-09-18 12:29:02.380][798][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
[2019-09-18 12:29:02.380][798][debug][http] [source/common/http/conn_manager_impl.cc:1111] [C285][S679184262732628339] request end stream
[2019-09-18 12:29:02.380][798][debug][connection] [source/common/network/connection_impl.cc:561] [C286] delayed connection error: 111
[2019-09-18 12:29:02.380][798][debug][connection] [source/common/network/connection_impl.cc:190] [C286] closing socket: 0
[2019-09-18 12:29:02.380][798][debug][client] [source/common/http/codec_client.cc:82] [C286] disconnect. resetting 0 pending requests
[2019-09-18 12:29:02.380][798][debug][pool] [source/common/http/http2/conn_pool.cc:149] [C286] client disconnected
[2019-09-18 12:29:02.380][798][debug][router] [source/common/router/router.cc:868] [C285][S679184262732628339] upstream reset: reset reason connection failure
[2019-09-18 12:29:02.380][798][debug][http] [source/common/http/conn_manager_impl.cc:1186] [C285][S679184262732628339] Sending local reply with details upstream_reset_before_response_started{connection failure}
[2019-09-18 12:29:02.380][798][debug][http] [source/common/http/conn_manager_impl.cc:1378] [C285][S679184262732628339] encoding headers via codec (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection failure'
'date', 'Wed, 18 Sep 2019 12:29:02 GMT'
'server', 'envoy'

score 3 · Answer 1 · answered Oct 03 '19 at 18:07

You can also use one of the following approaches:

Enable active health check in envoy for your clusters. To do this you must also expose a health check endpoint from your service, which should be fairly easy. Refer https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/core/health_check.proto
Switch over to dynamic configuration of envoy using additional components: a) control-plane such as go-control-plane and b) service discovery service such as consul. This will certainly increase the complexity of your service setup but it would also support a more dynamic and robust solution.

score 2 · Accepted Answer · answered Sep 20 '19 at 04:50

outlier detection

Helped

  clusters:
      -
        name: "grpc-service"
        connect_timeout: "0.25s"
        type: "static"
        lb_policy: "ROUND_ROBIN"
        http2_protocol_options: {}
        hosts:
          -
            socket_address:
              address: "10.19.17.188"
              port_value: 12011
          -
            socket_address:
              address: "10.19.17.188"
              port_value: 12012
        outlier_detection:      # where amazing happened 
            consecutive_5xx: 1

How can I config envoy proxy to failover when some upstream being unreachable

2 Answers2