1

I am using a go client and server which is connected with grpc bidirectional stream. I need that stream to long running forever without any disconnection, but the stream disconnects within 3 minutes when the internet is down. Is there any way to stop the client from disconnecting or is there any way to reconnect automatically with the server when internet is down. If so please guide me with this. Thankyou.

Anand Samuel
  • 71
  • 2
  • 1
    If the network goes down then the stream will go down. When you receive an error (generally `io.EOF` when connection drops) you need to restart the stream (just add this to the loop in which you are calling `Recv`) - see [this answer](https://stackoverflow.com/a/66359526/11810946) for an example. If you are still struggling please share your current attempt. – Brits Mar 18 '22 at 19:10

1 Answers1

0

When gRPC connection is closed, the state of the gRPC client connection will be IDLE or TRANSIENT_FAILURE. Here is my example for a custom reconnect mechanism for gRPC bi-directional streaming in Go. First, I have a for loop to keep reconnecting until the gRPC server is up, which the state will become ready after calling conn.Connect().

for {
    select {
    case <-ctx.Done():
        return false
    default:
            if client.Conn.GetState() != connectivity.Ready {
                client.Conn.Connect()
            }

            // reserve a short duration (customizable) for conn to change state from idle to ready if grpc server is up
            time.Sleep(500 * time.Millisecond)

            if client.Conn.GetState() == connectivity.Ready {
                return true
            }

            // define reconnect time interval (backoff) or/and reconnect attempts here
            time.Sleep(2 * time.Second)
    }
}

Also, a goroutine will be spawned in order to execute the reconnect tasks. After successfully reconnect, it will spawn another goroutine to listen to gRPC server.

for {
    select {
    case <-ctx.Done():
        return
    case <-reconnectCh:
        if client.Conn.GetState() != connectivity.Ready && *isConnectedWebSocket {
            if o.waitUntilReady(client, isConnectedWebSocket, ctx) {
                err := o.generateNewProcessOrderStream(client, ctx)
                if err != nil {
                    logger.Logger.Error("failed to establish stream connection to grpc server ...")
                }

                // re-listening server side streaming
                go o.listenProcessOrderServerSide(client, reconnectCh, ctx, isConnectedWebSocket)
            }
        }
    }
}

Note that the listening task is handled concurrently by another goroutine.

// listening server side streaming
go o.listenProcessOrderServerSide(client, reconnectCh, websocketCtx, isConnectedWebSocket)

You can check out my code example here. Hope this helps.

yyh
  • 21
  • 1
  • Why is all of this needed? [ClientConn](https://pkg.go.dev/google.golang.org/grpc#ClientConn) "handles errors on established connections by re-resolving the name and reconnecting" so the library already handles much of this for you (generally there should be no need for your code to force a reconnect). – Brits Jan 05 '23 at 01:05
  • Of course ClientConn itself has its own built-in retry mechanism. Based on this [question](https://www.reddit.com/r/golang/comments/xywfk8/how_to_enable_grpc_reconnection_in_go/), it seems like only reconnecting until you send a request. This is just my reconnect implementation, to try reconnecting even before sending a request. Besides, I can set whether to reconnect based on error codes, eg: EOF, Canceled, Unavailable, etc. – yyh Jan 05 '23 at 03:26
  • Thanks - I added the comment because I felt that your answer adds complexity that is unnecessary in many cases (where [adding a loop](https://github.com/omri86/longlived-grpc/blob/master/client/client.go#L75) around the streaming call is sufficient). Just wanted to flag this for those who run across the answer in the future (yours is a perfectly valid approach, but its quite difficult to follow and easy to introduce data races/deadlocks). – Brits Jan 05 '23 at 05:16