sendTransaction Fails when it's invoked after the network is idle for 15-20 minutes

Question

I have a hyperledger fabric network deployed in GCP. I use fabric node sdk to communicate to the network.

When there's no activity between sdk and network, If i try to invoke the chaincode, the API call fails at sendTransaction to the Orderer with below logs.

error: [Orderer.js]: sendBroadcast - on error: "Error: 14 UNAVAILABLE: TCP Read failed\n at createStatusError (/usr/src/app/node_modules/grpc/src/client.js:64:15)\n at ClientDuplexStream._emitStatusIfDone (/usr/src/app/node_modules/grpc/src/client.js:270:19)\n at ClientDuplexStream._receiveStatus (/usr/src/app/node_modules/grpc/src/client.js:248:8)\n at /usr/src/app/node_modules/grpc/src/client.js:804:12" Request failed: POST /channels/stanfinandcredth/chaincodes/sc1547746253: SERVICE_UNAVAILABLE Error: SERVICE_UNAVAILABLE at ClientDuplexStream. (/usr/src/app/node_modules/fabric-client/lib/Orderer.js:136:21) at emitOne (events.js:116:13) at ClientDuplexStream.emit (events.js:211:7) at ClientDuplexStream._emitStatusIfDone (/usr/src/app/node_modules/grpc/src/client.js:271:12) at ClientDuplexStream._receiveStatus (/usr/src/app/node_modules/grpc/src/client.js:248:8) at /usr/src/app/node_modules/grpc/src/client.js:804:12 Error: SERVICE_UNAVAILABLE at ClientDuplexStream. (/usr/src/app/node_modules/fabric-client/lib/Orderer.js:136:21) at emitOne (events.js:116:13) at ClientDuplexStream.emit (events.js:211:7) at ClientDuplexStream._emitStatusIfDone (/usr/src/app/node_modules/grpc/src/client.js:271:12) at ClientDuplexStream._receiveStatus (/usr/src/app/node_modules/grpc/src/client.js:248:8) at /usr/src/app/node_modules/grpc/src/client.js:804:12

It seems that SDK is not able to reach to Orderer. The immediate next invoke request succeeds. this happens eerytime after idle time of 15 mins.

I tried setting grpc timeouts for the connection but it doesn't seem to help.

I created a new Orderer object everytime I wanted instead of using the single orderer object. I'm not sure whether it's right way to do it but seems to be working for me. Also, I had tried setting gRPC connection timeouts, but I'm not sure if it helped much. — jignesh, May 13 '19 at 15:17

score 2 · Accepted Answer · answered Jan 20 '19 at 20:54

It's likely that idle connections are being torn down by GCP. You'll need to set the gRPC keepalive options for your client. Something like:

    "connection-options": {
        "grpc.max_receive_message_length": -1,
        "grpc.max_send_message_length": -1,
        "grpc.keepalive_time_ms": 120000, 
        "grpc.http2.min_time_between_pings_ms": 120000, 
        "grpc.keepalive_timeout_ms": 20000, 
        "grpc.http2.max_pings_without_data": 0, 
        "grpc.keepalive_permit_without_calls": 1 
    }

See https://fabric-sdk-node.github.io/release-1.4/tutorial-grpc-settings.html for details on setting these using the Node SDK

sendTransaction Fails when it's invoked after the network is idle for 15-20 minutes

1 Answers1