Failed to connect before the deadline on Endorser

Question

connection-profile.yaml

{
    "name": "network",
    "version": "1.0.0",
    "client": {
        "organization": "Freight",
        "connection": {
            "timeout": {
                "peer": {
                    "endorser": "300"
                }
            }
        }
    },
    "organizations": {
        "Freight": {
            "mspid": "FreightMSP",
            "peers": [
                "peer0.freight.example.com",
                "peer1.freight.example.com"
            ],
            "certificateAuthorities": [
                "ca.freight.example.com"
            ]
        }
    },
    "peers": {
        "peer0.freight.example.com":{
            "url": "grpcs://peer0.freight.example.com:9051",
            "grpcOption": {
                "ssl-target-name-override": "peer0.freight.example.com",
                "hostnameOverride": "peer0.freight.example.com"
            },
            "tlsCACerts": {
                "path": "../../network/organizations/peerOrganizations/freight.example.com/tlsca/tlsca.freight.example.com-cert.pem"
            }
        },
        "peer1.freight.example.com":{
            "url": "grpcs://peer1.freight.example.com:10051",
            "grpcOption": {
                "ssl-target-name-override": "peer1.freight.example.com",
                "hostnameOverride": "peer1.freight.example.com"
            },
            "tlsCACerts": {
                "path": "../../network/organizations/peerOrganizations/freight.example.com/tlsca/tlsca.freight.example.com-cert.pem"
            }
        }
    },
    "certificateAuthorities": {
        "ca.freight.example.com": {
            "url": "https://ca.freight.example.com:9054",
            "caName": "ca.freight.example.com",
            "httpOptions": {
                "verify": false
            },
            "tlsCACerts": {
                "path": "../../network/organizations/peerOrganizations/freight.example.com/ca/ca.freight.example.com-cert.pem"
            }
        }
    }
}

Discovery Service:

discovery: { 
        enabled: true, 
        asLocalhost: false 
    }

asLocalhost is false because all orgs are hosted on different hosts and are in docker swarm network. But i get this error when call the add asset invoke function.

    Wallet Path: /home/hlfabric/fabric/api/freight/wallet
2021-03-06T17:58:15.407Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer0.freight.example.com, url:grpcs://peer0.freight.example.com:9051, connected:false, connectAttempted:true
2021-03-06T17:58:15.408Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer0.freight.example.com url:grpcs://peer0.freight.example.com:9051 timeout:3000
2021-03-06T17:58:15.409Z - info: [NetworkConfig]: buildPeer - Unable to connect to the endorser peer0.freight.example.com due to Error: Failed to connect before the deadline on Endorser- name: peer0.freight.example.com, url:grpcs://peer0.freight.example.com:9051, connected:false, connectAttempted:true
2021-03-06T17:58:18.412Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer1.freight.example.com, url:grpcs://peer1.freight.example.com:10051, connected:false, connectAttempted:true
2021-03-06T17:58:18.412Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer1.freight.example.com url:grpcs://peer1.freight.example.com:10051 timeout:3000
2021-03-06T17:58:18.412Z - info: [NetworkConfig]: buildPeer - Unable to connect to the endorser peer1.freight.example.com due to Error: Failed to connect before the deadline on Endorser- name: peer1.freight.example.com, url:grpcs://peer1.freight.example.com:10051, connected:false, connectAttempted:true
2021-03-06T17:58:21.423Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: peer0.freight.example.com, url:grpcs://peer0.freight.example.com:9051, connected:false, connectAttempted:true
2021-03-06T17:58:21.423Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer0.freight.example.com url:grpcs://peer0.freight.example.com:9051 timeout:3000
2021-03-06T17:58:21.423Z - error: [ServiceEndpoint]: ServiceEndpoint grpcs://peer0.freight.example.com:9051 reset connection failed :: Error: Failed to connect before the deadline on Discoverer- name: peer0.freight.example.com, url:grpcs://peer0.freight.example.com:9051, connected:false, connectAttempted:true
2021-03-06T17:58:24.425Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: peer1.freight.example.com, url:grpcs://peer1.freight.example.com:10051, connected:false, connectAttempted:true
2021-03-06T17:58:24.425Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer1.freight.example.com url:grpcs://peer1.freight.example.com:10051 timeout:3000
2021-03-06T17:58:24.425Z - error: [ServiceEndpoint]: ServiceEndpoint grpcs://peer1.freight.example.com:10051 reset connection failed :: Error: Failed to connect before the deadline on Discoverer- name: peer1.freight.example.com, url:grpcs://peer1.freight.example.com:10051, connected:false, connectAttempted:true
2021-03-06T17:58:24.425Z - error: [DiscoveryService]: send[supplychain-channel] - no discovery results
Failed to evaluate transaction: function () Discovery Service has failed to return results

The network topology is 5 orgs with 2 orgs with 1 peer without orderer and 3 orgs with 2 peers each with orderer. Orderers are the part of peerOrganization itself. And each organization has its own CA. What can be the cause of this error?

Updated Discovery:

await gateway.connect(ccp, {
  wallet,
  identity: username,
  discovery: {
    enabled: true,
    asLocalhost: true
  },
  clientTlsIdentity: 'freight'
})

const network = await gateway.getNetwork('supplychain-channel')
console.log(network)

console:

Wallet Path: /home/hlfabric/fabric/api/freight/wallet
2021-03-10T04:54:26.827Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Committer- name: orderer0.freight.example.com:9050, url:grpcs://localhost:9050, connected:false, connectAttempted:true
2021-03-10T04:54:26.828Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server orderer0.freight.example.com:9050 url:grpcs://localhost:9050 timeout:3000
2021-03-10T04:54:26.828Z - error: [DiscoveryService]: _buildOrderer[supplychain-channel] - Unable to connect to the discovered orderer orderer0.freight.example.com:9050 due to Error: Failed to connect before the deadline on Committer- name: orderer0.freight.example.com:9050, url:grpcs://localhost:9050, connected:false, connectAttempted:true
2021-03-10T04:54:29.830Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Committer- name: orderer1.carrier.example.com:11050, url:grpcs://localhost:11050, connected:false, connectAttempted:true .....
NetworkImpl {
      contracts: Map {},
      initialized: true,
      commitListeners: Map {},
      blockListeners: Map {},
      gateway:
       Gateway {
         networks: Map { 'supplychain-channel' => [Circular] },
         options:
          { queryHandlerOptions: [Object],
            eventHandlerOptions: [Object],
            discovery: [Object],
            wallet: [Wallet],
            identity: 'sai',
            clientTlsIdentity: 'freight' },
         client:
          Client {
            type: 'Client',
            name: 'gateway client',
            mspid: null,
            _tls_mutual: [Object],
            endorsers: [Map],
            committers: [Map],
            channels: [Map],
            centralizedOptions: null },
         identity:
          { credentials: [Object], mspId: 'FreightMSP', type: 'X.509' },
         identityContext:
          IdentityContext {
            type: 'IdentityContext',
            client: [Client],
            user: [User],
            name: 'sai',
            mspid: 'FreightMSP',
            transactionId: null,
            nonce: null } },
      channel:
       Channel {
         type: 'Channel',
         name: 'supplychain-channel',
         client:
          Client {
            type: 'Client',
            name: 'gateway client',
            mspid: null,
            _tls_mutual: [Object],
            endorsers: [Map],
            committers: [Map],
            channels: [Map],
            centralizedOptions: null },
         endorsers:
          Map {
            'peer0.freight.example.com:9051' => [Endorser],
            'peer1.freight.example.com:10051' => [Endorser] },
         committers: Map { 'orderer0.freight.example.com:9050' => [Committer] },
         msps:
          Map {
            'FreightMSP' => [Object],
            'ConsigneeMSP' => [Object],
            'ShipperMSP' => [Object],
            'CarrierMSP' => [Object],
            'CustomMSP' => [Object] } },
      eventServiceManager:
       EventServiceManager {
         eventServices: Map {},
         network: [Circular],
         channel:
          Channel {
            type: 'Channel',
            name: 'supplychain-channel',
            client: [Client],
            endorsers: [Map],
            committers: [Map],
            msps: [Map] },
         mspId: 'FreightMSP',
         identityContext:
          IdentityContext {
            type: 'IdentityContext',
            client: [Client],
            user: [User],
            name: 'sai',
            mspid: 'FreightMSP',
            transactionId: null,
            nonce: null } },
      realtimeFilteredBlockEventSource:
       BlockEventSource {
         listeners: Set {},
         started: false,
         eventServiceManager:
          EventServiceManager {
            eventServices: Map {},
            network: [Circular],
            channel: [Channel],
            mspId: 'FreightMSP',
            identityContext: [IdentityContext] },
         blockQueue:
          OrderedBlockQueue { queue: Map {}, nextBlockNumber: undefined },
         asyncNotifier:
          AsyncNotifier {
            running: false,
            readCallback: [Function: bound getNextBlock],
            notifyCallback: [AsyncFunction: bound notifyListeners] },
         blockType: 'filtered' },
      realtimeFullBlockEventSource:
       BlockEventSource {
         listeners: Set {},
         started: false,
         eventServiceManager:
          EventServiceManager {
            eventServices: Map {},
            network: [Circular],
            channel: [Channel],
            mspId: 'FreightMSP',
            identityContext: [IdentityContext] },
         blockQueue:
          OrderedBlockQueue { queue: Map {}, nextBlockNumber: undefined },
         asyncNotifier:
          AsyncNotifier {
            running: false,
            readCallback: [Function: bound getNextBlock],
            notifyCallback: [AsyncFunction: bound notifyListeners] },
         blockType: 'full' },
      realtimePrivateBlockEventSource:
       BlockEventSource {
         listeners: Set {},
         started: false,
         eventServiceManager:
          EventServiceManager {
            eventServices: Map {},
            network: [Circular],
            channel: [Channel],
            mspId: 'FreightMSP',
            identityContext: [IdentityContext] },
         blockQueue:
          OrderedBlockQueue { queue: Map {}, nextBlockNumber: undefined },
         asyncNotifier:
          AsyncNotifier {
            running: false,
            readCallback: [Function: bound getNextBlock],
            notifyCallback: [AsyncFunction: bound notifyListeners] },
         blockType: 'private' },
      discoveryService:
       DiscoveryService {
         type: 'DiscoveryService',
         name: 'supplychain-channel',
         channel:
          Channel {
            type: 'Channel',
            name: 'supplychain-channel',
            client: [Client],
            endorsers: [Map],
            committers: [Map],
            msps: [Map] },
         client:
          Client {
            type: 'Client',
            name: 'gateway client',
            mspid: null,
            _tls_mutual: [Object],
            endorsers: [Map],
            committers: [Map],
            channels: [Map],
            centralizedOptions: null },
         refreshAge: undefined,
         discoveryResults:
          { msps: [Object],
            orderers: [Object],
            peers_by_org: [Object],
            timestamp: 1615352105872 },
         asLocalhost: true,
         currentTarget:
          Discoverer {
            name: 'peer0.freight.example.com',
            mspid: 'FreightMSP',
            client: [Client],
            connected: true,
            connectAttempted: true,
            endpoint: [Endpoint],
            service: [ServiceClientImpl],
            serviceClass: [Function],
            type: 'Discoverer',
            options: [Object] },
         targets: [ [Discoverer], [Discoverer] ],
         _action: { init: false, request: [Request] },
         _payload:
          <Buffer 0a c4 07 0a 9f 07 0a 0a 46 72 65 69 67 68 74 4d 53 50 12 90 07 2d 2d 2d 2d 2d 42 45 47 49 4e 20 43 45 52 54 49 46 49 43 41 54 45 2d 2d 2d 2d 2d 0a 4d ... >,
         _signature:
          <Buffer 30 44 02 20 18 dc 09 b5 2c 95 5d 98 14 c8 ee a0 47 48 f9 95 ae 74 1d b0 a9 d3 0e 21 cc dd 06 40 18 bc f4 f9 02 20 21 49 01 65 0b 2d 67 23 35 51 a8 dc ... >,
         requestTimeout: undefined },
      queryHandler:
       SingleQueryHandler { currentPeerIndex: 0, peers: [ [Endorser], [Endorser] ] } }

@GariSingh same on which Freight organization peers and orderer are running. — Saicharan Pogul, Mar 08 '21 at 15:41
So in a Docker container attached to the same Docker Swarm network? — Gari Singh, Mar 08 '21 at 15:53
@GariSingh So actually i have my complete network setup locally i mean as i said, i have 5 organizations hosted on 5 different VMs which is on host machine using Virtualbox. These 5 VMs are a part of docker swarm. The take here is that most of the examples including fabric docs and various medium articles have different orderer organization which doesn't make sense in terms of distributed network. So following to an Medium article by KC Tam i have 3 peer organizations hosting orderers on their machines. So all the docker containers i have started are using networks: ab: external: name: ab — Saicharan Pogul, Mar 08 '21 at 16:20
@GariSingh Yes all the docker containers are connected to the docker swarm overlay network. I tried with different grpc options but nothing is working. But i changed the above discovery object and added this `await gateway.connect(ccp, { wallet, identity: username, discovery: { enabled: true, asLocalhost: false }, clientTlsIdentity: 'tlsid' })` and I’m getting this now `Failed to evaluate transaction: function () { [native code] }` not sure where I’m making mistake. — Saicharan Pogul, Mar 09 '21 at 19:04
@GariSingh this is the error I’m getting actually i missed the error parameter for catch . `Failed to evaluate transaction: Error: DiscoveryService has failed to return results` — Saicharan Pogul, Mar 10 '21 at 04:12
@GariSingh Yes I have added env variable **CORE_PEER_GOSSIP_EXTERNALPOINT** for all peer services in docker compose files. I'm trying to run the Client from a docker container attached to same network maybe that will work because i can ping from one container to another. Maybe that will solve the issue. — Saicharan Pogul, Mar 10 '21 at 10:48
@GariSingh yes but node alpine doesn’t include python so I’m getting this errors `gyp ERR! find Python gyp ERR! find Python Python is not set from command line or npm configuration .... gyp ERR! find Python checking if "python3" can be used gyp ERR! find Python - "python3" is not in PATH or produced an error gyp ERR! find Python gyp ERR! find Python ********************************************************** gyp ERR! find Python You need to install the latest version of Python. gyp ERR! find Python Node-gyp should be able to find and use Python. If not,..` — Saicharan Pogul, Mar 10 '21 at 11:30

score 0 · Answer 1 · answered Mar 09 '21 at 10:14

DNS names ending example.com will not resolve unless you have entries for them in your local hosts file:

https://en.wikipedia.org/wiki/Example.com

The node names need to be resolvable and accessible by your client (and by other nodes since they will want to communicate with each other). If all of those addresses are mapped from private (e.g. Docker) networks to the same port number on your local host's network, you should be able to enable asLocalhost, which will simply cause the discovery service to treat and host names it receives as the localhost address (but leave the port number unchanged). Alternatively, you could make those addresses resolvable by either:

Giving them real public network addresses; or
Adding entries for them in your client's hosts file

score 0 · Accepted Answer · answered Mar 10 '21 at 13:21

Running client in docker environment within same network the fabric is running solves the above issue

Dockerfile

FROM node:10.24
WORKDIR /usr/src/app
COPY package*.json ./
RUN apt-get update || : && apt-get install python -y
RUN npm install
COPY . .
EXPOSE 9080
CMD [ "node", "server.js" ]

First RUN is important because node image doesn’t include python and fabric node sdk requires python for some modules like pkcs11, resolves errors gyp ERR! find Python checking if "python3" can be used occurred while installing dependencies.

docker-compose.yml

version: "2.1"

networks: 
  supplychain:
    external: 
      name: supplychain

services: 
  api:
    image: api:1.0
    build: 
      context: "client/"
    networks: 
      - supplychain
    ports: 
      - 9080:9080

setting asLocalhost: false and urls of peers to "url": "grpcs://peer0.freight.example.com:9051" the peers are discovered by the client.

await gateway.connect(ccp, {
      wallet,
      identity: username,
      discovery: {
        enabled: true,
        asLocalhost: false
      }
    })

Failed to connect before the deadline on Endorser

2 Answers2