Hyperledger Blockchain Explorer-Fail to connect before the deadline on Endorser, fail to connect to remote gRPC server

Question

I am trying to set up a Hyperledger Fabric Network with Hyperledger Explorer. I spin up a VM on the digital ocean cloud with ubuntu OS. From there, I spin up 3 orderers node, and 2 peers node. Which result in total of 5 nodes. (I am using RAFT setup).

However, I encounter the error as below when trying to start the hyperledger fabric explorer docker-container images.

Error:

> hyperledger-explorer@1.1.4 app-start /opt/explorer
> ./start.sh

[2021-02-18T07:32:21.828] [INFO] PgService - SSL to Postgresql disabled
[2021-02-18T07:32:21.829] [INFO] PgService - connecting to Postgresql postgres://hppoc:******@explorerdb.mynetwork.com:5432/fabricexplorer
[2021-02-18T07:32:21.898] [INFO] Platform -  network_config.id  test-network  network_config.profile  ./connection-profile/test-network.json
[2021-02-18T07:32:22.013] [INFO] FabricConfig - config.client.tlsEnable  true
[2021-02-18T07:32:22.013] [INFO] FabricConfig - FabricConfig, this.config.channels  airlinechannel
[2021-02-18T07:32:22.016] [INFO] FabricGateway - enrollUserIdentity: userName : exploreradmin
2021-02-18T07:32:25.221Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer1.acme.com, url:grpcs://peer1.acme.com:7051, connected:false, connectAttempted:true
2021-02-18T07:32:25.222Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer1.acme.com url:grpcs://peer1.acme.com:7051 timeout:3000
2021-02-18T07:32:25.223Z - info: [NetworkConfig]: buildPeer - Unable to connect to the endorser peer1.acme.com due to Error: Failed to connect before the deadline on Endorser- name: peer1.acme.com, url:grpcs://peer1.acme.com:7051, connected:false, connectAttempted:true
2021-02-18T07:32:28.250Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: peer1.acme.com, url:grpcs://peer1.acme.com:7051, connected:false, connectAttempted:true
2021-02-18T07:32:28.250Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer1.acme.com url:grpcs://peer1.acme.com:7051 timeout:3000
2021-02-18T07:32:28.250Z - error: [ServiceEndpoint]: ServiceEndpoint grpcs://peer1.acme.com:7051 reset connection failed :: Error: Failed to connect before the deadline on Discoverer- name: peer1.acme.com, url:grpcs://peer1.acme.com:7051, connected:false, connectAttempted:true
2021-02-18T07:32:28.251Z - error: [DiscoveryService]: send[airlinechannel] - no discovery results
[2021-02-18T07:32:28.251] [ERROR] FabricClient - Error: DiscoveryService has failed to return results
    at DiscoveryService.send (/opt/explorer/node_modules/fabric-network/node_modules/fabric-common/lib/DiscoveryService.js:370:10)
    at process._tickCallback (internal/process/next_tick.js:68:7)
[2021-02-18T07:32:28.252] [INFO] FabricClient - ********* call to initializeDetachClient **********
[2021-02-18T07:32:28.253] [INFO] FabricClient - initializeDetachClient, network config)  { name: 'test-network',
  version: '1.0.0',
  client:
   { tlsEnable: true,
     adminCredential: { id: 'exploreradmin', password: 'exploreradminpw' },
     enableAuthentication: true,
     organization: 'AcmeMSP',
     connection: { timeout: [Object] } },
  channels: { airlinechannel: { peers: [Object] } },
  organizations:
   { AcmeMSP:
      { mspid: 'AcmeMSP',
        adminPrivateKey: [Object],
        peers: [Array],
        signedCert: [Object] } },
  peers:
   { 'peer1.acme.com': { tlsCACerts: [Object], url: 'grpcs://peer1.acme.com:7051' } } }
[2021-02-18T07:32:28.253] [INFO] FabricClient - ************************************* initializeDetachClient *************************************************
[2021-02-18T07:32:28.254] [INFO] FabricClient - Error : Failed to connect client peer, please check the configuration and peer status
[2021-02-18T07:32:28.254] [INFO] FabricClient - Info :  Explorer will continue working with only DB data
[2021-02-18T07:32:28.254] [INFO] FabricClient - ************************************** initializeDetachClient ************************************************
[2021-02-18T07:32:28.259] [INFO] Platform - initializeListener, network_id, network_client  test-network { name: 'test-network',
  version: '1.0.0',
  client:
   { tlsEnable: true,
     adminCredential: { id: 'exploreradmin', password: 'exploreradminpw' },
     enableAuthentication: true,
     organization: 'AcmeMSP',
     connection: { timeout: [Object] } },
  channels: { airlinechannel: { peers: [Object] } },
  organizations:
   { AcmeMSP:
      { mspid: 'AcmeMSP',
        adminPrivateKey: [Object],
        peers: [Array],
        signedCert: [Object] } },
  peers:
   { 'peer1.acme.com': { tlsCACerts: [Object], url: 'grpcs://peer1.acme.com:7051' } } }
[2021-02-18T07:32:28.260] [INFO] main - Please open web browser to access ：http://localhost:8080/
[2021-02-18T07:32:28.261] [INFO] main - pid is 20
[2021-02-18T07:32:28.263] [ERROR] main - <<<<<<<<<<<<<<<<<<<<<<<<<< Explorer Error >>>>>>>>>>>>>>>>>>>>>
[2021-02-18T07:32:28.263] [ERROR] main - Error :  [ 'Default client peer is down and no channel details available database' ]
[2021-02-18T07:32:30.264] [INFO] main - Received kill signal, shutting down gracefully
[2021-02-18T07:32:30.266] [INFO] Platform - <<<<<<<<<<<<<<<<<<<<<<<<<< Closing explorer  >>>>>>>>>>>>>>>>>>>>>
[2021-02-18T07:32:30.266] [INFO] main - Closed out connections

Version Detail

Hyperledger Fabric: 2.3.1
Hyperledger Explorer: v1.1.1 (latest tag)

Part 1: Docker Container Setup

a) Docker PS

CONTAINER ID   IMAGE                                                                                                                                                                  COMMAND                  CREATED          STATUS                   PORTS                                            NAMES
9e8768914140   hyperledger/explorer:latest                                                                                                                                            "docker-entrypoint.s…"   3 minutes ago    Up 3 minutes             0.0.0.0:8080->8080/tcp                           explorer.mynetwork.com
903c8c4a4326   hyperledger/explorer-db:latest                                                                                                                                         "docker-entrypoint.s…"   3 minutes ago    Up 3 minutes (healthy)   5432/tcp                                         explorerdb.mynetwork.com
9fed267ae9b1   dev-peer1.budget.com-gocc1.1.0-1.0-2593f1a95def85c64fdfed35e3d3b1051b92ed91549cfe789292ec5475d7db86-e014c6d933da036b6c79b53d29a800d8a6886e374ceb435a30642b885735f8f5   "chaincode -peer.add…"   21 minutes ago   Up 21 minutes                                                             dev-peer1.budget.com-gocc1.1.0-1.0-2593f1a95def85c64fdfed35e3d3b1051b92ed91549cfe789292ec5475d7db86
12407497fa4c   dev-peer1.acme.com-gocc1.1.0-1.0-2593f1a95def85c64fdfed35e3d3b1051b92ed91549cfe789292ec5475d7db86-a3b6caa9293bb826d231b1a31cc47437e58764abe5169a20ed0ee43f25c16b24     "chaincode -peer.add…"   21 minutes ago   Up 21 minutes                                                             dev-peer1.acme.com-gocc1.1.0-1.0-2593f1a95def85c64fdfed35e3d3b1051b92ed91549cfe789292ec5475d7db86
0448e71f68e1   hyperledger/fabric-peer:latest                                                                                                                                         "peer node start"        22 minutes ago   Up 22 minutes            0.0.0.0:8051->7051/tcp, 0.0.0.0:8052->7052/tcp   peer1.budget.com
3266ec37b360   hyperledger/fabric-peer:latest                                                                                                                                         "peer node start"        22 minutes ago   Up 22 minutes            0.0.0.0:7051-7052->7051-7052/tcp                 peer1.acme.com
47ebe9ad79d1   hyperledger/fabric-orderer:latest                                                                                                                                      "orderer"                22 minutes ago   Up 22 minutes            0.0.0.0:8050->7050/tcp                           orderer2.acme.com
09a5f771f47f   hyperledger/fabric-tools:latest                                                                                                                                        "/bin/bash"              22 minutes ago   Up 22 minutes                                                             tools
e132bb01ce22   hyperledger/fabric-orderer:latest                                                                                                                                      "orderer"                22 minutes ago   Up 22 minutes            0.0.0.0:9050->7050/tcp                           orderer3.acme.com
3c61b0316385   hyperledger/fabric-orderer:latest                                                                                                                                      "orderer"                22 minutes ago   Up 22 minutes

b) I use 3 docker-compose files for my configuration setting.

$ docker-compose -f ./config/docker-compose-base.yaml -f ./tls/docker-compose-tls.yaml -f ./raft/docker-compose-raft.yaml up -d

docker-compose-base.yaml

https://gist.github.com/Skyquek/03d1ffad5643d67d8da5b268a4814a7d

docker-compose-tls.yaml

https://gist.github.com/Skyquek/b3b314cb2152ab541e822f72c60a2cbd

docker-compose-raft.yaml

https://gist.github.com/Skyquek/7f8ec2d4d1876283f4a9444675971be8

c) Core.yaml

acme core.yaml

https://gist.github.com/Skyquek/8cdcbc4ee3d53a2277b1c34bb2fca704

Part 2: Blockchain Explorer Setting

1. connection-profile.json

{
        "name": "test-network",
        "version": "1.0.0",
        "client": {
                "tlsEnable": true,
                "adminCredential": {
                        "id": "exploreradmin",
                        "password": "exploreradminpw"
                },
                "enableAuthentication": true,
                "organization": "AcmeMSP",
                "connection": {
                        "timeout": {
                                "peer": {
                                        "endorser": "300"
                                },
                                "orderer": "300"
                        }
                }
        },
        "channels": {
                "airlinechannel": {
                        "peers": {
                                "peer1.acme.com": {}
                        }
                }
        },
        "organizations": {
                "AcmeMSP": {
                        "mspid": "AcmeMSP",
                        "adminPrivateKey": {
                                "path": "/tmp/crypto/peerOrganizations/acme.com/users/Admin@acme.com/msp/keystore/priv_sk"
                        },
                        "peers": ["peer1.acme.com"],
                        "signedCert": {
                                "path": "/tmp/crypto/peerOrganizations/acme.com/users/Admin@acme.com/msp/signcerts/Admin@acme.com-cert.pem"
                        }
                }
        },
        "peers": {
                "peer1.acme.com": {
                        "tlsCACerts": {
                                "path": "/tmp/crypto/peerOrganizations/acme.com/tlsca/tlsca.acme.com-cert.pem"
                        },
                        "url": "grpcs://peer1.acme.com:7051"
                }
        }
}

2. docker-compose.yaml


# SPDX-License-Identifier: Apache-2.0
version: '2.1'

volumes:
  pgdata:
  walletstore:

networks:
  mynetwork.com:
    external:
      name: acloudfan_airline

services:

  explorerdb.mynetwork.com:
    image: hyperledger/explorer-db:latest
    container_name: explorerdb.mynetwork.com
    hostname: explorerdb.mynetwork.com
    environment:
      - DATABASE_DATABASE=fabricexplorer
      - DATABASE_USERNAME=hppoc
      - DATABASE_PASSWORD=password
    healthcheck:
      test: "pg_isready -h localhost -p 5432 -q -U postgres"
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
      - pgdata:/var/lib/postgresql/data
    networks:
      - mynetwork.com

  explorer.mynetwork.com:
    image: hyperledger/explorer:latest
    container_name: explorer.mynetwork.com
    hostname: explorer.mynetwork.com
    environment:
      - DATABASE_HOST=explorerdb.mynetwork.com
      - DATABASE_DATABASE=fabricexplorer
      - DATABASE_USERNAME=hppoc
      - DATABASE_PASSWD=password
      - LOG_LEVEL_APP=debug
      - LOG_LEVEL_DB=debug
      - LOG_LEVEL_CONSOLE=info
      - LOG_CONSOLE_STDOUT=true
      - DISCOVERY_AS_LOCALHOST=true
    volumes:
      - ./config.json:/opt/explorer/app/platform/fabric/config.json
      - ./connection-profile:/opt/explorer/app/platform/fabric/connection-profile
      - ./organizations:/tmp/crypto
      - walletstore:/opt/explorer/wallet
    ports:
      - 8080:8080
    depends_on:
      explorerdb.mynetwork.com:
        condition: service_healthy
    networks:
      - mynetwork.com

Solution that I tried

1. Change the tlscs certs path

As stated in Hyperledger Fabric 2.0.1: Error: Failed to connect before the deadline on Discoverer- name:

The problem most likely is due to the error from the path. But the problem still persists.

2. Change the env variable DISCOVERY_AS_LOCALHOST=true to false

Some of them mention that this will fixed the problem. But I can't seem to fix it this way.

3. Tried with hyperledger fabric 2.0 test-network

I tried to run with fabric sample test-network and its run perfectly fine.

4. docker exec -it sh to the explorer.mynetwork.com to ping the peer

The ping can run perfectly fine.

/opt/explorer # ping peer1.acme.com:7051
PING peer1.acme.com:7051 (172.23.0.6): 56 data bytes
64 bytes from 172.23.0.6: seq=0 ttl=64 time=0.138 ms
64 bytes from 172.23.0.6: seq=1 ttl=64 time=0.087 ms
64 bytes from 172.23.0.6: seq=2 ttl=64 time=0.090 ms
64 bytes from 172.23.0.6: seq=3 ttl=64 time=0.089 ms
64 bytes from 172.23.0.6: seq=4 ttl=64 time=0.101 ms
64 bytes from 172.23.0.6: seq=5 ttl=64 time=0.088 ms
^C
--- peer1.acme.com:7051 ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 0.087/0.098/0.138 ms

Its keep on showing that the peer is down or the connection to the peer is fail

I am struggling with this error for few days now. Hope someone can help me to identify the problem. Thank you very much.

score 4 · Accepted Answer · edited Mar 25 '21 at 23:28

4

I think you should double-check your network. Explorer should be spun up on the same network with Fabric, so that the peers and other nodes can be connected. To check the Fabric network name, you should check in the docker-compose file which set it up and look for CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE or you can navigate to you CLI and run docker network ls. You should be able to find your fabric network name with the DRIVER which should be bridge

Also, you should make sure that your Fabric network is up and running properly before bringing Explorer up.

edited Mar 25 '21 at 23:28

Dharman

30,962
25
85
135

answered Mar 25 '21 at 23:22

jasonph

134
5

Exactly. My problem was they were on separate networks. Make sure your explorer runs on the same network as fabric. – Navid K May 10 '21 at 05:43
Yes, this solved my problem. I had to make sure all of the container runs in the same network. – Quek Yao Jing May 27 '21 at 13:47

jkmartin · Answer 2 · 2022-04-26T00:09:53.397

With a similar error, the problem I had was that the keys and certificates for the network had gotten out of sync with the keys and certs for the blockchain explorer. This is a particularly significant problem when running a test network, because the keys and certs will (AFAICT) be regenerated whenever you restart a network, and a test network restarts much more often than a production network.

Copy over everything into the organizations directory, or whatever you call it; repeat this every time you bring up the network you want to use the explorer on. Symlinks also work, though that is probably more brittle and I wouldn't want to rely on it in production.

score 0 · Answer 3 · answered Feb 20 '21 at 23:54

All configurations seems good, however you have to upgrade explorer version to be compatible with hyperledger fabric version.

So please use v1.1.4 instead of v1.1.1

Also make sure that you have mounted crypto config correctly, try to access this path inside the container /tmp/crypto/peerOrganizations/acme.com/tlsca/tlsca.acme.com-cert.pem

Try to change tlsCACerts path to use peer tls ca.crt /tmp/crypto/peerOrganizations/acme.com/peers/peer1.acme.com/tls/ca.crt

You have mentioned that the same configurations works with hyperledger fabric v2, if you have tried it locally not on the same server so I please disable the firewall on the server and give it a try

To check if you can reach domain and port please try this

cat > /dev/tcp/peer1.acme.com/7051

Check this https://support.bluemedora.com/s/article/Using-Bash-to-test-if-a-TCP-port-on-a-remote-system-is-open

score 0 · Answer 4 · answered Jun 06 '21 at 00:15

I've had this error

[INFO] FabricClient - Error : Failed to connect client peer, please check the configuration and peer status
...
Error :  [ 'Default client peer is down and no channel details available database' ]

What I found was that when we use 256-bit vs 384-bit key length, the error occurs.

Setting local/host made no difference.

I'm using v1.1.4. Will likely test the same with v1.1.5 soon enough and suspect it'll have the same issue.

score 0 · Answer 5 · answered Aug 18 '23 at 10:52

This error may also be caused by a wrong setting of the channel name. By default, the channel name is mychannel, that is the default channel name of the Test Network. If you named the channel differently, then you have to modify the content of the test-network.json file accordingly.