How can I set up hyperledger fabric with multiple hosts using Docker?

Question

I work on the Hyperledger Fabric v1.0 and would like to make the Getting Setup work on multiple hosts. For now, 2 would be great.

Here is what I want to do:

Host1: start an orderer and 2 peers
Host2: start 1 peer
Host2: A client creates a channel (using the channel_test.sh updated with the good hosts IP) and join all the 3 peers
Host1: Call de deploy.js of the given example to deploy the chaincode

I have a problem on the 3rd step. I think the channel creation works but on my peers log I have the same warnings on the 3 peers:

Remote endpoint claims to be a different peer, expected [host1 IP:8051] but got [172.17.0.4:7051]
Failed obtaining connection for 172.31.9.126:8051, PKIid:[49 55 50 ...] reason: Authentication failure

It looks like they can't communicate with each other. Any idea where the problem is?

I still tried my step 4 but I can't deploy it unless I remove the host2: peer1 from the config.json. And even then, I can only query from the host1: peer0, not the host1: peer2.

Here are the commands I use to set up my network:

Host1: Orderer

docker run --rm -it --name orderer -p 8050:7050 
-e ORDERER_GENERAL_LEDGERTYPE=ram 
-e ORDERER_GENERAL_BATCHTIMEOUT=10s 
-e ORDERER_GENERAL_BATCHSIZE_MAXMESSAGECOUNT=10 
-e ORDERER_GENERAL_MAXWINDOWSIZE=1000 
-e ORDERER_GENERAL_ORDERERTYPE=solo 
-e ORDERER_GENERAL_LOGLEVEL=debug 
-e ORDERER_GENERAL_LISTENADDRESS=0.0.0.0 
-e ORDERER_GENERAL_LISTENPORT=7050 
-e ORDERER_RAMLEDGER_HISTORY_SIZE=100 
sfhackfest22017/fabric-orderer:x86_64-0.7.0-snapshot-c7b3fe0 orderer

Host1: Peer0

docker run --rm -it --name peer0 -p 8051:7051 -p 8053:7053
-v /var/run/:/host/var/run/ -v $BASE_DIR/tmp/peer0:/etc/hyperledger/fabric/msp/sampleconfig 
-e CORE_PEER_ADDRESSAUTODETECT=true 
-e CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock 
-e CORE_LOGGING_LEVEL=DEBUG 
-e CORE_PEER_NETWORKID=peer0 
-e CORE_NEXT=true 
-e CORE_PEER_ENDORSER_ENABLED=true 
-e CORE_PEER_ID=peer0 
-e CORE_PEER_PROFILE_ENABLED=true 
-e CORE_PEER_COMMITTER_LEDGER_ORDERER=$ORDERER_IP:7050 
-e CORE_PEER_GOSSIP_ORGLEADER=true 
-e CORE_PEER_GOSSIP_IGNORESECURITY=true 
sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 peer node start --peer-defaultchain=false

Host1: Peer2

docker run --rm -it --name peer2 -p 8055:7051 -p 8057:7053 
-v /var/run/:/host/var/run/ -v $BASE_DIR/tmp/peer0:/etc/hyperledger/fabric/msp/sampleconfig
-e CORE_PEER_ID=peer2 
[Other parameters are the same as Peer0]
sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 peer node start --peer-defaultchain=false

Host2: Peer1

docker run --rm -it --name peer1 -p 8051:7051 
-v /var/run/:/host/var/run/ -v $BASE_DIR/tmp/peer0:/etc/hyperledger/fabric/msp/sampleconfig
-e CORE_PEER_ID=peer1 
[Other parameters are the same as Peer0]
sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 peer node start --peer-defaultchain=false

Host2: Cli

docker run --rm -it --name cli
    -v /var/run/:/host/var/run/ -v $BASE_DIR/tmp/peer3:/etc/hyperledger/fabric/msp/sampleconfig -v $BASE_DIR/src/github.com/example_cc/example_cc.go:/opt/gopath/src/github.com/hyperledger/fabric/examples/example_cc.go -v $BASE_DIR/channel_test.sh:/opt/gopath/src/github.com/hyperledger/fabric/peer/channel_test.sh
    --workdir /opt/gopath/src/github.com/hyperledger/fabric/peer  
    -e GOPATH=/opt/gopath 
    -e CORE_PEER_ADDRESSAUTODETECT=true
    -e CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock 
    -e CORE_LOGGING_LEVEL=DEBUG 
    -e CORE_NEXT=true 
    -e CORE_PEER_ID=cli 
    -e CORE_PEER_ENDORSER_ENABLED=true 
    -e CORE_PEER_COMMITTER_LEDGER_ORDERER=$ORDERER_IP:8050 
    -e CORE_PEER_ADDRESS=$PEER0_IP:8051 
    sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 ./channel_test.sh

If you need more information feel free to ask.

Note: I'm not very familiar with docker, any improvement/advice on how I use it is welcome :)

Can you explain the purpose of the `CORE_NEXT=true` variable? — PassKit, Mar 30 '17 at 12:47
I tried to do the same as the [docker-compose-gettingstarted.yml](https://github.com/hyperledger/fabric/blob/master/examples/sfhackfest/docker-compose-gettingstarted.yml). as for the purpose of the environment variable, I couldn't say. Do you know where I can find a documentation ? — ecn, Mar 30 '17 at 12:57
Looks like your issue is with port mappings - If the hosts are on different IPs then there is no need to move the ports. Change `-p 805x:7051 -p 805x:7053` to `-p 7051:7051 -p 7053:7053` across all peers — PassKit, Mar 30 '17 at 13:09
The message is now the same but with the ports changed : Exp: 172.3X.X.XXX:7051 - Got: 172.17.0.3:7051 (The first ip is the machine's, second one is the local docker's ip). But thanks for the port advice — ecn, Mar 30 '17 at 13:28
I think that docker containers should have different port numbers, :-/ — Mr. Pundir, Apr 03 '17 at 17:22

score 15 · Accepted Answer · answered Apr 11 '17 at 14:27

I found a solution that seems to work using docker swarm mode.

Initialize a swarm: (docker swarm documentation for mor information)
Join the swarm with the other host as a manager
Create a network ("hyp-net" in my case)

docker network create --attachable --driver overlay hyp-net

Changes I had to do:

Linked the containers with the --link docker parameter
Added the --network docker parameter (--network=hyp-net)
Added a new environment varialble todocker run command used:
```
-e CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=hyp-net
```

Here are the commands that works for me:

Orderer

docker run --rm -it --network="hyp-net" --name orderer -p 8050:7050 
-e ORDERER_GENERAL_LEDGERTYPE=ram 
-e ORDERER_GENERAL_BATCHTIMEOUT=10s 
-e ORDERER_GENERAL_BATCHSIZE_MAXMESSAGECOUNT=10 
-e ORDERER_GENERAL_MAXWINDOWSIZE=1000 
-e ORDERER_GENERAL_ORDERERTYPE=solo 
-e ORDERER_GENERAL_LOGLEVEL=debug 
-e ORDERER_GENERAL_LISTENADDRESS=0.0.0.0 
-e ORDERER_GENERAL_LISTENPORT=7050 
-e ORDERER_RAMLEDGER_HISTORY_SIZE=100 
-e CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=hyp-net 
sfhackfest22017/fabric-orderer:x86_64-0.7.0-snapshot-c7b3fe0 orderer

Peer0

docker run --rm -it --link orderer:orderer --network="hyp-net" --name peer0 -p 8051:7051 -p 8053:7053 
-v /var/run/:/host/var/run/ -v $BASE_DIR/tmp/peer0:/etc/hyperledger/fabric/msp/sampleconfig  
-e CORE_PEER_ADDRESSAUTODETECT=true 
-e CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock 
-e CORE_LOGGING_LEVEL=DEBUG 
-e CORE_PEER_NETWORKID=peer0 
-e CORE_NEXT=true 
-e CORE_PEER_ENDORSER_ENABLED=true 
-e CORE_PEER_ID=peer0 
-e CORE_PEER_PROFILE_ENABLED=true
-e CORE_PEER_COMMITTER_LEDGER_ORDERER=orderer:7050 
-e CORE_PEER_GOSSIP_ORGLEADER=true 
-e CORE_PEER_GOSSIP_IGNORESECURITY=true 
-e CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=hyp-net 
sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 peer node start --peer-defaultchain=false

Peer1

docker run --rm -it --network="hyp-net" --link orderer:orderer --link peer0:peer0 [...] -e CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=hyp-net sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 peer node start --peer-defaultchain=false

Peer2

docker run --rm -it --network="hyp-net" --link orderer:orderer --link peer0:peer0 --link peer1:peer1 [...] -e CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=hyp-net sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 peer node start --peer-defaultchain=false

Cli

docker run --rm -it --network="hyp-net" --link orderer:orderer --link peer0:peer0 --link peer1:peer1 --link peer2:peer2 [...] -e CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=hyp-net sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 ./channel_test.sh

With this, I am able to deploy, invoke and query my chaincode.

Awesome! Your post is helpful!!! But i have a question. Does this example work for v1.0 too? Looks like the original url, http://hyperledger-fabric.readthedocs.io/en/latest/asset_setup.html, use v0.6 because peers don't have install and instantiate commands... — Ko Ohhashi, Apr 12 '17 at 00:28
I don't know for the latest version but I planned to try it. I will update this post if I have any luck — ecn, Apr 12 '17 at 06:50
@KoOhhashi I tried the [getting started](http://hyperledger-fabric.readthedocs.io/en/latest/getting_started.html) of the latest version on two hosts and it works — ecn, Apr 13 '17 at 12:12
Hi Erik! I am facing the same question. Now the swarm environment has been built, but all the fabric nodes run with an panic error. root@25f4f7531d54:/usr/local/bin# peer node start panic: Fatal error when initializing core config : Error when reading core config file: Unsupported Config Type "" goroutine 1 [running]: panic(0xc82b40, 0xc420187c10) /opt/go/src/runtime/panic.go:500 +0x1a1 main.main() /opt/gopath/src/github.com/hyperledger/fabric/peer/main.go:94 +0x7c2 I built the docker images using docker export and import the images onto two hosts. — allenchen, Jun 18 '17 at 07:41
@allenchen Seems you didn't set FABRIC_CFG_PATH properly.. See the answer for your question in trouble shooting guide. https://github.com/hyperledger/fabric/blob/master/examples/e2e_cli/end-to-end.rst#troubleshooting.. — rameshthoomu, Jun 28 '17 at 18:19
@ecn It'd be great if you posted a tutorial on how you did this somewhere... it seems that some of the details on how this process works were lost in this answer on SO and the "getting started" leaves a bit to be desired at the moment as well. — Cleanshooter, Jul 24 '17 at 15:10
@ecn the possible values of CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE are `host`(default),`bridge`,`ipvlan`,`none` according to documentation. Are you sure that CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE environment variable has any affect here? — Şeref Acet, Jun 21 '18 at 07:02

score 5 · Answer 2 · answered Feb 14 '18 at 10:36

I was able to host hyperledger fabric network on multiple machines using docker swarm mode. Swarm mode provides a network across multiple hosts/machines for the communication of the fabric network components.

This post explains the deployment process.It creates a swarm network and all the other machines join the network. https://medium.com/@wahabjawed/hyperledger-fabric-on-multiple-hosts-a33b08ef24f

It works with Fabric 1.0+

score 0 · Answer 3 · answered Jul 01 '20 at 12:57

Check with the server names in fabric-samples\first-network\connection-org*.yaml and fabric-samples\first-network\connection-org*.json. These would be templated and generated from ccp-template.json and ccp-template.yaml.

Also, have entries for peers in fabric-samples\first-network\crypto-config.yaml under 'Specs'.

How can I set up hyperledger fabric with multiple hosts using Docker?

3 Answers3

Linked