MinIO cluster on different hosts by Docker

Question

Many post, forums and video has been checked, but I didn't see any solution.
I would like to build a MinIO Cluster with 4 different hosts with docker-compose.
Is there any solution?

Recent error message:

API: SYSTEM()
Time: 13:26:13 UTC 11/23/2021
Error: Read failed. Insufficient number of disks online (*errors.errorString)
       5: cmd/prepare-storage.go:268:cmd.connectLoadInitFormats()
       4: cmd/prepare-storage.go:317:cmd.waitForFormatErasure()
       3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
       2: cmd/server-main.go:637:cmd.newObjectLayer()
       1: cmd/server-main.go:542:cmd.serverMain()
Waiting for a minimum of 2 disks to come online (elapsed 1m4s)

Unable to read 'format.json' from http://10.10.10.1:9000/data: Server expects 'storage' API version 'v41', instead found 'v41' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (DEVELOPMENT.2021-11-09T03-21-45Z)

Unable to read 'format.json' from http://10.10.10.2:9000/data: Server expects 'storage' API version 'v41', instead found 'v41' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (DEVELOPMENT.2021-11-09T03-21-45Z)

Unable to read 'format.json' from http://10.10.10.3:9000/data: Server expects 'storage' API version 'v41', instead found 'v41' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (DEVELOPMENT.2021-11-09T03-21-45Z)

Unable to read 'format.json' from http://10.10.10.4:9000/data: Server expects 'storage' API version 'v41', instead found 'v41' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (DEVELOPMENT.2021-11-09T03-21-45Z)

docker-compose.yml

version: '2'

services:
  minio1:
    image: 'bitnami/minio:latest'
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      - MINIO_ACCESS_KEY=user
      - MINIO_SECRET_KEY=pass123
      - MINIO_DISTRIBUTED_MODE_ENABLED=yes
      - MINIO_DISTRIBUTED_NODES=10.10.10.1,10.10.10.2,10.10.10.3,10.10.10.4
      - MINIO_SKIP_CLIENT=yes
    volumes:
      - data1-1:/data1
      - data1-2:/data2
volumes:
  data1-1:
  data1-2:

networks:
  custom:
    driver: bridge

zeerk · Accepted Answer · 2021-11-24T14:18:14.133

I found the solution.
You can use it on different nodes.
https://github.com/berserkbuddhist/minio/blob/main/docker-compose.yml

version: '3.7'

services:
  minio1:                       # rename on different nodes
    hostname: minio1            # rename on different nodes
    image: quay.io/minio/minio:RELEASE.2021-11-09T03-21-45Z
    command: server --console-address ":9001" http://minio{1...3}/data{1...2}
    ports:
      - "9000:9000"
      - "9001:9001"
    extra_hosts:
      - "minio1:10.10.10.1"  # 1st node
      - "minio2:10.10.10.2"  # 2nd node
      - "minio3:10.10.10.3"  # 3rd node
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
    volumes:
      - data1-1:/data1          # rename on different nodes
      - data1-2:/data2          # rename on different nodes
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

volumes:
  data1-1:                      # rename on different nodes
  data1-2:                      # rename on different nodes

score 0 · Answer 2 · answered May 19 '22 at 23:07

This works for me (4 nodes cluster on 4 servers)

version: '3.7'

services:
  minio1: # rename on different nodes
    restart: always
    image: 'bitnami/minio:latest'
    container_name: minio1 # rename on different nodes
    hostname: minio1 # rename on different nodes
    ports:
      - '9000:9000'
      - '9001:9001'
    environment:
      - MINIO_ROOT_USER=minioadmin
      - MINIO_ROOT_PASSWORD=minioadmin
      - MINIO_DISTRIBUTED_MODE_ENABLED=yes
      - MINIO_DISTRIBUTED_NODES=minio1,minio2,minio3,minio4
      - MINIO_SKIP_CLIENT=yes
    extra_hosts:
      - "minio1:10.10.10.1"  # 1st node
      - "minio2:10.10.10.2"  # 2nd node
      - "minio3:10.10.10.3"  # 3rd node
      - "minio4:10.10.10.4"  # 4th node
    volumes:
      - /opt/docker/minio/data:/data
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
      interval: 30s
      timeout: 20s
      retries: 3

MinIO cluster on different hosts by Docker

2 Answers2