Distributed MinIO Self-Contained Example Fails: "Server expects 'storage' API version 'v40', instead found 'v40'"

Question

I'm trying to create a self-contained example of using MinIO in a distributed sense and I'm running into errors which I cannot find any solutions for.

I also can't find any self-contained example of how to use MinIO in the distributed mode with both the nodes and the server, so I don't actually know I'm doing it right.

Here's a docker compose file:

version: "3.9"

x-minio-common: &minio-common
    image: "minio/minio:RELEASE.2021-10-13T00-23-17Z"
    command: server --console-address ":9001" /data/disk1
    environment:
        MINIO_ROOT_USER: minioadmin
        MINIO_ROOT_PASSWORD: "this is a strong password"
    networks:
        -   "minio-net"
    healthcheck:
        test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
        interval: 30s
        timeout: 20s
        retries: 3
x-minio-server-common: &minio-server-common
    image: "minio/minio:RELEASE.2021-10-13T00-23-17Z"
    command: server --console-address ":9001" "http://store-{1...4}:9000"
    environment:
        MINIO_ROOT_USER: minioadmin
        MINIO_ROOT_PASSWORD: "this is a strong password"
    networks:
        -   "minio-net"
    healthcheck:
        test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
        interval: 30s
        timeout: 20s
        retries: 3

services:
    store-1:
        <<: *minio-common
        volumes:
            -   type:   "bind"
                source: 'C:\Users\MyUser\Documents\Docker-Share\minio-1'
                target: "/data/disk1"
        ports:
            -   "9005:9001"
            -   "9025:9000"
    store-2:
        <<: *minio-common
        volumes:
            -   type:   "bind"
                source: 'C:\Users\MyUser\Documents\Docker-Share\minio-2'
                target: "/data/disk1"
        ports:
            -   "9006:9001"
            -   "9026:9000"
    store-3:
        <<: *minio-common
        volumes:
            -   type:   "bind"
                source: 'C:\Users\MyUser\Documents\Docker-Share\minio-3'
                target: "/data/disk1"
        ports:
            -   "9007:9001"
            -   "9027:9000"
    store-4:
        <<: *minio-common
        volumes:
            -   type:   "bind"
                source: 'C:\Users\MyUser\Documents\Docker-Share\minio-4'
                target: "/data/disk1"
        ports:
            -   "9008:9001"
            -   "9028:9000"
    server-1:
        <<: *minio-server-common
        ports:
            -   "9010:9001"
        depends_on:
            -   store-1
            -   store-2
            -   store-3
            -   store-4

networks:
    minio-net:
        driver: "bridge"

And here's the output I get when I call docker compose up:

Container test-compose-store-3-1  Created
Container test-compose-store-2-1  Created
Container test-compose-store-4-1  Created
Container test-compose-store-1-1  Created
Container test-compose-server-1-1  Created
Attaching to test-compose-server-1-1, test-compose-store-1-1, test-compose-store-2-1, test-compose-store-3-1, test-compose-store-4-1
test-compose-store-1-1   | API: http://172.19.0.2:9000  http://127.0.0.1:9000
test-compose-store-1-1   |
test-compose-store-1-1   | Console: http://172.19.0.2:9001 http://127.0.0.1:9001
test-compose-store-1-1   |
test-compose-store-1-1   | Documentation: https://docs.min.io
test-compose-store-3-1   | API: http://172.19.0.3:9000  http://127.0.0.1:9000
test-compose-store-3-1   |
test-compose-store-3-1   | Console: http://172.19.0.3:9001 http://127.0.0.1:9001
test-compose-store-3-1   |
test-compose-store-3-1   | Documentation: https://docs.min.io
test-compose-store-2-1   | API: http://172.19.0.4:9000  http://127.0.0.1:9000
test-compose-store-2-1   |
test-compose-store-2-1   | Console: http://172.19.0.4:9001 http://127.0.0.1:9001
test-compose-store-2-1   |
test-compose-store-2-1   | Documentation: https://docs.min.io
test-compose-store-4-1   | API: http://172.19.0.5:9000  http://127.0.0.1:9000
test-compose-store-4-1   |
test-compose-store-4-1   | Console: http://172.19.0.5:9001 http://127.0.0.1:9001
test-compose-store-4-1   |
test-compose-store-4-1   | Documentation: https://docs.min.io
test-compose-server-1-1  |
test-compose-server-1-1  | API: SYSTEM()
test-compose-server-1-1  | Time: 21:53:32 UTC 04/11/2022
test-compose-server-1-1  | Error: WARNING: Expected number of all hosts (4) to be remote +1 (4) (*errors.errorString)
test-compose-server-1-1  |        4: cmd/peer-rest-client.go:933:cmd.newPeerRestClients()
test-compose-server-1-1  |        3: cmd/notification.go:1276:cmd.NewNotificationSys()
test-compose-server-1-1  |        2: cmd/server-main.go:219:cmd.newAllSubsystems()
test-compose-server-1-1  |        1: cmd/server-main.go:470:cmd.serverMain()
test-compose-server-1-1  |
test-compose-server-1-1  | API: SYSTEM()
test-compose-server-1-1  | Time: 21:53:32 UTC 04/11/2022
test-compose-server-1-1  | Error: Read failed. Insufficient number of disks online (*errors.errorString)
test-compose-server-1-1  |        5: cmd/prepare-storage.go:268:cmd.connectLoadInitFormats()
test-compose-server-1-1  |        4: cmd/prepare-storage.go:317:cmd.waitForFormatErasure()
test-compose-server-1-1  |        3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
test-compose-server-1-1  |        2: cmd/server-main.go:637:cmd.newObjectLayer()
test-compose-server-1-1  |        1: cmd/server-main.go:542:cmd.serverMain()
test-compose-server-1-1  | Waiting for a minimum of 2 disks to come online (elapsed 1s)
test-compose-server-1-1  |
test-compose-server-1-1  |
test-compose-server-1-1  | API: SYSTEM()
test-compose-server-1-1  | Time: 21:53:33 UTC 04/11/2022
test-compose-server-1-1  | Error: Read failed. Insufficient number of disks online (*errors.errorString)
test-compose-server-1-1  |        5: cmd/prepare-storage.go:268:cmd.connectLoadInitFormats()
test-compose-server-1-1  |        4: cmd/prepare-storage.go:317:cmd.waitForFormatErasure()
test-compose-server-1-1  |        3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
test-compose-server-1-1  |        2: cmd/server-main.go:637:cmd.newObjectLayer()
test-compose-server-1-1  |        1: cmd/server-main.go:542:cmd.serverMain()
test-compose-server-1-1  | Waiting for a minimum of 2 disks to come online (elapsed 1s)
test-compose-server-1-1  |
test-compose-server-1-1  |
test-compose-server-1-1  | API: SYSTEM()
test-compose-server-1-1  | Time: 21:53:33 UTC 04/11/2022
test-compose-server-1-1  | Error: Read failed. Insufficient number of disks online (*errors.errorString)
test-compose-server-1-1  |        5: cmd/prepare-storage.go:268:cmd.connectLoadInitFormats()
test-compose-server-1-1  |        4: cmd/prepare-storage.go:317:cmd.waitForFormatErasure()
test-compose-server-1-1  |        3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
test-compose-server-1-1  |        2: cmd/server-main.go:637:cmd.newObjectLayer()
test-compose-server-1-1  |        1: cmd/server-main.go:542:cmd.serverMain()
test-compose-server-1-1  | Waiting for a minimum of 2 disks to come online (elapsed 2s)
test-compose-server-1-1  |
test-compose-server-1-1  |
test-compose-server-1-1  | API: SYSTEM()
test-compose-server-1-1  | Time: 21:53:34 UTC 04/11/2022
test-compose-server-1-1  | Error: Read failed. Insufficient number of disks online (*errors.errorString)
test-compose-server-1-1  |        5: cmd/prepare-storage.go:268:cmd.connectLoadInitFormats()
test-compose-server-1-1  |        4: cmd/prepare-storage.go:317:cmd.waitForFormatErasure()
test-compose-server-1-1  |        3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
test-compose-server-1-1  |        2: cmd/server-main.go:637:cmd.newObjectLayer()
test-compose-server-1-1  |        1: cmd/server-main.go:542:cmd.serverMain()
test-compose-server-1-1  | Waiting for a minimum of 2 disks to come online (elapsed 2s)
test-compose-server-1-1  |
test-compose-server-1-1  |
test-compose-server-1-1  | API: SYSTEM()
test-compose-server-1-1  | Time: 21:53:34 UTC 04/11/2022
test-compose-server-1-1  | Error: Read failed. Insufficient number of disks online (*errors.errorString)
test-compose-server-1-1  |        5: cmd/prepare-storage.go:268:cmd.connectLoadInitFormats()
test-compose-server-1-1  |        4: cmd/prepare-storage.go:317:cmd.waitForFormatErasure()
test-compose-server-1-1  |        3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
test-compose-server-1-1  |        2: cmd/server-main.go:637:cmd.newObjectLayer()
test-compose-server-1-1  |        1: cmd/server-main.go:542:cmd.serverMain()
test-compose-server-1-1  | Waiting for a minimum of 2 disks to come online (elapsed 3s)
test-compose-server-1-1  |
test-compose-server-1-1  | Unable to read 'format.json' from http://store-1:9000/.: Server expects 'storage' API version 'v40', instead found 'v40' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (RELEASE.2021-10-13T00-23-17Z)
test-compose-server-1-1  |
test-compose-server-1-1  | Unable to read 'format.json' from http://store-2:9000/.: Server expects 'storage' API version 'v40', instead found 'v40' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (RELEASE.2021-10-13T00-23-17Z)
test-compose-server-1-1  |
test-compose-server-1-1  | Unable to read 'format.json' from http://store-3:9000/.: Server expects 'storage' API version 'v40', instead found 'v40' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (RELEASE.2021-10-13T00-23-17Z)
test-compose-server-1-1  |
test-compose-server-1-1  | Unable to read 'format.json' from http://store-4:9000/.: Server expects 'storage' API version 'v40', instead found 'v40' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (RELEASE.2021-10-13T00-23-17Z)
test-compose-server-1-1  |
test-compose-server-1-1  |
test-compose-server-1-1  | API: SYSTEM()
test-compose-server-1-1  | Time: 21:53:35 UTC 04/11/2022
test-compose-server-1-1  | Error: Read failed. Insufficient number of disks online (*errors.errorString)
test-compose-server-1-1  |        5: cmd/prepare-storage.go:268:cmd.connectLoadInitFormats()
test-compose-server-1-1  |        4: cmd/prepare-storage.go:317:cmd.waitForFormatErasure()
test-compose-server-1-1  |        3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
test-compose-server-1-1  |        2: cmd/server-main.go:637:cmd.newObjectLayer()
test-compose-server-1-1  |        1: cmd/server-main.go:542:cmd.serverMain()
test-compose-server-1-1  | Waiting for a minimum of 2 disks to come online (elapsed 3s)
test-compose-server-1-1  |
test-compose-server-1-1  | Unable to read 'format.json' from http://store-1:9000/.: Server expects 'storage' API version 'v40', instead found 'v40' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (RELEASE.2021-10-13T00-23-17Z)
test-compose-server-1-1  |
test-compose-server-1-1  | Unable to read 'format.json' from http://store-2:9000/.: Server expects 'storage' API version 'v40', instead found 'v40' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (RELEASE.2021-10-13T00-23-17Z)
test-compose-server-1-1  |
test-compose-server-1-1  | Unable to read 'format.json' from http://store-3:9000/.: Server expects 'storage' API version 'v40', instead found 'v40' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (RELEASE.2021-10-13T00-23-17Z)
test-compose-server-1-1  |
test-compose-server-1-1  | Unable to read 'format.json' from http://store-4:9000/.: Server expects 'storage' API version 'v40', instead found 'v40' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (RELEASE.2021-10-13T00-23-17Z)
test-compose-server-1-1  |
test-compose-server-1-1  |
test-compose-server-1-1  | API: SYSTEM()
test-compose-server-1-1  | Time: 21:53:35 UTC 04/11/2022
test-compose-server-1-1  | Error: Read failed. Insufficient number of disks online (*errors.errorString)
test-compose-server-1-1  |        5: cmd/prepare-storage.go:268:cmd.connectLoadInitFormats()
test-compose-server-1-1  |        4: cmd/prepare-storage.go:317:cmd.waitForFormatErasure()
test-compose-server-1-1  |        3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
test-compose-server-1-1  |        2: cmd/server-main.go:637:cmd.newObjectLayer()
test-compose-server-1-1  |        1: cmd/server-main.go:542:cmd.serverMain()
test-compose-server-1-1  | Waiting for a minimum of 2 disks to come online (elapsed 4s)
test-compose-server-1-1  |
test-compose-server-1-1  | Unable to read 'format.json' from http://store-1:9000/.: Server expects 'storage' API version 'v40', instead found 'v40' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (RELEASE.2021-10-13T00-23-17Z)
test-compose-server-1-1  |
test-compose-server-1-1  | Unable to read 'format.json' from http://store-2:9000/.: Server expects 'storage' API version 'v40', instead found 'v40' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (RELEASE.2021-10-13T00-23-17Z)
test-compose-server-1-1  |
test-compose-server-1-1  | Unable to read 'format.json' from http://store-3:9000/.: Server expects 'storage' API version 'v40', instead found 'v40' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (RELEASE.2021-10-13T00-23-17Z)
test-compose-server-1-1  |
test-compose-server-1-1  | Unable to read 'format.json' from http://store-4:9000/.: Server expects 'storage' API version 'v40', instead found 'v40' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (RELEASE.2021-10-13T00-23-17Z)
test-compose-server-1-1  |
test-compose-server-1-1  |
test-compose-server-1-1  | API: SYSTEM()
test-compose-server-1-1  | Time: 21:53:36 UTC 04/11/2022
test-compose-server-1-1  | Error: Read failed. Insufficient number of disks online (*errors.errorString)
test-compose-server-1-1  |        5: cmd/prepare-storage.go:268:cmd.connectLoadInitFormats()
test-compose-server-1-1  |        4: cmd/prepare-storage.go:317:cmd.waitForFormatErasure()
test-compose-server-1-1  |        3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
test-compose-server-1-1  |        2: cmd/server-main.go:637:cmd.newObjectLayer()
test-compose-server-1-1  |        1: cmd/server-main.go:542:cmd.serverMain()
test-compose-server-1-1  | Waiting for a minimum of 2 disks to come online (elapsed 4s)
test-compose-server-1-1  |
Gracefully stopping... (press Ctrl+C again to force)
Container test-compose-server-1-1  Stopping
Container test-compose-server-1-1  Stopped
Container test-compose-store-4-1  Stopping
Container test-compose-store-1-1  Stopping
Container test-compose-store-2-1  Stopping
Container test-compose-store-3-1  Stopping
Container test-compose-store-2-1  Stopped
Container test-compose-store-4-1  Stopped
Container test-compose-store-1-1  Stopped
Container test-compose-store-3-1  Stopped
canceled

shell returned 127

I truly don't understand why this isn't working.

I've tried adding /data/disk1 to the end of the URL for the server configuration and several other things... but I have yet to have luck.

The "nodes" (i.e. the MinIO instances which have a /data/disk1 in them) all work and I'm able to log into them and access them, but the server will never start.

What am I doing wrong?

score 2 · Accepted Answer · edited Aug 05 '22 at 19:36

I figured it out. The error shown has nothing to do with the actual error.

In my mind, I was thinking the way this worked is that you:

Started a bunch of individual nodes; then
Started a distributed node to group the individuals together (by giving it the URLs to each individual node)

As it turns out, this is the wrong way to think of it. Instead, you:

Start a bunch of distributed nodes, and give them what their addresses will be once they start.

For example, if you're going to start four containers with hostnames host1, host2, etc, you would run this command on each of the hosts:

server --console-address ":9001" "http://host{1...4}:9000"

That's it. They'll auto-discover each other.

What I don't know is why it gives such cryptic errors that appear to have nothing to do with the actual problem, but this was the actual problem.

Distributed MinIO Self-Contained Example Fails: "Server expects 'storage' API version 'v40', instead found 'v40'"

1 Answers1