Distributed secure MinIO in docker-compose

Question

I have a quite complex system in Docker. Everything runs in a big docker-compose file. Previously everything runs on one (manager) node in my Docker Swarm so I have generated a CERT for my domain (with certbot) and I have used the below MinIO service in my compose file:

  object_storage:
    image: minio/minio:RELEASE.2020-12-10T01-54-29Z
    ports:
      - 9000:9000
    environment:
      MINIO_ACCESS_KEY_FILE: object_storage_user
      MINIO_SECRET_KEY_FILE: object_storage_password
    command: server /data
    depends_on:
      - fluentd
    volumes:
      - object_storage_data:/data
      - ./certs/domain.crt:/root/.minio/certs/public.crt
      - ./certs/domain.key:/root/.minio/certs/private.key
    networks:
      - object_storage_net
    secrets:
      - object_storage_user
      - object_storage_password
    logging:
      driver: "fluentd"
      options:
        fluentd-address: ${SYSTEM_HOST}:24224
        tag: object-storage

The above implementation works as expected! But now I have 2 separated servers to run the MinIO. These servers are joined to my Docker Swarm as worker nodes. The MinIO shouldn't run on manager node (Only on two separated worker nodes)!

>>> docker node ls
ID                                          HOSTNAME    STATUS      AVAILABILITY   MANAGER STATUS   ENGINE VERSION
mcbkz9m5nzf7oa3fiqk0lf4qo *  manager         Ready           Active                    Leader                    20.10.1
dz4e3k70g8ik2z4bcx8u0ft9ao   minio_1          Ready           Active                                                   20.10.2
r0qpdn2guyy5773vo8vg2trzo    minio_2          Ready           Active                                                   20.10.2

My current MinIO implementation in my docker-compose file:

object_storage_1:
   image: minio/minio:RELEASE.2020-12-10T01-54-29Z
   ports:
     - 9000:9000
   environment:
     MINIO_ACCESS_KEY_FILE: object_storage_user
     MINIO_SECRET_KEY_FILE: object_storage_password
   command: server https://object_storage_{1...2}/data{1...2}
   depends_on:
     - fluentd
   volumes:
     - object_storage_data_1_1:/data1
     - object_storage_data_1_2:/data2
     - ./certs/domain.crt:/root/.minio/certs/public.crt
     - ./certs/domain.key:/root/.minio/certs/private.key
   networks:
     - object_storage_net
   secrets:
     - object_storage_user
     - object_storage_password
   deploy:
     restart_policy:
       condition: on-failure
     placement:
       constraints:
         - node.hostname == minio_1
   logging:
     driver: "fluentd"
     options:
       fluentd-address: ${SYSTEM_HOST}:24224
       tag: object-storage

 object_storage_2:
   image: minio/minio:RELEASE.2020-12-10T01-54-29Z
   ports:
     - 9000
   environment:
     MINIO_ACCESS_KEY_FILE: object_storage_user
     MINIO_SECRET_KEY_FILE: object_storage_password
   command: server https://object_storage_{1...2}/data{1...2}
   depends_on:
     - fluentd
   volumes:
     - object_storage_data_2_1:/data1
     - object_storage_data_2_2:/data2
     - ./certs/domain.crt:/root/.minio/certs/public.crt
     - ./certs/domain.key:/root/.minio/certs/private.key
   networks:
     - object_storage_net
   secrets:
     - object_storage_user
     - object_storage_password
   deploy:
     restart_policy:
       condition: on-failure
     placement:
       constraints:
         - node.hostname == minio_2
   logging:
     driver: "fluentd"
     options:
       fluentd-address: ${SYSTEM_HOST}:24224
       tag: object-storage

If I check the log of an instance of my MinIO service, I got the following error:

Unable to read 'format.json' from https://object_storage_1:9000/data1: Post "https://object_storage_1:9000/minio/storage/data1/v22/readall?disk-id=&file-path=format.json&volume=.minio.sys": x509: certificate is valid for my_domain.app, not object_storage_1
Unable to read 'format.json' from https://object_storage_2:9000/data1: Post "https://object_storage_2:9000/minio/storage/data1/v22/readall?disk-id=&file-path=format.json&volume=.minio.sys": x509: certificate is valid for my_domain.app, not object_storage_2

But I can reach the MinIO on 9000 port, just there is a pop-up error:

I want to access to MinIO only through my domain (my_domain.app:9000). The MinIO in this case doesn't use the real server name however it uses the "virtual" Docker network (Eg.: https://object_storage_2:9000).

My questions:

How can I generate certs for "virtual" Docker networks (Eg.: object_storage_1 or object_storage_2)?
Where should I put the generated certs?
Is is possible to solve with only my generate (for my domain) cert?

I am open for every hint and solution!

score 0 · Accepted Answer · answered Sep 24 '21 at 20:46

I had to put the (Domain) CERT file to minio/certs/CAs folder instead of /root/.minio/certs folder. Furthermore I had to copy the CERT to the worker nodes (separated servers) if I didn't copy it to nodes the service didn't find it on the worker node.

The correct volumes parameter looks like as below:

volumes:
  - object_storage_data_1_1:/data1
  - object_storage_data_1_2:/data2
  - ./certs/domain.crt:/root/.minio/certs/CAs/public.crt

One working service of my several MinIO services:

  object-storage-1:
    image: minio/minio:RELEASE.2021-08-17T20-53-08Z
    expose:
      - "9000"
      - "9001"
    environment:
      MINIO_ACCESS_KEY_FILE: object_storage_user
      MINIO_SECRET_KEY_FILE: object_storage_password
      MINIO_BROWSER_REDIRECT_URL: https://${SYSTEM_HOST}:9001
      MINIO_SERVER_URL: https://${SYSTEM_HOST}:9000
    command: server --console-address ":9001" http://object-storage-{1...4}/data{1...2}
    hostname: object-storage-1
    depends_on:
      - fluentd
    volumes:
      - object_storage_data_1_1:/data1
      - object_storage_data_1_2:/data2
      - ./certs/domain.crt:/root/.minio/certs/CAs/public.crt
    networks:
      - object_storage_net
    secrets:
      - object_storage_user
      - object_storage_password
    deploy:
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == minio1
    logging:
      driver: "fluentd"
      options:
        fluentd-address: ${SYSTEM_HOST}:24224
        tag: object-storage

AND I had to create a NgInx config:

upstream minio {
    server object-storage-1:9000;
    server object-storage-2:9000;
    server object-storage-3:9000;
    server object-storage-4:9000;
}

upstream console {
    ip_hash;
    server object-storage-1:9001;
    server object-storage-2:9001;
    server object-storage-3:9001;
    server object-storage-4:9001;
}

server {
    listen              9000 ssl;
    listen              [::]:9000 ssl;
    server_name         my.server.com;
    ssl_certificate     /ssl/domain.crt;
    ssl_certificate_key /ssl/domain.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;

    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;


    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://minio;

    }
}

server {
    listen              9001 ssl;
    listen              [::]:9001 ssl;
    server_name         my.server.com;
    ssl_certificate     /ssl/domain.crt;
    ssl_certificate_key /ssl/domain.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;

    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-NginX-Proxy true;

        # This is necessary to pass the correct IP to be hashed
        real_ip_header X-Real-IP;

        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://console;

    }
}

Distributed secure MinIO in docker-compose

1 Answers1

Linked