4

I have a problem with Minio installation.

Minio tenants stucked in state 'Waiting for MinIO TLS Certificate'. It doesn't matter if tenant created by helm chart or by additional yaml. Even if I'm creating tenant from Minio web console the result is the same.

I use chart installation from: https://github.com/minio/operator

helm repo remove minio
helm repo add minio https://operator.min.io/
helm install --namespace minio-operator --create-namespace --generate-name minio/minio-operator
3 kubectl apply -f https://github.com/minio/operator/blob/master/examples/tenant.yaml

Operator installs fine. Tenant after creation stacked with 'Waiting for MinIO TLS Certificate' message.

Logs from operator:

E0729 11:06:17.788400       1 operator.go:137] Unexpected error during the creation of the csr/operator-minio-csr: timeout during certificate fetching of csr/operator-minio-csr
I0729 11:06:17.788419       1 main-controller.go:627] Waiting for the operator certificates to be issued timeout during certificate fetching of csr/operator-minio-csr
I0729 11:06:27.795784       1 main-controller.go:625] operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)
I0729 11:06:27.817912       1 csr.go:145] Start polling for certificate of csr/operator-minio-csr, every 5s, timeout after 20m0s
E0729 11:26:07.973014       1 minio.go:213] Unexpected error during the creation of the csr/minio-minio-csr: timeout during certificate fetching of csr/minio-minio-csr
E0729 11:26:07.973050       1 main-controller.go:754] error syncing 'minio/minio': timeout during certificate fetching of csr/minio-minio-csr
E0729 11:26:27.823681       1 operator.go:137] Unexpected error during the creation of the csr/operator-minio-csr: timeout during certificate fetching of csr/operator-minio-csr
I0729 11:26:27.823700       1 main-controller.go:627] Waiting for the operator certificates to be issued timeout during certificate fetching of csr/operator-minio-csr
I0729 11:26:37.831111       1 main-controller.go:625] operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)
I0729 11:26:37.845819       1 csr.go:145] Start polling for certificate of csr/operator-minio-csr, every 5s, timeout after 20m0s
E0729 11:27:08.019483       1 main-controller.go:754] error syncing 'minio/minio': secrets "operator-tls" not found
I0729 11:28:08.036307       1 minio.go:141] Generating private key
I0729 11:28:08.036396       1 minio.go:154] Generating CSR with CN=minio
I0729 11:28:08.054702       1 csr.go:145] Start polling for certificate of csr/minio-minio-csr, every 5s, timeout after 20m0s

CSR request exists:

minio-minio-csr      15m    kubernetes.io/kubelet-serving   system:serviceaccount:minio:minio-operator   Approved
operator-minio-csr   163m   kubernetes.io/kubelet-serving   system:serviceaccount:minio:minio-operator   Approved

Tenant exists:

minio minio Waiting for MinIO TLS Certificate 37s

Example of tenant.yaml

---
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: minio
  namespace: minio
  labels:
    app: minio
  annotations:
    prometheus.io/path: /minio/v2/metrics/cluster
    prometheus.io/port: "9000"
    prometheus.io/scrape: "true"
spec:
  image: minio/minio:RELEASE.2021-06-17T00-10-46Z
  imagePullPolicy: IfNotPresent
  credsSecret:
    name: minio-creds-secret
  pools:
    - servers: 4
      name: pool-0
      volumesPerServer: 4
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        fsGroup: 1000
  mountPath: /export
  requestAutoCert: true
  s3:
    bucketDNS: false
  certConfig:
    commonName: "minio"
    organizationName: []
    dnsNames: []
  podManagementPolicy: Parallel
  serviceMetadata:
    minioServiceLabels:
      label: minio-svc
    minioServiceAnnotations:
      v2.min.io: minio-svc
    consoleServiceLabels:
      label: console-svc
    consoleServiceAnnotations:
      v2.min.io: console-svc
  console:
    image: minio/console:v0.7.5
    replicas: 2
    consoleSecret:
      name: console-secret
    securityContext:
      runAsUser: 1000
      runAsGroup: 2000
      runAsNonRoot: true
      fsGroup: 2000
user2820186
  • 51
  • 1
  • 4
  • 1
    It takes roughly 5-10 min. Keep wait – celcin Nov 03 '21 at 15:32
  • 1
    This is a very old image `RELEASE.2021-06-17T00-10-46Z`, please try with latest image available and latest Operator and also specify if you are using Kubernetes Vanilla or OpenShift as I haven't been able to reproduce this old issue. – Cesar Celis Mar 18 '23 at 21:09

3 Answers3

3

I remember there was some settings that need to be in place in k8s to enable requestAutoCert: true.

Something like:

kube-controller:
  extra_args:
    cluster-signing-cert-file: "/etc/kubernetes/ssl/kube-ca.pem"
    cluster-signing-key-file: "/etc/kubernetes/ssl/kube-ca-key.pem"
Henry Ecker
  • 34,399
  • 18
  • 41
  • 57
Blafasel42
  • 187
  • 7
  • 1
    I've disabled auto cert but the operator pod shows: ```1 operator.go:104] operator TLS secret not found: secrets "operator-tls" not found``` – Vahid Alimohamadi Feb 19 '22 at 13:43
2

You must restart the console and operator pods after the cluster has been updated with below section.

kube-controller:
  extra_args:
    cluster-signing-cert-file: "/etc/kubernetes/ssl/kube-ca.pem"
    cluster-signing-key-file: "/etc/kubernetes/ssl/kube-ca-key.pem"
Aniesh
  • 31
  • 1
  • 3
1

One way I found to install MinIO is like this:

  • File: kind-config.yaml

  • Content:

# four node (two workers) cluster config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker
  - role: worker
  1. Start by deleting any previous cluster
kind delete cluster

It should look like:

$ kind delete cluster
Deleting cluster "kind" ...
  1. Create the cluster:
kind create cluster --config kind-config.yaml

It should look like:

$ kind create cluster --config kind-config.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.24.0)  
 ✓ Preparing nodes       
 ✓ Writing configuration  
 ✓ Starting control-plane ️ 
 ✓ Installing CNI  
 ✓ Installing StorageClass  
 ✓ Joining worker nodes  
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next?   Check out https://kind.sigs.k8s.io/docs/user/quick-start/
  1. Remove and add the chart:
helm repo remove minio
helm repo add minio https://operator.min.io/

It should look like:

$ helm repo remove minio
"minio" has been removed from your repositories
$ helm repo add minio https://operator.min.io/
"minio" has been added to your repositories
  1. Download the repository locally:
cd ~/
git clone git@github.com:minio/operator.git

It should look like:

$ cd ~/
$ git clone git@github.com:minio/operator.git
Cloning into 'operator'...
remote: Enumerating objects: 13159, done.
remote: Counting objects: 100% (881/881), done.
remote: Compressing objects: 100% (196/196), done.
remote: Total 13159 (delta 674), reused 822 (delta 659), pack-reused 12278
Receiving objects: 100% (13159/13159), 8.65 MiB | 3.60 MiB/s, done.
Resolving deltas: 100% (8259/8259), done.
  1. Install the Operator:
cd ~/operator
helm install \
  --namespace minio-operator \
  --create-namespace minio-operator \
  minio/operator

It should look like:

$ cd ~/operator
helm install \
  --namespace minio-operator \
  --create-namespace minio-operator \
  minio/operator
NAME: minio-operator
LAST DEPLOYED: Fri Jun 24 17:50:19 2022
NAMESPACE: minio-operator
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the JWT for logging in to the console:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: console-sa-secret
  namespace: minio-operator
  annotations:
    kubernetes.io/service-account.name: console-sa
type: kubernetes.io/service-account-token
EOF
kubectl -n minio-operator  get secret console-sa-secret -o jsonpath="{.data.token}" | base64 --decode

2. Get the Operator Console URL by running these commands:
  kubectl --namespace minio-operator port-forward svc/console 9090:9090
  echo "Visit the Operator Console at http://127.0.0.1:9090"
  1. In case your cluster has no access to the internet, you can load image to the cluster:
kind load docker-image minio/console:v0.19.0

It should look like:

$ kind load docker-image minio/console:v0.19.0
Image: "minio/console:v0.19.0" with ID "sha256:739e933b5d9ddb22f690f3773cbcf4c7409113d6739d905e31e480cfa5c0a21d" not yet present on node "kind-worker2", loading...
Image: "minio/console:v0.19.0" with ID "sha256:739e933b5d9ddb22f690f3773cbcf4c7409113d6739d905e31e480cfa5c0a21d" not yet present on node "kind-worker4", loading...
Image: "minio/console:v0.19.0" with ID "sha256:739e933b5d9ddb22f690f3773cbcf4c7409113d6739d905e31e480cfa5c0a21d" not yet present on node "kind-worker", loading...
Image: "minio/console:v0.19.0" with ID "sha256:739e933b5d9ddb22f690f3773cbcf4c7409113d6739d905e31e480cfa5c0a21d" not yet present on node "kind-worker3", loading...
Image: "minio/console:v0.19.0" with ID "sha256:739e933b5d9ddb22f690f3773cbcf4c7409113d6739d905e31e480cfa5c0a21d" not yet present on node "kind-control-plane", loading...
  1. Install the tenant with Helm:
helm install --namespace tenant-ns \
  --create-namespace tenant minio/tenant

It should look like:

$ helm install --namespace tenant-ns \
  --create-namespace tenant minio/tenant
NAME: tenant
LAST DEPLOYED: Fri Jun 24 17:52:55 2022
NAMESPACE: tenant-ns
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To connect to the minio1 tenant if it doesn't have a service exposed, you can port-forward to it by running:

  kubectl --namespace tenant-ns port-forward svc/minio1-console 9443:9443

  Then visit the MinIO Console at https://127.0.0.1:9443
  1. Also provide the image to the cluster:
kind load docker-image quay.io/minio/minio:RELEASE.2022-05-26T05-48-41Z

It should look like:

$ kind load docker-image quay.io/minio/minio:RELEASE.2022-05-26T05-48-41Z
Image: "quay.io/minio/minio:RELEASE.2022-05-26T05-48-41Z" with ID "sha256:ee8072647d5aed0c6fd23090acdcc26da93787d329b091fbdeeb33d64409a28a" not yet present on node "kind-worker2", loading...
Image: "quay.io/minio/minio:RELEASE.2022-05-26T05-48-41Z" with ID "sha256:ee8072647d5aed0c6fd23090acdcc26da93787d329b091fbdeeb33d64409a28a" not yet present on node "kind-worker4", loading...
Image: "quay.io/minio/minio:RELEASE.2022-05-26T05-48-41Z" with ID "sha256:ee8072647d5aed0c6fd23090acdcc26da93787d329b091fbdeeb33d64409a28a" not yet present on node "kind-worker", loading...
Image: "quay.io/minio/minio:RELEASE.2022-05-26T05-48-41Z" with ID "sha256:ee8072647d5aed0c6fd23090acdcc26da93787d329b091fbdeeb33d64409a28a" not yet present on node "kind-worker3", loading...
Image: "quay.io/minio/minio:RELEASE.2022-05-26T05-48-41Z" with ID "sha256:ee8072647d5aed0c6fd23090acdcc26da93787d329b091fbdeeb33d64409a28a" not yet present on node "kind-control-plane", loading...
  1. Then look at the MinIO pods:
$ k get pods -n tenant-ns -l app=minio
NAME              READY   STATUS    RESTARTS   AGE
minio1-pool-0-0   1/1     Running   0          12m
minio1-pool-0-1   1/1     Running   0          12m
minio1-pool-0-2   1/1     Running   0          12m
minio1-pool-0-3   1/1     Running   0          12m

The reason for the TLS Message can be because of the type of cluster, I have seen similar problem with OpenShift Cluster but please specify the cluster to have an idea of what else could be. Also try with latest versions of MinIO and Operator to get proper functionality.

Cesar Celis
  • 166
  • 1
  • 4
  • 8