1

I try to deploy Matabase on my GKE cluster but I got Readiness probe failed.

I build on my local and get localhost:3000/api/health i got status 200 but on k8s it's not works.

Dockerfile. I create my own for push and build to my GitLab registry

FROM metabase/metabase:v0.41.6
EXPOSE 3000
CMD ["/app/run_metabase.sh" ]

my deployment.yaml

# apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metaba-dev
spec:
  selector:
    matchLabels:
      app: metaba-dev
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 50%
      maxSurge: 100%
  template:
    metadata:
      labels:
        app: metaba-dev
    spec:
      restartPolicy: Always
      imagePullSecrets:
        - name: gitlab-login
      containers:
        - name: metaba-dev
          image: registry.gitlab.com/team/metabase:dev-{{BUILD_NUMBER}}
          command: ["/app/run_metabase.sh" ]
          livenessProbe:
            httpGet:
              path: /api/health
              port: 3000
            initialDelaySeconds: 60
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /api/health
              port: 3000
            initialDelaySeconds: 60
            periodSeconds: 10

          imagePullPolicy: Always
          ports:
            - name: metaba-dev-port
              containerPort: 3000



      terminationGracePeriodSeconds: 90

I got this error from

kubectl describe pod metaba-dev

  Warning  Unhealthy                61s (x3 over 81s)      kubelet                                Readiness probe failed: Get "http://10.207.128.197:3000/api/health": dial tcp 10.207.128.197:3000: connect: connection refused
  Warning  Unhealthy                61s (x3 over 81s)      kubelet                                Liveness probe failed: Get "http://10.207.128.197:3000/api/health": dial tcp 10.207.128.197:3000: connect: connection refused

kubectl logs

Picked up JAVA_TOOL_OPTIONS: -Xmx1g -Xms1g -Xmx1g
Warning: environ value jdk-11.0.13+8 for key :java-version has been overwritten with 11.0.13
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2022-01-28 15:32:23,966 INFO metabase.util :: Maximum memory available to JVM: 989.9 MB
2022-01-28 15:33:09,703 INFO util.encryption :: Saved credentials encryption is ENABLED for this Metabase instance. 
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html

Here Solution

I add initialDelaySeconds: to 1200 and check logging it's cause about network mypod cannot connect to database and when i checking log i did not see that cause i has been restart and it's was a new log

Thotsaphon Sirikutta
  • 128
  • 7
  • Anytime I see `localhost` and "not work" I want to encourage you to try with the Pod's IP, not localhost, since if metabase is listening on 127.0.0.1 and not 0.0.0.0 that's a fine way for that `containerPort:` to not do what you expect, and thus kubelet cannot reach port 3000, either – mdaniel Jan 28 '22 at 16:01
  • @mdaniel i get u, but so how readinessProbe work I got another services i using same code and call health check with containerPort: it's work but why this one it's not – Thotsaphon Sirikutta Jan 28 '22 at 16:18
  • Well, did you try the curl without using localhost like I asked? because it's a common bug, so if you're tested it and the bug doesn't apply to you, we'll look for other problems, but it's a _common_ bug so start with the most obvious and work toward the most esoteric – mdaniel Jan 29 '22 at 01:22

1 Answers1

1

Try to change your initialDelaySeconds: 60 to 100 And you should always set the resource request and limit in your container to avoid the probe failure, this is because when your app starts hitting the resource limit the kubernetes starts throttling your container.

containers:


- name: app
    image: images.my-company.example/app:v4
    resources:
      requests:
        memory: "128Mi"
        cpu: "250m"
      limits:
        memory: "100Mi"
        cpu: "500m"
Leo
  • 695
  • 3
  • 11