0

I'm deploying rocketmq on a managed GKE Kubernetes cluster. The name server starts correctly, but the broker runs in error and I cannot recognize which kind of problem occurs. It follows the yaml configuration:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rocketmq-broker-log-storage
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard
  fstype: ext4
  replication-type: none

---

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rocketmq-broker-store-storage
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard
  fstype: ext4
  replication-type: none

---

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rocketmq-namesrv-storage
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard
  fstype: ext4
  replication-type: none

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rocketmq-broker-log-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Mi
  storageClassName: rocketmq-broker-log-storage

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rocketmq-broker-store-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Mi
  storageClassName: rocketmq-broker-store-storage

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rocketmq-namesrv-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Mi
  storageClassName: rocketmq-namesrv-storage

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rocketmq-namesrv
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rocketmq-namesrv
  template:
    metadata:
      labels:
        app: rocketmq-namesrv
    spec:
      containers:
      - name: namesrv
        image: myrepo/rocketmq:4.9.3-alpine
        command: ["sh", "mqnamesrv"]
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9876
        volumeMounts:
        - name: namesrv-log
          mountPath: /var/log
      volumes:
      - name: namesrv-log
        persistentVolumeClaim:
          claimName: rocketmq-namesrv-pvc

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rocketmq-broker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rocketmq-broker
  template:
    metadata:
      labels:
        app: rocketmq-broker
    spec:
      containers:
      - name: broker
        image: myrepo/rocketmq:4.9.3-alpine
        command: ["sh", "mqbroker", "-n", "rocketmq-namesrv:9876"]
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 10909
        - containerPort: 10911
        volumeMounts:
        - name: broker-log
          mountPath: /home/rocketmq/logs
        - name: broker-store
          mountPath: /home/rocketmq/store
      volumes:
      - name: broker-log
        persistentVolumeClaim:
          claimName: rocketmq-broker-log-pvc
      - name: broker-store
        persistentVolumeClaim:
          claimName: rocketmq-broker-store-pvc

---

apiVersion: v1
kind: Service
metadata:
  name: rocketmq-namesrv
spec:
  type: NodePort
  selector:
    app: rocketmq-namesrv
  ports:
  - name: namesrv
    port: 9876

---

apiVersion: v1
kind: Service
metadata:
  name: rocketmq-broker
spec:
  type: NodePort
  selector:
    app: rocketmq-broker
  ports:
  - name: broker1
    port: 10909
  - name: broker2
    port: 10911

The image rocketmq:4.9.3-alpine was created following the procedure on the apache/rocketmq-docker repo.

I tried kubectl logs and kubectl describe, but I cannot figure out what could be the problem because they give me no information.

What could be the problem?

sctx
  • 128
  • 2
  • 11
  • Which GKE version are you using? Could you share what exactly error do you have? Do you have any issues with the pods? Information about errors are by default deleted within 1 hour. Did you check Cloud Logging if there was any error/warning? Did you follow any documentation? In most RocketMQ configs I can see that there is at least `1Gi` storage, could you try with `1Gi` instead `200Mi`? – PjoterS Mar 10 '22 at 08:31
  • Thanks @PjoterS for your answer. Version is 1.20.12-gke.1500 (stable). Increasing the storage does not solve the problem. The exit code is 253 and the message on gke is "Does not have minimum availability". I tried to set cpu and memory limits, gradually increasing them, but it does not seem to be the solution. For me, it's the first time I'm using rocketmq and I followed the official documentation + instructions on the `apache/rocketmq-docker` github repo. – sctx Mar 10 '22 at 15:44
  • Do you have anything in `$ kubectl get events`? Which machine type did you chose? It might be related to insufficient resources or Liveliness/Readiness proble faliure. To clarify, you've used [this instruction](https://github.com/apache/rocketmq-docker)? In which exactly step you've encountered issue? – PjoterS Mar 11 '22 at 10:06
  • `kubectl get events` returns messages that seem to me not so relevant.. Machines are e2-medium. I used the instructions you provided. The deployments they provide on the repo are old and they do not work on gke. I modified them to get them work, but nothing... – sctx Mar 11 '22 at 14:27
  • I've tried to replicate this and I've encounter `OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N` Could you share your exact steps and also provide output of `kubectl get po -o wide` ? Also did you try to use image `apache/rocketmq`? – PjoterS Mar 14 '22 at 11:48
  • @PjoterS I get the same message from the container with the broker. I create a cluster in GKE stable version with e2-medium machines. Then I connect from my local shell with `gcloud container clusters get-credentials ...` and then I launch `kubectl apply -f depl.yaml`. I tried using both official `apache/rocketmq` image and my own created image. Output with `-o wide` is: `rocketmq-broker-6886577599-2ftbm 0/1 Error 5 3m55s 10.120.1.3 gke-cluster-1-xxx ` – sctx Mar 22 '22 at 07:33
  • If the message `OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N` is causing the error, I don't actually know the reason of this error. I tried increasing disk to 20G, memory to 2G, cpu to 800m... but nothing changes – sctx Mar 22 '22 at 07:53

0 Answers0