How to best run Symfony Messenger Worker on Kubernetes

Question

What's the best way to run the messenger:consume task on Kubernetes?

Deployment?

If we do it with a deployment with a certain number of replicas this could work, but what if we do a rolling update of the deployment which then results in a pod to be replaced although a long running message is handled by it right now?

To prevent this we could probably set a high terminationGracePeriodSeconds and use lifecycle.preStop in combination?

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    run: deploy
  name: deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      run: deploy
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        run: deploy
    spec:
      terminationGracePeriodSeconds: 6000
      containers:
      - command:
        - sh
        - -c
        - sleep 1d # bin/console messenger:consume
        image: bash
        name: deploy
        lifecycle:
          preStop:
            exec:
              command:
              - sh
              - -c
              - echo "Test if a message is consumed at the moment and prevent POD shutdown till then?"

But during my tests, even if the lifecycle.preStop tasks stops early, the full time defined by terminationGracePeriodSeconds is still waited till the pod is terminated.

Anyone got a good idea out there?

Experimental AND hot :D But this could also be seen as a Kubernetes issue for any other long running processes in a Deployment. — Kim, Jul 03 '19 at 16:24
it's seems is working as expected. Please refer to [Container hooks execution](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) and [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods). I would like suggest to use another approach like blue green deployment. — Mark, Jul 05 '19 at 12:56

score 2 · Answer 1 · answered Mar 04 '20 at 10:01

This works for me:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent
spec:
  replicas: 8
  selector:
    matchLabels:
      id: agent-pod
  template:
    metadata:
      labels:
        id: agent-pod
    spec:
      terminationGracePeriodSeconds: 3600
      containers:
      - name: agent
        volumeMounts:
          - mountPath: /share
            name: share
        lifecycle:
          preStop:
            exec:
              command: ["sh", "-c", "touch /tmp/kill_me"]
        command:
        - /bin/sh
        - -c
        - >
          sleep 3 && rm -rf var/cache/* && bin/console cache:clear &&
          while ! [ -f /tmp/kill_me ];
            do
            bin/console messenger:consume queue --limit=1 --time-limit=60 -vv
          done;
        image: my-symfony-agent-image

More here.

How to best run Symfony Messenger Worker on Kubernetes

1 Answers1