Service loses connection to Etcd DB when pod restarts

Question

I have a Go Lang REST service and ETCD DB in one container, deployed in kubernetes cluster using Deployment type. Whenever I try to restart the service pod, the service loses connectivity to ETCD, I have tried using stateful sets instead of deployment but still didn't help. My deployment looks something like below.

The ETCD fails restarting due to this issue: https://github.com/etcd-io/etcd/issues/10487

PVC :

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
     name: XXXX
     namespace: XXXX
     annotations:
       volume.beta.kubernetes.io/storage-class: glusterfs-storage
    spec:
     accessModes:
      - ReadWriteMany
     resources:
       requests:
         storage: 1Gi

Deployment:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: XXX
      namespace: XXX
    spec:
      replicas: X
      XXXXXXX
      template:
    metadata:
      labels:
        app: rest-service
        version: xx
      spec:
        hostAliases:
        - ip: 127.0.0.1
          hostnames:
          - "etcd.xxxxx"
        containers:
        - name: rest-service
          image: xxxx
          imagePullPolicy: IfNotPresent
          ports:
          - containerPort: xxx
          securityContext:
            readOnlyRootFilesystem: false
            capabilities:
              add:
              - IPC_LOCK
        - name: etcd-db
          image: quay.io/coreos/etcd:v3.3.11
          imagePullPolicy: IfNotPresent
          command:
          - etcd
          - --name=etcd-db
          - --listen-client-urls=https://0.0.0.0:2379
          - --advertise-client-urls=https://etcd.xxxx:2379
          - --data-dir=/var/etcd/data
          - --client-cert-auth
          - --trusted-ca-file=xxx/ca.crt
          - --cert-file=xxx/tls.crt
          - --key-file=xxx/tls.key
          volumeMounts:
          - mountPath: /var/etcd/data
            name: etcd-data
            XXXX
          ports:
          - containerPort: 2379
        volumes:
        - name: etcd-data
          persistentVolumeClaim:
            claimName: XXXX

I would expect the DB to be able to connect to pod even when it restarts

What is the address you are using in Golang service to connect to etcd ? — Ankit Deshpande, May 01 '19 at 16:51
You pod consists of two containers: application and Etcd. When you restart pod - both application and Etcd are restarted. There is no surprise that your application loses connectivity to Etcd. — Vasili Angapov, May 01 '19 at 16:58
@VasilyAngapov what would be the best course of action to fix this ? — Tarun Golthi, May 01 '19 at 17:00

score 1 · Answer 1 · answered May 01 '19 at 17:07

1

Keeping application and database in one pod is one of the worst practices in Kubernetes. If you update application code - you have to restart pod to apply changes. So you restart database also just for nothing.

Solution is very simple - you should run application in one deployment and database - in another. That way you can update application without restarting database. In that case you can also scale app and DB separately, like add more replicas to app while keeping DB at 1 replicas or vice versa.

answered May 01 '19 at 17:07

Vasili Angapov

8,061
15
31

That would work, but my application is not transaction intensive and data is shared accross PVC, i wanted to know why the Etcd is not starting up, The Etcd issue i get is this https://github.com/etcd-io/etcd/issues/10487 – Tarun Golthi May 01 '19 at 18:15

Service loses connection to Etcd DB when pod restarts

1 Answers1