Kubernetes pod start up

Question

I have 2 pods running. They are;

mail-services-pod
redis-pod

I need to make sure that the redis server is (via redis-pod) up and running before creating the mail-services-pod as its is dependant on redis-pod.

I am new to kubernetes and would like to know what are the best ways to implement this check.

Cheers

The mail-services-pod should behave gracefully if redis is unavailable. Designing kubernetes services requires a mind-set that expects things to fail. — w08r, Oct 01 '20 at 14:54

score 2 · Answer 1 · answered Oct 01 '20 at 15:53

I am new to kubernetes and would like to know what are the best ways to implement this check

Kubernetes is a distributed environment and instances will change e.g. address on deployment on new versions of your apps.

It is important that your app is recilient e.g. to network issues, and that your app properly do retry if a connection fails.

When recilient connections is properly handled by your app, the start order of your apps is no longer an issue.

score 0 · Answer 2 · answered Oct 01 '20 at 17:45

So the best way I believe this can be done is via an initContainer

So let me use a contrived example. I want to start any app named some-app-that-uses-redis which uses alpine in my example before redis starts.

I would use a pod with an initContainer to do a little check to see if redis is up and running:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: some-app-that-uses-redis
  name: some-app-that-uses-redis
spec:
  initContainers:
  - name: wait-for-redis
    image: goodsmileduck/redis-cli
    imagePullPolicy: IfNotPresent
    args: [ 'sh', '-c', 'until redis-cli  -h redis-service.default.svc.cluster.local -p 6379  get hello; do echo "Sleeping a bit"; sleep 1; done; echo "ready!"; ' ]
  containers:
  - args:
    - sleep
    - "1000"
    image: alpine
    name: some-app-that-uses-redis
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

The meat of the check is the following bit, which is just a for loop that does a redis query to the service where redis should work.

initContainers:
  - name: wait-for-redis
    image: goodsmileduck/redis-cli
    imagePullPolicy: IfNotPresent
    args: [ 'sh', '-c', 'until redis-cli  -h redis-service.default.svc.cluster.local -p 6379  get hello; do echo "Sleeping a bit"; sleep 1; done; echo "ready!"; ' ]

Then I'd make sure redis is part of a service so that this test will always be true:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: redis
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis
    spec:
      containers:
      - image: redis:5.0.9
        name: redis
        resources: {}
status: {}
---
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: redis
  name: redis-service
spec:
  ports:
  - port: 6379
    protocol: TCP
    targetPort: 6379
  selector:
    app: redis
  type: NodePort
status:
  loadBalancer: {}

If I understand things correctly, pods should only speak to other pods by means of services anyway.

What happens if the Redis pod (or its node) fails _after_ the container starts up? — David Maze, Oct 01 '20 at 17:48
If they made the app themselves, then I agree they should add retry logic like a good 12 factor app should have. But it's not clear to me from the original question. As for node failure, isn't that something that can be remedied by things like stateful sets? — stephanve, Oct 01 '20 at 18:05

Kubernetes pod start up

2 Answers2