Cant access redis from fastapi api when they are in a same cluster, same pod, using a single node cluster via docker desktop

Question

2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
2023-01-07 21:05:14     result = await app(  # type: ignore[func-returns-value]
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
2023-01-07 21:05:14     return await self.app(scope, receive, send)
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 270, in __call__
2023-01-07 21:05:14     await super().__call__(scope, receive, send)
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 124, in __call__
2023-01-07 21:05:14     await self.middleware_stack(scope, receive, send)
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
2023-01-07 21:05:14     raise exc
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
2023-01-07 21:05:14     await self.app(scope, receive, _send)
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
2023-01-07 21:05:14     raise exc
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
2023-01-07 21:05:14     await self.app(scope, receive, sender)
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
2023-01-07 21:05:14     raise e
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
2023-01-07 21:05:14     await self.app(scope, receive, send)
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 706, in __call__
2023-01-07 21:05:14     await route.handle(scope, receive, send)
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
2023-01-07 21:05:14     await self.app(scope, receive, send)
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
2023-01-07 21:05:14     response = await func(request)
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 235, in app
2023-01-07 21:05:14     raw_response = await run_endpoint_function(
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 161, in run_endpoint_function
2023-01-07 21:05:14     return await dependant.call(**values)
2023-01-07 21:05:14   File "/app/./main.py", line 49, in login
2023-01-07 21:05:14     stored_password = await redis_client.hget(name=user.username, key="password")
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/aioredis/client.py", line 1082, in execute_command
2023-01-07 21:05:14     conn = self.connection or await pool.get_connection(command_name, **options)
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/aioredis/connection.py", line 1416, in get_connection
2023-01-07 21:05:14     await connection.connect()
2023-01-07 21:05:14   File "/usr/local/lib/python3.10/site-packages/aioredis/connection.py", line 698, in connect
2023-01-07 21:05:14     raise ConnectionError(self._error_message(e))
2023-01-07 21:05:14 aioredis.exceptions.ConnectionError: Error 111 connecting to redis:6379. 111.

deployment.yaml file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: auth-service
  template:
    metadata:
      labels:
        app: auth-service
    spec:
      containers:
      - name: auth-service
        image: localhost:5000/auth-service:latest
        ports:
        - containerPort: 8000
        resources:
          limits:
            cpu: "1"
            memory: "1Gi"
          requests:
            cpu: "0.5"
            memory: "500Mi"
        envFrom:
          - secretRef:
              name: auth-service-fastapi-secrets
      - name: redis
        image: redis:alpine
        ports:
        - containerPort: 6379
        resources:
          limits:
            cpu: "1"
            memory: "1Gi"
          requests:
            cpu: "0.5"
            memory: "500Mi"
---
apiVersion: v1
kind: Service
metadata:
  name: auth-service
spec:
  type: NodePort
  selector:
    app: auth-service
  ports:
  - port: 8000
    targetPort: 8000
    nodePort: 30080
---
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  selector:
    app: redis
  ports:
  - port: 6379
    targetPort: 6379

my docker-compose did work: docker-compose.yml:

version: "3"
services:
  auth-service:
    build: .
    ports:
      - "8000:8000"
    environment:
      REDIS_HOST: redis
      REDIS_PORT: 6379
    env_file: fastapi.env
    depends_on:
      - redis
  redis:
    image: "redis:alpine"
    volumes:
      - redis_data:/data

volumes:
  redis_data:

fastapi code:

import os
import bcrypt
import jwt
import aioredis

from datetime import datetime
from fastapi import FastAPI, HTTPException, Cookie, Depends, Response
from pydantic import BaseModel, EmailStr


app = FastAPI()

# user model


class User(BaseModel):
    username: str
    password: str

# env variables
SECRET_KEY = os.environ["JWT_SECRET"]

# redis connection
redis_client = aioredis.from_url(
    "redis://redis:6379", encoding="utf-8", decode_responses=True)

So I am not sure what the problem is.

I have tried talking to chatGPT, didn't quite worked for me, also, I tried using the cluster ip of redis in the fastapi code instead of the name "redis":

aioredis.from_url( "redis://redis:6379", encoding="utf-8", decode_responses=True)

still not working: ConnectionRefusedError: [Errno 111] Connect call failed ('10.107.169.72', 6379)

saw some similar question on stackoverflow, but still confused after reading it.

Answer 1

When your application code connects to redis:6379, in Kubernetes, it connects to the Service named redis in the same namespace. In the setup you've shown, that forwards requests to Pods with a label app: redis. There aren't any of those Pods, though, which results in your error.

You should also be able to see this comparing kubectl describe service auth-service and kubectl describe service redis. The redis service should end with a line like

Endpoints:                <none>

which is usually a sign that the Service's selector: doesn't match the Pods' labels:.

In your case, the right answer is to split the Deployment into two, with only one container each. Especially:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  ...
spec:
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:alpine

This has a couple of other technical advantages. If you rebuild your application and change the image: tag in your main Deployment, the restart won't also restart Redis, so you'll keep your cache. If you set the main application to have multiple replicas:, they'll all share the same single Redis in the other Deployment.

(If you want to set up your Redis to also persist its data to disk, use a StatefulSet rather than a Deployment. This is a more complicated setup, and comes with requirements like an additional Service. If you're fine with your Redis occasionally losing its state than a Deployment is fine.)

Answer 2

According to this answer, you can access other containers in the same pod on localhost, so in your case localhost:6379.

I would also consider why you're running both your application and redis on the same pod. The whole point of using kubernetes is to be able to scale your application, and having several containers in the same pod makes it impossible to scale your app without also scaling redis, and vice versa.