Failed to connect PostgreSQL in Kubernetes

Question

I am using Kubernetes. I have two pods.

In one pod, it is running a server. Here is the part of codes trying to connect PostgreSQL service in another pod.

import (
    "context"
    "github.com/jackc/pgx/v4/pgxpool"
    "github.com/rs/zerolog/log"
)

databaseURL := "postgres://admin:passw0rd@my-database-service.hm:40072/my_db"
pg, err := pgxpool.Connect(context.Background(), databaseURL)
if err != nil {
    log.Error().Err(err).Msg("conn.Close")
    return nil
}

Currently, above code gives the error

{"level":"error","error":"failed to connect to host=my-database-service.hm user=admin database=my_db: dial error (dial tcp 10.43.140.140:40072: connect: connection refused)","time":1627849943,"message":"conn.Close"}

However, after I ssh into the server pod, I can successfully connect the PostgreSQL pod by psql.

/usr/src/app # psql --host=my-database-service.hm --port=40072 --dbname=my_db --username=admin --password
Password: 
psql (13.3)
Type "help" for help.

my_db=#

I tried to restart my server pod to make sure it runs after PostgreSQL database is ready, and still got same error.

Any suggestion for further debugging would be helpful. Thanks!

UPDATE

I am actually using Linkerd. Might be related with

https://linkerd.io/2.10/features/protocol-detection/

I tried to change the PostgreSQL to default port 5432, but still have issue. Will update if find the solution.

Hongbo Miao · Accepted Answer · 2021-08-02T12:37:14.783

After some experiments, I can confirm my issue is not related with Linkerd.

I think my PostgreSQL containerPort/targetPort is still using the default 5432. And it exposed by ClusterIP inside of the network by port 40072, which is why I don't need mark the port as opaque in Linkerd.

It turns out that after adding os.Exit(1),

pg, err := pgxpool.Connect(context.Background(), databaseURL)
if err != nil {
    log.Error().Err(err).Msg("conn.Close")
    os.Exit(1) // <- this helps
}

my Kubernetes will help restart the server pod when it fails initially to wait the PostgreSQL is ready.

After a few times restarts, the server pod can connect PostgreSQL well now.

I think if I manually restarts more times early, it might help too. But definitely not reliable like this way letting Kubernetes to help restart.

Failed to connect PostgreSQL in Kubernetes

1 Answers1