1

I'm building an application on GKE (Google Kubernetes Engine) and a system using GCE instances of Redis. When I try to connect from the application pod on GKE to Redis on GCE, I get connection refused.(dial tcp <REMOTE-IP>:6379: connect: connection refused) The application is written in Go, and the redis library is go-redis(v8). Why can't I connect?

The source code for the connection part and the part where the error occurs is as follows.

    redisServerName = os.Getenv("REDIS_SERVER") // "sample.com:6379"
    redisClient = redis.NewClient(&redis.Options{
        Addr:     redisServerName,
        Password: "",
        DB:       0,
    })

    p, err := redisClient.Ping(ctx).Result()
    log.Println(p, err)

The hostname is resolved, so it is not a DNS problem, and the redis-cli commands are executable, so it does not seem to be a Firewall problem.

#  redis-cli -h <REMOTE_IP> ping
PONG

  • Postscript

Here is the result of running the command from the Pod with the Go application running

/# redis-cli -h redis.sample.com
redis.sample.com:6379>   // can connect

/# nc redis.sample.com 6379
// There is NO response.
lulu_39
  • 43
  • 8
  • Debugging details here would be regarding the K8s configuration. Did you run `redis-cli` on the exact container that is running the Go? – erik258 Sep 29 '21 at 15:14
  • Thank you for the comment. I deployed and ran redis-cli as a debug pod. (The environment where Go is running is alpine, so installing redis was a bit of a hassle...) The ping command (not redis-cli) does communicate from the alpine environment running Go. I did this to isolate whether the problem is with the GCP network environment or the go-redis library. – lulu_39 Sep 29 '21 at 16:18
  • ↑ The network seemed to communicate, so I thought it was a library problem.. – lulu_39 Sep 29 '21 at 16:20

1 Answers1

1

I assert that every application in a container will have the same layer 4 (for redis, TCP) access to the network. Since Redis provides no significant access control, this means that if one app on your container has network access to redis server, all other apps on the same container will too. And if one can't contact redis, neither will the other.

On the same container. This is where testing gets tricky, because it isn't helpful or feasible to reproduce your k8s and gke config here.

ICMP Ping and tcp/6379 are different. Just because ping works, doesn't mean Redis can, and vise versa. And different containers will have different network access in k8s and gke.

Do this test on the app container, to take everything possible out of he equation.

apk add redis only pulls in a few packages, only 8MB and provides redis-cli when I tested, but you don't need any client app for redis; it's simple enough to be done with, say, netcat . You don't have to issue a valid redis cmd, either - if you get an -ERR unknown command response, you know network works:

/ # echo "hi, redis!" |nc localhost 6379
-ERR unknown command `hi,`, with args beginning with: `redis!`,

If it works there and not in Go, it's probably because the environment variable REDIS_SERVER isn't set properly. So you might want to test that at the command line, as well.

nc $REDIS_SERVER 6379
erik258
  • 14,701
  • 2
  • 25
  • 31
  • Thanks for the reply. You are right, and I also tried installing netcat command and redis-cli to debug.I found that with redis-cli, I can make connections, and with netcat, I don't get any response back. (The results of the command execution will be added in the text.) – lulu_39 Oct 01 '21 at 16:59
  • 1
    I'll do some more debug and then Approve it! – lulu_39 Oct 01 '21 at 17:07
  • 1
    NewClient() in the init() function was the problem. and moving it to main() solved the problem. Debugging with nc command and looking at the redis log was very helpful. Thank you very much. – lulu_39 Oct 13 '21 at 15:56