0

Recently in one of our k8s deployments we are seeing a weird IP showing up in our logs. This error log is related to one of our services which is using the go-redis. Here is the body of the error log:

"jsonPayload":{
        "action":"HotStorage.Find",
        "error":"dial tcp: lookup 10.16.126.2.17: no such host",
        "event":{
          "caller":{
            "file":"/go/src/xxxxxxxx/repository/redis/invocations.go",
            "function":"xxxxxxxxxx/repository/redis.(*HotStorage).Find",
            "lineno":227
          },
        }

The weird thing about that message is that it's trying to connect to a 5 octet IP address which is obviously invalid. Actually that IP has an extra .17 at the end and rest is the correct address of one our Redis pods and this was not the first time this has happened.
During one occurrence I logged into one of the pods that was having this issue and tried to resolve the IP of the Redis pod manually using ping and it returned the correct address (with no extra parts). And it's good to note that we are using k8s hostnames to find our pods.

Note 1: It's always an extra .17 that messes up everything. Where is it coming from and why that number?

Note 2: We are using golang:1.16.3-alpine to build our artefacts and alpine:3.12 image to run them.

I'd be tremendously grateful if someone could help me or point to a direction for me to keep looking.

2hamed
  • 8,719
  • 13
  • 69
  • 112
  • Are you using `/17` (`255.255.128.0`) network mask? Maybe someone accidently put a `.` instead of a `/` somewhere? – Dominik Jun 28 '21 at 08:41
  • If this is getting the IP from Kubernetes cluster DNS, check the Endpoints object for the related Service. That may not answer where the bad data came from but it will narrow the field at least. – coderanger Jun 28 '21 at 08:43
  • @Dominik I'm not sure, but this IP is being resolved automatically using k8s dns and as I mentioned everything's fine when I resolve it manually. How can a netmask creep into the address? – 2hamed Jun 28 '21 at 08:49
  • @coderanger that's one of the first places I looked but I don't know what to look for. Everything looks fine to me. All the IP addresses are fine. – 2hamed Jun 28 '21 at 08:56
  • Okay, so it's not Kubernetes itself losing its mind, good narrowing. Next thing to check is either exec into the pod or run a secondary pod and try `nslookup` on the hostname to see if coredns is somehow sad. – coderanger Jun 28 '21 at 09:04
  • @coderanger since this error is not happening right now I have to try it at later time when it's happening again. But a qq, can this be caused by the internal Go's dns resolver? – 2hamed Jun 28 '21 at 09:28
  • Seems unlikely but everything that could cause this is unlikely so take your pick. – coderanger Jun 28 '21 at 09:38
  • @2hamed Just check your container with podman/docker/whatever and check the subnetmask. I cannot imagine this being a simple missconfiguration or a bug in go's dns resolver... – Dominik Jun 28 '21 at 13:11

0 Answers0