Our GKE cluster started showing Konnectivity agent errors:
connection read failure" err="read tcp 172.16.51.22:49332->172.16.51.8:10250: use of closed network connection" connectionID=1
and
conn write failure" err="write tcp 172.16.51.22:49332->172.16.51.8:10250: use of closed network connection" connectionID=1
and
could not read stream" err="rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: ENHANCE_YOUR_CALM, debug data: \"too_many_pings\
These errors appear every 5 minutes and the last error sometimes doesn't show up.
We tried to update GKE cluster to version 1.22.6-gke.300 but it didn't help.
So the question is how do we get rid of this error? Or what could be the cause of the error?