We have a vanilla Kubernetes cluster on AWS which has our deployments running . The pods are configured with Istio and we havent faced any issue all these years . All of a sudden since few hours , we started seeing this error being thrown from one of the istiod pods which might be a reason that the application is not able to connect to external service . The error looks like this . If anyone has faced a similar error in the past and was able to resolve it , could you please throw some light -
warn serverca Authentication failed for xx.xx.xx.xx:xxxxx: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: the service account authentication returns an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, unknown]. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found.
We tried the following:
- Doing curl from one of the application pod to the external service which gave us "upstream connect error or disconnect/reset before headers. reset reason: local reset" which is same as what we saw in application logs
- Disabled istio-injection at pod level and were able to curl to the external service and received response. However, from inside the application that is running, we received a connection timeout error (in stead of the error of upstream connect)
The external service is hosted on a stand alone ec2 instance
