We have a system that we are migrating from AWS to Google Cloud. The servers are containerized and deployed through Kubernetes.
We have the frontend making api calls to the backend. A lot of the time these api calls succeed.
Intermittently, we see that instead of hitting the backend, the api call fails with a 400 response, with the body content set to the generic Google error:
"400. That’s an error.
Your client has issued a malformed or illegal request. That’s all we know. "
We can't see the request hitting the backend in our logs. It's as if it's become lost inside Google.
Doing exactly the same call from the command line from inside the frontend container, or from our local machines works every time.
My questions are:
i) Has anyone seen something similar and have a workaround for the issue?
ii) How can we debug this?
fyi the request is a simple GET that looks like: "https://example.com/v1/session/12345?frontend_ip=1.2.3.4%2C+5.6.7.8&req_timestamp=2018-04-19+10%3A46"