I'm running a spring boot (3.1.0) server, which uses an embedded Tomcat server. In front of this we have an api-gateway, emmissary-ingress, which is built on top of Envoy. This all works well, but we get the occasional (1 in every 100,000 or so) 503, we could of course do a retry, but I want to get to the bottom of this.
These 503's all have response_flag=UC, which according to the docs is: Upstream connection termination in addition to 503 response code. Which means I expect this being the fault of Tomcat. Especially since most (not all) of the 503's belong to the same customer.
However, in the logs of the upstream pod (that should handle the request) we don't see anything. In the config I have logging.level.root=info, which should also cover Tomcat (as manually setting them to debug gives me debug messages). Also if I look at JMX metrics of Tomcat I don't see a particularly high usage:
Where the red line corresponds to the 503.
What is the best course of action for me to debug this? And is it possible that I'm missing Tomcat logs? Could it be caused by envoy?
Thanks!
