0

I have an API that process a lot of information by calling another API. So for example, when the process starts in the API A, it makes a lot of http requests do the API B. It works fine for most requests, but some of them sometimes takes a lot of time to proccess on API B (more than 20 minutes). When it happens, the API B responds to the request successfully, but API A appears to not be aware of this response. So it just hangs infinitely waiting for a response.

Both API's are using kubernetes and nginx as a proxy. I was having some timeout problems, so I changed my timeout to 2000 seconds in my config file and for a while I didn't have any problems.

    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "2000"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "2000"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "2000"
    nginx.ingress.kubernetes.io/send-timeout: "2000"

This is how I'm calling the API B from API A:

    @Override
    public void startProcess(List<Long> ids) {
        logger.info("Starting process...");
        post("/start", ids);
        logger.info("Process finished!");
    }

    public <T, S> T post(String endpoint, S body, Class<?>... clazz) {
        var uri = generateURI(endpoint);
        var requestBuilder = HttpRequest.newBuilder(uri).timeout()
                .POST(HttpRequest.BodyPublishers.ofString(Serializer.serialize(body)));
        addCustomHeaders(requestBuilder, HttpMethod.POST);
        var request = requestBuilder.build();
        return doRequest(request, clazz);
    }

    @SneakyThrows
    private <T> T doRequest(HttpRequest request, Class<?>... clazz) {

        SSLContext context = obterContextoSsl();

        HttpClient client = HttpClient.newBuilder()
                .sslContext(context)
                .version(HttpClient.Version.HTTP_1_1)
                .build();

        try {
            HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
            if (!HttpStatus.valueOf(response.statusCode()).is2xxSuccessful()) {
                logger.error(String.format("Request error! %d: %s", response.statusCode(), response.body()));
                throw new HttpException(String.format("%d: %s", response.statusCode(), response.body()));
            }
            if (clazz.length == 0 || response.statusCode() == HttpStatus.NO_CONTENT.value())
                return null;
            return Serializer.deserialize(response.body(), clazz);
        } catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
            logger.error(ex);
            throw new HttpException(ex.getMessage());
        } catch (IOException | HttpException ex) {
            logger.error(ex);
            throw new HttpException(ex.getMessage());
        }
    }

This is the log from API A:

2022-07-13 17:33:35.323  INFO 1 --- [pool-5-thread-1] c.v.f.m.r.Repository: Starting process...

That's the only log from API A. Even after API B respond successfully to the request, API A just hangs there forever. Maybe somehow the connection is being closed but API A is not aware of it? Does anyone knows a way to fix this?

Gustavo Cesário
  • 1,310
  • 1
  • 12
  • 23
  • Does it really hang? Does it wait forever? Or does a worker thread die and the client thread is waiting for the result forever? Depending on the root cause your mitigation could vary. – Queeg Jul 13 '22 at 18:57
  • Where is the method `post(String, List)`? – Queeg Jul 13 '22 at 18:59
  • @HiranChaudhuri The post method is the one below startProcess. I'm not sure, I just believe it hangs because it stayed in that same state for almost an entire day. How could I verify if the worker thread died? – Gustavo Cesário Jul 13 '22 at 19:30
  • The method you posted has the signature `post(String, S, Class>...)` so it is different from what is called in startProcess(). – Queeg Jul 13 '22 at 19:35
  • @HiranChaudhuri Actually it's the right method. The first param is the endpoint, so it's just a string. The second param is generic, so I can pass any object and it will be serialized. For the third parameter I used varargs, so as I didn't pass anything to it, it will just be resolved to an empty array. I added a check for it in the doRequest method ```if (clazz.length == 0) ...``` – Gustavo Cesário Jul 13 '22 at 20:32
  • Which version of the JDK are you using? If 11 you might want to try the same with a more recent version. Also I'd suggest to enable logging. `-Djdk.httpclient.HttpClient.log=errors,requests,headers,channel` – daniel Aug 04 '22 at 13:47

0 Answers0