I'm experiencing an issue with the restTemplate while making requests to an external API in my service. Most of the time, the API calls are functioning normally with an average response time of 2-3ms. However, occasionally, the response time suddenly spikes to 3000-4000ms, persisting for a short period, triggering API circuit breaking. From low response time to high response time, there is no gradual increase; instead, it happens in an instant.
My service is deployed on three nodes in the production environment, but the circuit breaking only occurs on one specific node each time, and there is no specific pattern regarding the timing. The frequency of occurrence is approximately once every two days.
To provide more context, there is no significant increase in the number of requests prior to the circuit breaking, and the external API service is returning responses without any noticeable increase in response time.
This is the function where problem happens
@Cacheable(value = "rep-content-linked", key = "#infoCode.concat(#clientForLink).concat(#md5)", unless = "#result == null or #result == ''")
public String getLinkedContent(String clientForLink, String infoCode, String content, String md5, String reqTrace) {
LinkedContentRequest request = new LinkedContentRequest();
request.setArtCode(infoCode);
request.setTxt(content);
request.setClient(clientForLink);
request.setTxtType("report");
request.setReq_trace(reqTrace);
LinkedContentResponse contentWithLink = restTemplateForLink.postForObject(getContentWithLink, request, LinkedContentResponse.class);
return (contentWithLink != null && contentWithLink.getData() != null)
? contentWithLink.getData()
.getResultTxt()
: "";
}
According to log, I'm sure the problem occurr at the following code line
LinkedContentResponse contentWithLink = restTemplateForLink.postForObject(getContentWithLink, request, LinkedContentResponse.class);
This is my restTemplate config:
@Bean(name = "httpRequestFactoryForLink")
public ClientHttpRequestFactory httpRequestFactoryForLink() {
HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
factory.setReadTimeout(2000);
factory.setConnectTimeout(5000);
return factory;
}
@Bean
public RestTemplate restTemplateForLink(ClientHttpRequestFactory httpRequestFactoryForLink) {
return new RestTemplate(httpRequestFactoryForLink);
}
I would appreciate any insights into the potential causes of this issue.