0

I try to get the big data from FRED (Federal Reserve Economic Data) using java but there are some problems. Belows are the sample codes.

for(US_STATE state : US_STATE.values()) {   // for every us-states loop

    String fredUrl = "https://api.stlouisfed.org/fred/series/search?search_text=Unemployment Rate in " + state.toString() + &api_key=**********&file_type=json";
    ObjectMapper mapper = new ObjectMapper();
    JsonNode rootNode = mapper.readTree(new URL(fredUrl)); // request url in this line
    Thread.sleep(500);  // insert sleep line to avoid the http errors
    ArrayNode nodeSeriess = (ArrayNode)rootNode.get("seriess");
    // I make the jsonnode stream , extract series_id from rootNode and make another url string
    Stream<JsonNode> elementStream = StreamSupport.stream(nodeSeriess.spliterator(), false);
    elementStream.flatMap(pojo -> {
        String observUrl = "https://api.stlouisfed.org/fred/series/observations?series_id=" + pojo.getSeriesId() + "&api_key=***********&file_type=json";
        // I request url again in the stream
        JsonNode nodeValue = mapper.readTree(new URL(observUrl));
        Thread.sleep(500);  // insert sleep line to avoid the http errors
        ArrayNode nodeValueObserv = (ArrayNode)nodeValue.get("observations");
        ...... // processing data from this line
   });

As you see, my codes execute too much Java URL api. So I think I face the HTTP response code: 504 for URL.

I found with googling that the 504 error is not related with client codes, but when I execute too much Java URL, I always receive this error. I want to know how to avoid this error programmatically. I am afraid my Java program calls the Java URL too much in for loop.

halfer
  • 19,824
  • 17
  • 99
  • 186
Joseph Hwang
  • 1,337
  • 3
  • 38
  • 67
  • it is known as **Gateway Timeout** error. it happens when you don't get a response in time from the upstream server that it needed in order to complete the request. – priyranjan Nov 29 '20 at 03:01
  • Thanks for your reply. This error is brought randomly from server. Is there any way to avoid this error programmatically on client side? – Joseph Hwang Nov 29 '20 at 03:45
  • you mean to say everything is correct from your side n you are getting response but sometimes not getting response from client side? ryt? and you want to handle this error n throw custom error response? – priyranjan Nov 29 '20 at 03:49
  • I mean everything is correct from my side and I am getting response but sometimes not getting response from client side. I wonder if my java code architecture could bring this 504 http error. – Joseph Hwang Nov 29 '20 at 03:56
  • you can make use of @ControllerAdvice and you can catch exception there. I think this link might help you https://www.baeldung.com/exception-handling-for-rest-with-spring – priyranjan Nov 29 '20 at 03:59

0 Answers0