1

It appears to me that my server only allows 60 files to be downloaded per second, but I have 63 of them - all tiny, YAML files. As a result, the last 3 files don't get downloaded and throw error 503. I am using Baeldung's NIO example:

public static void downloadWithJavaNIO(String fileURL, String localFilename) throws MalformedURLException {
        String credit = "github.com/eugenp/tutorials/blob/master/core-java-modules/core-java-networking-2/src/main/java/com/baeldung/download/FileDownload.java";

        URL url = new URL(fileURL);
        try (
            ReadableByteChannel readableByteChannel = Channels.newChannel(url.openStream());
            FileOutputStream fileOutputStream = new FileOutputStream(localFilename);
            FileChannel fileChannel = fileOutputStream.getChannel()
        ) {
            fileChannel.transferFrom(readableByteChannel, 0, Long.MAX_VALUE);
            fileOutputStream.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
        
    }

I was thinking of saving currentTimeMillis somewhere and checking if a second had passed by the time the 61th file is pending. But are there any other good ideas?

jam berry
  • 25
  • 5

1 Answers1

0

FileChannel.transferFrom: An invocation of this method may or may not transfer all of the requested bytes.

So I am not sure whether it works in all cases. Maybe with those small files.

I would first try a correct (non-optimized) version:

public static void downloadWithJavaNIO(String fileURL, String localFilename)
        throws MalformedURLException {
    URL url = new URL(fileURL);
    Path targetPath = Paths.get(localFilename);
    try (InputStream in = url.openStream()) {
        Files.copy(in, targetPath);
    } catch (IOException e) {
        System.getLogger(getClass().getName()).log(Level.ERROR, fileURL, e);
    }        
}
  • How fast now? Does it still need throttling? (slowing down, waiting)
  • Does the capacity error still persist?
Joop Eggen
  • 107,315
  • 7
  • 83
  • 138
  • Hey, thanks for the reply! I managed to use a Thread#sleep in order to wait for a second. It indeed was the server problem that only allowed 60 usages per second. I found no harm in my Thread#sleep, because this particular case only happens once while ever using my application and only in the very beginning of it, so nothing would be damaged. If I will ever decide to get rid of that hacky way, I'll look into your response. – jam berry May 30 '22 at 15:01
  • @jamberry thanks for the undeserved flowers (points). 60 Downloads per second can be solved differently. Try using one single URL connection on `url` and then several InputStreams on the same connection. – Joop Eggen Jun 01 '22 at 12:23