How to check content-length match after download image in java Nio?

Question

    static void downloadWebp(String url) {
        URL url1;
        try {
            url1 = new URL(url);
        } catch (MalformedURLException e) {
            throw new RuntimeException(e);
        }
        {
            // download webp
            ReadableByteChannel readableByteChannel;
            try {
                readableByteChannel = Channels.newChannel(url1.openStream());
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
            Path path = Paths.get(System.getProperty("user.home"), "image", Paths.get(url1.getPath()).getFileName().toString() + 1);
            try (FileOutputStream fileOutputStream = new FileOutputStream(path.toAbsolutePath().toString())) {
                fileOutputStream.getChannel().transferFrom(readableByteChannel, 0, Long.MAX_VALUE);
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
            // Can I check if download file's length matches the content-length header ?
        }
    }

Call like

downloadWebp("https://cdn.discordapp.com/attachments/1098794840742961242/1100590092957003877/Jeffery_cherry_blossoms_blooming_in_the_rain_by_Yoko_Ishii_wate_8a2f060e-32dc-4567-a3df-8bfdbb56adfd.webp");

As In previous java code, I had download a image, how can I double check if downloaded file's length matches the content-length header's value ?

As I found some file not really fully downloaded after fileOutputStream.getChannel().transferFrom(readableByteChannel, 0, Long.MAX_VALUE); returns, I have to retry in that case.

If it doesn't you should get an exception of some kind, but instead of using `URL.openStream()`, you should create an `HttpURLConnection` from the `URL`, get your input stream from that, and also the `Content-length` header. — user207421, Apr 26 '23 at 08:13
No, it didn't throw an exception, I just see the method wait for ~7 minutes and return, but the saved file has a shorter length than specified in `content-length` header. — djy, Apr 26 '23 at 08:24
So the sender didn't send it all. Maybe the problem is at that end. But you should always exert a read timeout. And actually if you study the Javadoc you should conclude that `transferFrom/To()` need to be called in a loop. — user207421, Apr 26 '23 at 10:07

ursa · Accepted Answer · 2023-06-15T08:54:45.107

Try this:

public static void main(String[] args) throws Exception {
    downloadWebp(
        Files.createTempFile("sprites", "svg"),
        URI.create("https://cdn.sstatic.net/Img/unified/sprites.svg?v=fcc0ea44ba27"),
        null // new Proxy(Proxy.Type.HTTP, new InetSocketAddress("<proxy-host-if-used>", 8080))
    );
}

private static void downloadWebp(Path path, URI uri, Proxy proxy) throws Exception {
    HttpURLConnection connection = (HttpURLConnection) (proxy == null 
        ? uri.toURL().openConnection() : uri.toURL().openConnection(proxy));
    try {
        //connection.setConnectTimeout(30_000);
        //connection.setReadTimeout(30_000);
        connection.connect();
        try (
            ReadableByteChannel readableByteChannel = Channels.newChannel(connection.getInputStream());
            FileChannel fileChannel = FileChannel.open(path, StandardOpenOption.WRITE)
        ) {
            int length = connection.getHeaderFieldInt("Content-Length", -1);
            System.out.println("Content-Length: " + length);
            while (length > 0) {
                long written = fileChannel.transferFrom(readableByteChannel, 0, length /*Math.min(length, 2048)*/);
                length -= written;
                System.out.println("written: " + written);
            }
        }
    } finally {
        connection.disconnect();
    }
}

You will get both content-length and actually transferred bytes:

Content-Length: 7542
written: 7542

As @user207421 mentioned in comments it's true, you need to use transferFrom in a loop.

It’s pointless to create a variable `url1`, when you keep repeating `uri.toURL()` each time you need the `URL` instead of using the `url1` variable. — Holger, Jun 13 '23 at 15:43

score 1 · Answer 2 · answered Apr 26 '23 at 13:12

1

If you just want to be sure that you are downloading the entire image, you can use Files.copy:

try (InputStream source = url1.openStream()) {
    Files.copy(source, path);
}

You can also use the java.net.http package:

HttpClient.newHttpClient().send(
    HttpRequest.newBuilder(url1.toURI()).build(),
    HttpResponse.BodyHandlers.ofFile(path));

answered Apr 26 '23 at 13:12

VGR

40,506
4
48
63

Doesn't answer the question. He wants to *check* the result against the content-length. There is nothing magical about using `Files.copy()` here. – user207421 Apr 27 '23 at 01:16
@user207421 I believe this is an XY problem. djy wouldn’t need to check the content length if they weren’t using FileChannel.transferFrom. – VGR Apr 27 '23 at 01:18
He states he is seeing a delay of ~7 minutes before his method returns. You can't possibly attribute that to `transferFrom()`, and you can't claim without explanation that `Files.copy()` or your other solution will magically cure it. – user207421 Apr 27 '23 at 02:23
thank you, i tried to refactor my code with Files.copy, wish the wired bug go away. ursa answered the question though – djy Apr 27 '23 at 11:31

How to check content-length match after download image in java Nio?

2 Answers2