0

I'm using curl to download a 10MB file over a poor connection.

I use the -C - option in a loop to resume the download if it breaks (see the snippet below).

The resuming works when I test locally and interrupt the download - after restarting the script, it will resume the download.

However, in some IoT devices in the field, midway through the transfer, curl errors out with a message like curl: (18) transfer closed with 2674439 bytes remaining to read.

After that, the download is resumed and completes - but the file is corrupt. This has happened multiple times on different devices, which have in common that they are in China and have poor connection (the curl download takes longer).

How is it possible for the download to be corrupt, given the TCP protocol uses checksums to verify the packets? Am I missing something, maybe I'm using an incompatible set of options with curl?

downloadFileResume() {
    # $1 contains the URL to download
    local fileName=$(basename "$1")
    for (( i=0; i<=$numCurlRetries; i++ ))
    do
        # capture curl http code (-w %{http_code}) in stdout
        curlHttpCode=$(curl -kSLf -u "$REPO_USERNAME:$REPO_PASSWORD" -w %{http_code} -O -C - "$1")
        local res=$?
        [ $res -eq 0 ] && return
        # curl gives error when attempting to resume a completely downloaded file, it's a 416 error from the server
        [ $curlHttpCode -eq 416 ] && echo "Error 416, assuming file is downloaded completely" && return
    done
    echo "failed to download $fileName"
    return 1
}

Many thanks, Mihail

Andrew Schulman
  • 8,811
  • 21
  • 32
  • 47
Mihail Kostira
  • 101
  • 2

1 Answers1

0

cURL error 18 happens when the server promised a certain size of content but sending more or less of data to the client. This can happen, for example:

  1. Server sends a Content-Length header and a body content with mismatching size (probably interrupted connection)
  2. Server sends a Transfer-Encoding: chunked but ended the chunk prematurely.

curl: (18) transfer closed with 2674439 bytes remaining to read

With that error message, seems that it's the first case. Try using --ignore-content-length with your cURL command line, or use a download manager, something like wget or aria2.

mforsetti
  • 2,666
  • 2
  • 16
  • 20
  • Thanks, I'll try the `--ignore-content-length`. The issue is not the `error 18` per se but rather the corrupted file after the download resumes - could that be caused by 1) or 2)? I can't use `wget` or `aria2` unfortunately, I'm limited to whatever is already installed. – Mihail Kostira Feb 18 '21 at 15:10