99

When downloading a file using curl, how would I follow a link location and use that for the output filename (without knowing the remote filename in advance)?

For example, if one clicks on the link below, you would download a filenamed "pythoncomplete.vim." However using curl's -O and -L options, the filename is simply the original remote-name, a clumsy "download_script.php?src_id=10872."

curl -O -L http://www.vim.org/scripts/download_script.php?src_id=10872

In order to download the file with the correct filename you would have to know the name of the file in advance:

curl -o pythoncomplete.vim -L http://www.vim.org/scripts/download_script.php?src_id=10872

It would be excellent if you could download the file without knowing the name in advance, and if not, is there another way to quickly pull down a redirected file via command line?

Nick S.
  • 1,643
  • 1
  • 12
  • 13

10 Answers10

180

The remote side sends the filename using the Content-Disposition header.

curl 7.21.2 or newer does this automatically if you specify --remote-header-name / -J.

curl -O -J -L $url

The expanded version of the arguments would be:

curl --remote-name --remote-header-name --location $url
Mehrad Mahmoudian
  • 3,466
  • 32
  • 36
jmanning2k
  • 9,297
  • 4
  • 31
  • 23
  • 17
    This is not working for me, the saved file name is still in download_script.php?src_id=10872 format. I am running curl 7.30.0 on my macbook pro – ilight Nov 01 '13 at 06:49
  • 1
    Work fine with curl 7.38.0 – ismail Oct 08 '14 at 17:20
  • 3
    Not working with curl on OSX: curl 7.42.1 (x86_64-apple-darwin14.4.0) libcurl/7.42.1 SecureTransport zlib/1.2.5 libssh2/1.5.0 – joelparkerhenderson May 31 '15 at 21:40
  • 1
    Not working on OSX El Capitan: curl 7.43.0 (x86_64-apple-darwin15.0) libcurl/7.43.0 SecureTransport zlib/1.2.5 – jgp Feb 12 '16 at 19:19
  • Works fine for me on El Capitan, same version string. Using the vim.org url above with -J results in: "curl: Saved to filename 'pythoncomplete.vim'", which is as expected. Are you using system curl, or something from brew/macports/fink @jgp? – jmanning2k Feb 16 '16 at 16:24
  • 4
    As @Diskutant "comments in an-answer" below. This fails in curl in the somewhat common case of `Content-Disposition: attachment; filename = foo.txt`. The spaces around the `=` cause it (see: https://gist.github.com/jnewman/23c993ef50bcd69f9086fcd4e2594928) – fncomp Apr 07 '16 at 20:45
  • 2
    In case someone else comes across this if you are using `localhost` you have to make it `http://localhost` or the `-J` is ignored – mvndaai Nov 17 '16 at 19:10
  • Full command: `curl --remote-name --remote-header-name --location $url` – Simon Perepelitsa Nov 08 '17 at 15:07
  • Thanks! Worked like a charm on macOS 10.14.6 Catalina with curl 7.54.0 Also `--remote-name` worked perfect in conjunction with a URL containing a sequential number pattern, in my example `curl -O -J -L "http://other-domain-but-path-like-this.com/index.php?controller=attachment&id_attachment=[0-99]"` – porg Aug 27 '19 at 22:16
  • It worked here. I tested using Golang gin framework for serving files with header "Content-Disposition": `attachment; filename="filename.pdf"`. – Yang Yu Aug 07 '20 at 09:00
  • Not work when `Content-Disposition: attachment; filename*=utf-8''filename`, in this case, use `wget --content-disposition URL` – Ren Sep 09 '21 at 02:59
  • This curl answer work for example for https://archlinux.org/packages/community/x86_64/geckodriver/download/ – Velkan Apr 08 '22 at 08:36
35

If you have a recent version of curl (7.21.2 or later), see @jmanning2k's answer.

I you have an older version of curl (like 7.19.7 which came with Snow Leopard), do two requests: a HEAD to get the file name from response header, then a GET: 

url="http://www.vim.org/scripts/download_script.php?src_id=10872"
filename=$(curl -sI  $url | grep -o -E 'filename=.*$' | sed -e 's/filename=//')
curl -o $filename -L $url
Community
  • 1
  • 1
David J. Liszewski
  • 10,959
  • 6
  • 44
  • 57
  • 5
    Thanks! It came back with a question mark at the end of the filename, so I added a `tr -d '\r\n'` between your grep and sed commands, and it worked marvelously. – Nick S. Jul 30 '11 at 20:03
  • In cygwin, I get a file name called -L :(. I used curl --remote-header-name --remote-name URL which seemed to work. – Sun Sep 27 '14 at 07:33
  • 2
    WIth curl 7.26.0 I could get file name by doing ´curl -sI $url | grep -oP '(?<=Location: )[^\s]+' | xargs basename´. – aesede Sep 15 '15 at 01:47
  • 2
    The first `curl` would probably also need the `-L` option. – TNT Jun 12 '17 at 15:37
  • 2
    I had to use `filename=$(curl -sI $url | grep -o -E 'location:.*$' | sed -e 's/location:=//' | xargs basename)` – hoijui May 28 '19 at 08:41
21

If you can use wget instead of curl:

wget --content-disposition $url
JacekM
  • 4,041
  • 1
  • 27
  • 34
  • This works for a url such as: https://plex.tv/downloads/latest/1?channel=8&build=linux-ubuntu-x86_64&distro=ubuntu&X-Plex-Token=qEZ52typa192wkxKmQMN – parsecpython Sep 07 '18 at 22:14
  • 1
    You are a f-ing legend!! I was monkey around with this A LOT and this worked out of the gates. Thanks. ProTip: If you have a lot of URLs, you can create a file like `urls.txt`, fill it with URLs (one per line, no surrounding quotes) and then use `wget --content-disposition -i urls.txt` to download them all in sequence. Boom! – Joshua Pinter Jan 10 '22 at 20:45
12

I wanted to comment to jmanning2k's answer but as a new user I can't, so I tried to edit his post which is allowed but the edit was rejected saying it was supposed to be a comment. sigh

Anyway, see this as a comment to his answer thanks.

This seems to only work if the header looks like filename=pythoncomplete.vim as in the example, but some sites send a header that looks like filename*=UTF-8' 'filename.zip' that one isn't recognized by curl 7.28.0

Antony Hatchkins
  • 31,947
  • 10
  • 111
  • 111
MetalSnake
  • 277
  • 3
  • 14
6

I wanted a solution that worked on both older and newer Macs, and the legacy code David provided for Snow Leopard did not behave well under Mavericks. Here's a function I created based on David's code:

function getUriFilename() {
    header="$(curl -sI "$1" | tr -d '\r')"

    filename="$(echo "$header" | grep -o -E 'filename=.*$')"
    if [[ -n "$filename" ]]; then
        echo "${filename#filename=}"
        return
    fi

    filename="$(echo "$header" | grep -o -E 'Location:.*$')"
    if [[ -n "$filename" ]]; then
        basename "${filename#Location\:}"
        return
    fi

    return 1
}

With this defined, you can run:

url="http://www.vim.org/scripts/download_script.php?src_id=10872"
filename="$(getUriFilename $url)"
curl -L $url -o "$filename"
Chaim Leib Halbert
  • 2,194
  • 20
  • 23
2

Please note that certain malconfigured webservers will serve the name using "Filename" as key, where RFC2183 specifies it should be "filename". curl only handles the latter case.

drater
  • 21
  • 1
2

I had the same Problem like John Cooper. I got no filename but a Location File name back. His answer also worked but are 2 commands. This oneliner worked for me....

url="https://download.mozilla.org/?product=firefox-latest-ssl&os=linux64&lang=de";url=$(curl -L --head -w '%{url_effective}' $url 2>/dev/null | tail -n1) ; curl -O $url

Stolen and added some stuff from https://unix.stackexchange.com/questions/126252/resolve-filename-from-a-remote-url-without-downloading-a-file

Jackfritt
  • 27
  • 6
  • this question gave me a lot of headache, and I have found no solution rather than: 1) using `wget -O` or ... using this answer!!! My opinion, THIS IS THE SOLUTION! I will put therefore a function below, to avoid further headaches... – loretoparisi May 19 '21 at 21:30
1

An example using the answer above for Apache Archiva artifact repository to pull latest version. The curl returns the Location line and the filename is at the end of the line. Need to remove the CR at end of file name.

url="http://archiva:8080/restServices/archivaServices/searchService/artifact?g=com.imgur.backup&a=snapshot-s3-util&v=LATEST"
filename=$(curl --silent -sI -u user:password $url | grep Location | awk -F\/ '{print $NF}' | sed 's/\r$//')
curl --silent -o $filename -L -u user:password $url
John Cooper
  • 151
  • 1
  • 1
1

instead of applying grep and other Unix-Fu operations, curl ships with a builtin "Write Out" option variable[1] specifically for such a case, e.g.

$ curl -OJsL "http://www.vim.org/scripts/download_script.php?src_id=10872" -w "%{filename_effective}"
pythoncomplete.vim

[1] https://everything.curl.dev/usingcurl/verbose/writeout#available-write-out-variables

lwei
  • 55
  • 2
  • 6
  • If the goal is to just save the file with a correct filename, using `-w` and this write-out variable to output file name to stdout isn't needed. The key option is `-O` (`--remote-name`) which tells that saved file should have the same name as remote. See the accepted answer. – YurkoFlisk Aug 06 '22 at 01:16
0

Using the solution proposed above, I wrote this helper function curl2file.

[UPDATED]

function curl2file() {
    url=$1
    url=$(curl -o /dev/null -L --head -w '%{url_effective}' $url 2>/dev/null | tail -n1) ; curl -O $url
}

Usage:

curl2file https://cloud.tsinghua.edu.cn/f/4666d28af98a4e63afb5/?dl=1
loretoparisi
  • 15,724
  • 11
  • 102
  • 146
  • 1
    You should use `-o /dev/null` when getting the redirect URL so that you don't need to process the output (`-o` affects the `--head` output but not the `-w` output). Also, you should not redirect stderr because you then hide any errors that occur. – cjs Sep 08 '22 at 03:47