Download files from a list if not already downloaded

Question

I can do this in c#, and the code is pretty long.

Would be cool if someone can show me how this would be done via python.

Pseudo code is:

url: www.example.com/somefolder/filename1.pdf

1. load file into an array (file contains a url on each line)
2. if file e.g. filename1.pdf doesn't exist, download file

The script can be in the following layout:

/python-downloader/
/python-downloader/dl.py
/python-downloader/urls.txt
/python-downloader/downloaded/filename1.pdf

score 23 · Accepted Answer · edited Jul 04 '10 at 03:05

23

This should do the trick, although I assume that the urls.txt file only contains the url. Not the url: prefix.

import os
import urllib

DOWNLOADS_DIR = '/python-downloader/downloaded'

# For every line in the file
for url in open('urls.txt'):
    # Split on the rightmost / and take everything on the right side of that
    name = url.rsplit('/', 1)[-1]

    # Combine the name and the downloads directory to get the local filename
    filename = os.path.join(DOWNLOADS_DIR, name)

    # Download the file if it does not exist
    if not os.path.isfile(filename):
        urllib.urlretrieve(url, filename)

edited Jul 04 '10 at 03:05

Grant Paul

5,852
2
33
36

answered Jul 04 '10 at 01:13

Wolph

78,177
11
137
148

wow, that is amazingly succinct! I'm begging to see what all the hype is about! thanks man! – Blankman Jul 04 '10 at 01:15
2

Instead of splitting on '/', use os.path.basename(url). – TravisThomas May 07 '17 at 18:36
1

For Python 3, replace last line with: urllib.request.urlretrieve(url, filename) – relizt May 24 '22 at 12:46

Brandon Amos · Answer 2 · 2013-04-07T07:05:00.330

14

Here is a slightly modified version of WoLpH's script for Python 3.3.

#!/usr/bin/python3.3
import os.path
import urllib.request

links = open('links.txt', 'r')
for link in links:
    link = link.strip()
    name = link.rsplit('/', 1)[-1]
    filename = os.path.join('downloads', name)

    if not os.path.isfile(filename):
        print('Downloading: ' + filename)
        try:
            urllib.request.urlretrieve(link, filename)
        except Exception as inst:
            print(inst)
            print('  Encountered unknown error. Continuing.')

edited Apr 07 '13 at 07:05

answered Apr 07 '13 at 04:56

Brandon Amos

940
1
9
19

you should add PATH to script. with PATH = "./downloads" – Tech Jan 13 '21 at 06:56

score 2 · Answer 3 · answered Jul 04 '10 at 01:14

It's less code in Python, you could use something like this:

import urllib2
improt os

url="http://.../"
# Translate url into a filename
filename = url.split('/')[-1]

if not os.path.exists(filename)
  outfile = open(filename, "w")
  outfile.write(urllib2.urlopen(url).read())
  outfile.close()

Download files from a list if not already downloaded

3 Answers3

Linked