How to exclude certain directories while using wget?

Question

I'd like to download a directory from a FTP, which contains some source codes. Initially, I did this:

wget -r ftp://path/to/src

Unfortunately, the directory itself is a result of a SVN checkout, so there are lots of .svn directories, and crawling over them would take longer time. Is it possible to exclude those .svn directories?

score 24 · Answer 1 · edited Dec 20 '15 at 18:05

24

wget -X directory_to_exclude[,other_directory_to_exclude] -r ftp://URL_ftp_server

SERVER
    |-logs
    |-etc
    |-cache
    |-public_html
      |-images
      |-videos ( want to exclude )
      |-files
      |-audio  (want to exclude)

wget -X /public_html/videos,/public_html/audio ftp:SERVER/public_html/*

edited Dec 20 '15 at 18:05

Jakuje

9,715
2
42
45

answered Dec 20 '15 at 15:59

josejavierfm

351
2
2

1

This actually seems to be the correct comment. The exclude-directories switch does not do patterns/regexes, so you have to include the entire path (from root). The higher-voted answer seems to be incorrect for all but the trivial cases. – John O Sep 28 '16 at 19:26
1

I agree. Although the documentation states that wildcards can be used, I could not get them to work. – Diomidis Spinellis Jan 16 '17 at 17:32
2

wildcards work, but are interpreted first on shell level, so you have to escape them to work. – Joop Kiefte Oct 01 '17 at 19:20

score 6 · Answer 2 · answered Oct 25 '11 at 09:35

6

wget --exclude-directories=.svn -r ftp://path/to/src

answered Oct 25 '11 at 09:35

quanta

51,413
19
159
217

score 2 · Answer 3 · answered Apr 02 '21 at 22:29

I'd like to answer this a bit broader, because the subject of this question can be found via a search engine:

--exclude-directories=list expects absolute paths [1]. This means with host.org/fu/bar/ you have to write --exclude-directories=/fu/bar.

This can be a problem, if you always want to exclude a folder with a specific name, no matter where it is exactly (for example a 'thumbs' folder).

For this we can use --reject-regex [2] like this: --reject-regex="/thumbs/". Given this is now regex and not a comma-separated string list, we can exclude multiple folders via regex1|regex2|regex3: --reject-regex="/thumbs/|/css/". Keep in mind that certain characters like . have a special meaning in regex and need to be escaped to be part of a folder name: "/\.svn/".

How to exclude certain directories while using wget?

3 Answers3