How do I mirror a directory with wget without creating parent directories?

Question

I want to mirror a folder via FTP, like this:

wget --mirror --user=x --password=x ftp://ftp.site.com/folder/subfolder/evendeeper

But I do not want to create a directory structure like this:

ftp.site.com -> folder -> subfolder -> evendeeper

I just want:

evendeeper

And anything below it to be the resulting structure. It would also be acceptable for the contents of evendeeper to wind up in the current directory as long as subdirectories are created for subdirectories of evendeeper on the server.

I am aware of the -np option, according to the documentation that just keeps it from following links to parent pages (a non-issue for the binary files I'm mirroring via FTP). I am also aware of the -nd option, but this prevents creating any directory structure at all, even for subdirectories of evendeeper.

I would consider alternatives as long as they are command-line-based, readily available as Ubuntu packages and easily automated like wget.

With `lftp`: `lftp -c mirror ` – HappyFace Aug 29 '21 at 16:07 — HappyFace, Aug 29 '21 at 16:07

score 86 · Accepted Answer · edited Oct 04 '17 at 19:35

86

For a path like: ftp.site.com/a/b/c/d

-nH would download all files to the directory a/b/c/d in the current directory, and -nH --cut-dirs=3 would download all files to the directory d in the current directory.

edited Oct 04 '17 at 19:35

waldyrious

3,683
4
33
41

answered Feb 18 '11 at 18:56

vs_inf

876
6
2

4

Thanks, --cut-dirs was the silver bullet I was looking for. – Tom Boutell Feb 20 '11 at 13:33
5

I would change to --cut-dirs=4 if you need all files from directory d – Mike Oct 08 '13 at 14:40
5

-nH would only ignore the host directory for me. You can set --cut-dirs=100 for preventing almost all the pre directorys -- tick – Rex Feb 19 '16 at 09:10
5

It's fairly obvious with an URL like `ftp.site.com/a/b/c/d/` one only wants the content in `d/` and its subfolders. Is there an auto-cut option or something similar for wget, that autodetects the cut depth? – con-f-use Mar 14 '16 at 12:48
2

@con-f-use I was hoping the `--cut-dirs=100` trick suggested by @Rex (and in [Tony TCG's answer below](https://stackoverflow.com/a/23185705/266309)) would achieve this, but alas, it works like `-nd`, i.e. it flattens the entire hierarchy, including subdirectories below the one you're downloading. Looks like there's no automatic solution using wget alone... – waldyrious Oct 04 '17 at 14:13

pr-pal · Answer 2 · 2019-09-07T15:15:08.767

I had a similar requirement and the following combination seems to be the perfect choice:

In the below example, all the files in http://url/dir1/dir2 (alone) are downloaded to local directory /dest/dir

wget  -nd -np -P /dest/dir --recursive http://url/dir1/dir2

Thanks @ffledgling for the hint on "-nd"

For the above example:

wget -nd -np --mirror --user=x --password=x ftp://ftp.site.com/folder/subfolder/evendeeper

Snippets from manual:

   -nd
   --no-directories
       Do not create a hierarchy of directories when retrieving recursively.  With this option turned on, all files will get saved to the current directory, without clobbering (if a name shows up more than once, the
       filenames will get extensions .n).


   -np
   --no-parent
       Do not ever ascend to the parent directory when retrieving recursively.  This is a useful option, since it guarantees that only the files below a certain hierarchy will be downloaded.

score 10 · Answer 3 · answered Feb 18 '11 at 15:39

10

-np (no parent) option will probably do what you want, tied in with -L 1 (I think, don't have a wget install before me), which limits the recursion to one level.

EDIT. ok. gah... maybe I should wait until I've had coffee.. There is a --cut or similar option, which allows you to "cut" a specified number of directories from the output path, so for /a/b/c/d, a cut of 2 would force wget to create c/d on your local machine

answered Feb 18 '11 at 15:39

Marc B

356,200
43
426
500

6

You mean: `--cut-dirs=2` — Also, `-L` is `--relative`, so you probably meant `-l` or `--level=1` – Denilson Sá Maia Feb 18 '11 at 15:51

score 9 · Answer 4 · edited Oct 04 '17 at 15:10

9

Instead of using:

-nH --cut-dirs=1

use:

-nH --cut-dirs=100

This will cut more directories and no folders will be created.

Note: 100 = the number of folders to skip creating. You can change 100 to any number.

edited Oct 04 '17 at 15:10

waldyrious

3,683
4
33
41

answered Apr 20 '14 at 17:46

Tony TCG

313
3
8

10

Why not just use -nd instead then? – ffledgling Mar 31 '15 at 01:15

How do I mirror a directory with wget without creating parent directories?

4 Answers4