wget ftp an entire site is very slow

Question

I'm trying to wget an entire site to transfer from Server A to Server B. It works, but incredibly slowly.

I tested this by uploading a 100mb psd file to my server, and grabbing it with wget ftp. Transfer speeds were 26.76mb/sec.

But when I try and grab the whole site (minus the psd file) it takes 5 minutes to transfer 92mb of html files. It's a wordpress install with a theme and images basically.

Is this because it opens a new connection for each file? I tried doing this with php's ftp library, but that took just as long if not longer.

scp, ssh are not an option because 98% of the time I don't own the server, so I have to work with some sort of ftp. I'm using:

wget -r ftp://user:pass@domain.com/dir is what I'm using.

score 0 · Answer 1 · answered May 03 '13 at 15:25

0

wget will try to keep the connection open, but even with a single connection there is a significant overhead at the server end to serve each request, which is most probably what's causing the slowdown you're seeing.

answered May 03 '13 at 15:25

Flup

7,978
2
32
43

score 0 · Accepted Answer · answered May 03 '13 at 15:26

0

It is always slower to transfer many files than it is to transfer one big file. Zip the contents of your website and the transfer will go much faster. You can use tar to do this.

answered May 03 '13 at 15:26

Stephen K

162
8

Can't zip, as I don't know if Server A has the library for it, and I don't have access other than ftp. I supposed I could log into the server, transfer a php file that would zip the contents, transfer that zip file, then delete with php file with unlink(_SELF_); or something. – David May 03 '13 at 15:28

score 0 · Answer 3 · answered May 03 '13 at 15:56

It's not as simple as -r I admit but you could try this: Make a list of the files, separate this list in e.g. six parts and start wget six times simultaneously each with a part of the list. A short look at the cumulated transfer speed should tell you whether the number of files is the problem.

score 0 · Answer 4 · answered May 03 '13 at 17:32

you can't speed up the transfer since it does a connection for each file, but try ftpcopy which allows you to mirror an ftp

from ftpcopy site:

Speed

ftpcopy needed about 110 seconds to copy my site. Subsequent runs which don't need to copy anything take about 1.04 seconds. mirror needed about 250 seconds to copy my site. Subsequent runs which don't need to copy anything take about 2.6 seconds.

wget ftp an entire site is very slow

4 Answers4