-2

I need to download all files from a FTP server but the server has a lookup limitation on 10,000 which complicates things.

I can't use wget -m ftp://username:password@server-adress since it will only download the first 10,000 files. I can use mget like mget 20[12]*, mget 20[34]* etc. but it is cumbersome AND some downloads fails without me knowing which ones.

Now I have come across lftp. I was getting my hopes up for this command

mirror --use-pget=10 --only-missing --no-symlinks

But it doesn't download anything, it just outputs

To be removed: 0 directories, 70695 files, 0 symlinks

Now I am clueless how to actually download all files from a FTP server with a limit

EDIT

Now I went ahead and created a script that produces a missing.txt with a list of files that I still need to download. How can I use that file to download the files one by one? The list is \n delimited.

mr.bjerre
  • 2,384
  • 2
  • 24
  • 37
  • 1
    1) Your question is off-topic. 2) If the server won't give you a list of all files, there's nothing you can do about it. Changing FTP client cannot help. All you can do is to guess the names (like your attempt with `mget 20[12]*`) or get the names using other means (not FTP) – Martin Prikryl Apr 04 '19 at 07:37
  • You can also move the already downloaded files (if possible) to a different folder and then repeat the download. Then you can move the files back. – Martin Prikryl Apr 04 '19 at 08:04
  • @MartinPrikryl thanks for your reply. Just out of interest, why is is off-topic? I see downvotes but I have no clue why. – mr.bjerre Apr 04 '19 at 09:16
  • Your question (at least your original question) is not about programming. – Martin Prikryl Apr 04 '19 at 09:18
  • I partly agree. It just so turned out that programming was the solution but I appreciate the feedback. – mr.bjerre Apr 04 '19 at 09:21

1 Answers1

0

So I went ahead and made a script of all the missing files called missing.txt. Then I created a loop to download each file. And now it is actually downloading one by one. The download.sh script looks like this

#!/bin/bash
HOST='HOSTNAME'
USER='USERNAME'
PASSWD='PASSWORD'

echo open $HOST > ftp.txt
# echo ascii >> ftp.txt
echo user $USER $PASSWD >> ftp.txt
# echo prompt no >> ftp.txt

while read p; do
    echo "get $p"
done <missing.txt >> ftp.txt

echo bye >> ftp.txt
ftp -in < ftp.txt
rm ftp.txt

And then it is just a matter of running ./download.sh.

mr.bjerre
  • 2,384
  • 2
  • 24
  • 37