I am trying to use wget to download from my AWS ubuntu server with apache2. I have tried several different options but they all result in either a file by the directory name, or an index.html file.
There are 3 pictures and an ogg format video in the directory.
This is the wget I am using:
wget -N -np http://domainorIP/dir/dir2/dirtodownload/
By default, wget only downloads the URL you give it, not any resources linked from an HTML page. To follow these links, you need the -r (or --recursive) option.
There is also the --page-requisites option which downloads any resources necessary for displaying the page you give it, such as inline images and stylesheets.
There are also "grouped" options such as --mirror which include -r along with several other options that are useful for mirroring.
The man page is enormous because wget has a lot of features. The section on Recursive Retrieval Options would be well worth your time to read.
Download a CLI browser and download the desired files, you can use ELinks or W3M
apt-get install elinks w3m
I've never had success in getting either curl or wget to download files which are being served from an Apache server where indexing directories is enabled. I'm thinking this is your problem as well. The directories show up like this when you browse them:
In Apache they're enabled like this for example:
<Directory /var/www/domain.com/pdfs>
Options Indexes FollowSymLinks
</Directory>
Using the shell you're options are limited to getting the list of files and then downloading them one at a time using a command like this one:
% URL="http://www.lamolabs.org/blog/wp-content/uploads/2012/10/"
% curl -s $URL | \
grep "href" | \
grep -v "C=D;O=A" | \
sed "s#^.*href=\"#$URL#" | \
sed 's/">.*$//' | \
xargs wget
Breaking this down:
- URL="..." - is the URL I want to download files from
- curl -s $URL - get's the contents of the index.html generated by Apache
- grep "href" - get lines that contain only href
- grep -v "C=D;O=A" - eliminate the header bar line generated by Apache
- sed "s#^.*href=\"#$URL#" - replace .*href=" lines with URL
- sed 's/">.*$//' - remove trailing characters >.*$
- xargs wget - download each file using wget
You can run this as a single command:
url="http://www.lamolabs.org/blog/wp-content/uploads/2012/10/"; curl -s $url | grep "href" | grep -v "C=D;O=A" | sed "s#^.*href=\"#$url#" | sed 's/">.*$//' | xargs wget
Running it results in the following files getting downloaded:
% ls -l
total 1652
-rw-rw-r-- 1 saml saml 1351400 Oct 8 23:35 Anatomy-of-the-Linux-file-system.mht
-rw-rw-r-- 1 saml saml 485 Oct 11 00:42 eratosthenes_prime_sieve.pl_.txt
-rw-rw-r-- 1 saml saml 27191 Oct 3 21:42 Selection_005-150x150.png
-rw-rw-r-- 1 saml saml 24202 Oct 3 21:42 Selection_005.png
-rw-rw-r-- 1 saml saml 27141 Oct 3 21:42 Selection_006-150x150.png
-rw-rw-r-- 1 saml saml 24906 Oct 3 21:42 Selection_006.png
-rw-rw-r-- 1 saml saml 25783 Oct 3 22:17 Selection_007-150x150.png
-rw-rw-r-- 1 saml saml 111915 Oct 3 22:17 Selection_007-650x180.png
-rw-rw-r-- 1 saml saml 48109 Oct 4 09:57 Selection_007-e1349359020755.png
-rw-rw-r-- 1 saml saml 29336 Oct 3 22:17 Selection_007.png
