21

There is a directory that is being served over the net which I'm interested in monitoring. Its contents are various versions of software that I'm using and I'd like to write a script that I could run which checks what's there, and downloads anything that is newer that what I've already got.

Is there a way, say with wget or something, to get a a directory listing. I've tried using wget on the directory, which gives me html. To avoid having to parse the html document, is there a way of retrieving a simple listing like ls would give?

Asclepius
  • 57,944
  • 17
  • 167
  • 143
ajwood
  • 18,227
  • 15
  • 61
  • 104

8 Answers8

31

I just figured out a way to do it:

$ wget --spider -r --no-parent http://some.served.dir.ca/

It's quite verbose, so you need to pipe through grep a couple of times depending on what you're after, but the information is all there. It looks like it prints to stderr, so append 2>&1 to let grep at it. I grepped for "\.tar\.gz" to find all of the tarballs the site had to offer.

Note that wget writes temporary files in the working directory, and doesn't clean up its temporary directories. If this is a problem, you can change to a temporary directory: 

$ (cd /tmp && wget --spider -r --no-parent http://some.served.dir.ca/)
ajwood
  • 18,227
  • 15
  • 61
  • 104
  • @A-B-B Are you sure about that? The `--spider` option makes this not actually download anything – ajwood Oct 30 '16 at 16:01
  • I tried `wget --spider -r --no-parent https://www.kernel.org/pub/software/scm/git/` and it started to create a nested directory structure on disk – this won't work. I don't want anything written to disk, even if it's a single directory. – Asclepius Oct 30 '16 at 17:41
  • 1
    Oh yeah, seems `wget` needs to write intermediate files. It cleans up the files but leaves the directory tree.. Could you just cd into /tmp while it runs? `(cd /tmp && wget --spider -r --no-parent https://www.kernel.org/pub/software/scm/git/)` – ajwood Oct 30 '16 at 18:29
  • ( (cd /tmp && wget --spider -r --no-parent https://www.kernel.org/pub/software/scm/git/) 2>&1 | grep wall ) , grep works when 2>&1 is appended as explained in the answer. - It was not clear to me at first glance. – Michael Dimmitt Nov 30 '19 at 17:51
  • this command is doing what I want, but saving the output is driving me nuts. When I try to redirect STDOUT everything seems to go into oblivion. Looking for insights, I'm probably missing something obvious. – Aaron Feb 09 '20 at 04:40
  • @liang I don't know the details of what all it prints, but the interesting stuff for this purposes is on STDERR. Did you try `wget --spider -r --no-parent http://some.served.dir.ca/ 2>&1 | next_command ...` – ajwood Feb 09 '20 at 15:47
  • @ajwood yes, I tried that. I think my problem is executing that command in a subshell. I'm still playing around with it, and making some progress. I want to do post-processing on the grep results. For some clarity, I'm trying to extract the list of categories from Internet Archive and download the XML metadata. – Aaron Feb 09 '20 at 18:36
6

What you are asking for best served using FTP, not HTTP.

HTTP has no concept of directory listings, FTP does.

Most HTTP servers do not allow access to directory listings, and those that do are doing so as a feature of the server, not the HTTP protocol. For those HTTP servers, they are deciding to generate and send an HTML page for human consumption, not machine consumption. You have no control over that, and would have no choice but to parse the HTML.

FTP is designed for machine consumption, more so with the introduction of the MLST and MLSD commands that replace the ambiguous LIST command.

Remy Lebeau
  • 555,201
  • 31
  • 458
  • 770
  • 4
    Actually, HTTP *does* have this concept, it's called WebDAV, and it's an optional extension. See RFC 4918. – Julian Reschke Jul 01 '14 at 06:24
  • 4
    WebDAV runs on top of HTTP but is not part of HTTP itself. Just like HTTP runs on top of TCP but is not part of TCP itself. You cannot use WebDAV to talk to any arbitrary HTTP server. It has to be implemented and enabled by each server. Like you said, it is optional. – Remy Lebeau Jul 01 '14 at 07:30
  • 2
    It's optional, but the remainder of your comparison is misleading. TCP and HTTP are different networking layers, while PROPFIND and GET are in the exactly same layer. – Julian Reschke Jul 01 '14 at 07:40
2

The following is not recursive, but it worked for me:

$ curl -s https://www.kernel.org/pub/software/scm/git/

The output is HTML and is written to stdout. Unlike with wget, there is nothing written to disk.

-s (--silent) is relevant when piping the output, especially within a script that must not be noisy.

Whenever possible, remember not to use ftp or http instead of https.

Asclepius
  • 57,944
  • 17
  • 167
  • 143
1

If it's being served by http then there's no way to get a simple directory listing. The listing you see when you browse there, which is the one wget is retrieving, is generated by the web server as an HTML page. All you can do is parse that page and extract the information.

Optimal Cynic
  • 797
  • 3
  • 6
0

AFAIK, there is no way to get a directory listing like that for security purposes. It is rather lucky that your target directory has the HTML listing because it does allow you to parse it and discover new downloads.

Samuel
  • 16,923
  • 6
  • 62
  • 75
  • The reason the page is served in the first place is to provide users with a source for the software. If it's intended to be viewed in a browser, it only seems reasonable that one could expect to access it from a script. – ajwood Dec 21 '10 at 04:52
  • 1
    If there was an index.html, or a similar page, it would make sense to disallow directory listing for security reasons. It seems odd to me that if a directory is being served raw (well, having html generated to make it pretty) it should be fully accessible for something as harmless as a directory listing. – ajwood Dec 21 '10 at 04:55
0

You can use IDM (internet download manager)
It has a utility named "IDM SITE GRABBER" input the http/https URLs and it will download all files and folders from http/https protocol for you.

Bhaskara Arani
  • 1,556
  • 1
  • 26
  • 44
babak
  • 1
0

elinks does a halfway decent job of this. Just elinks <URL> to interact with a directory tree through the terminal.

You can also dump the content to the terminal. In that case, you may want flags like --no-references and --no-numbering.

manabear
  • 431
  • 3
  • 6
0

Use lftp:

LS_COLORS=no lftp -e 'cls -1; exit' 'https://cdn.kernel.org/pub/linux/kernel/v1.0/' 2>/dev/null
Piotr Henryk Dabrowski
  • 702
  • 5
  • 13