Spider a Website and Return URLs Only

Question

I'm looking for a way to pseudo-spider a website. The key is that I don't actually want the content, but rather a simple list of URIs. I can get reasonably close to this idea with Wget using the --spider option, but when piping that output through a grep, I can't seem to find the right magic to make it work:

wget --spider --force-html -r -l1 http://somesite.com | grep 'Saving to:'

The grep filter seems to have absolutely no affect on the wget output. Have I got something wrong or is there another tool I should try that's more geared towards providing this kind of limited result set?

UPDATE

So I just found out offline that, by default, wget writes to stderr. I missed that in the man pages (in fact, I still haven't found it if it's in there). Once I piped the return to stdout, I got closer to what I need:

wget --spider --force-html -r -l1 http://somesite.com 2>&1 | grep 'Saving to:'

I'd still be interested in other/better means for doing this kind of thing, if any exist.

See this question/answer for another way using a python script: http://stackoverflow.com/questions/9561020/how-do-i-use-the-python-scrapy-module-to-list-all-the-urls-from-my-website — Adam F, Mar 06 '12 at 04:30
possible duplicate of [Get a list of URLs from a site](http://stackoverflow.com/questions/857653/get-a-list-of-urls-from-a-site) — Ciro Santilli OurBigBook.com, Feb 01 '15 at 15:27

score 67 · Accepted Answer · edited May 18 '21 at 11:50

67

The absolute last thing I want to do is download and parse all of the content myself (i.e. create my own spider). Once I learned that Wget writes to stderr by default, I was able to redirect it to stdout and filter the output appropriately.

wget --spider --force-html -r -l2 $url 2>&1 \
  | grep '^--' | awk '{ print $3 }' \
  | grep -v '\.\(css\|js\|png\|gif\|jpg\)$' \
  > urls.m3u

This gives me a list of the content resource (resources that aren't images, CSS or JS source files) URIs that are spidered. From there, I can send the URIs off to a third party tool for processing to meet my needs.

The output still needs to be streamlined slightly (it produces duplicates as it's shown above), but it's almost there and I haven't had to do any parsing myself.

edited May 18 '21 at 11:50

Gustavo Straube

3,744
6
39
62

answered May 14 '10 at 11:52

Rob Wilkerson

40,476
42
137
192

5

`wget -r --spider -l1 -A mp3 http://example.com/page-with-mp3s 2>&1 | grep -Eio http.+mp3` was a good magic ticket for me. Thanks! – AKX Mar 14 '13 at 17:45
4

I typically pass that output to `sort | uniq` to remove duplicates, FYI. – AL the X Jan 22 '14 at 20:59
2

I know 5 years has passed since this answer but can you speed up the process? It takes seconds or even minutes for sites with 200 urls – erdomester Jun 15 '15 at 06:26
You can shorten the time greatly on this if you replace the first `grep` and `awk` commands with a single `egrep -o 'https?://[^ ]+'`. I too also recommend piping to `sort | uniq` because that can reduce the work of the third party tool on repeat URLs. – Volomike Apr 12 '18 at 00:31

Jay Taylor · Answer 2 · 2010-05-10T17:12:05.250

Create a few regular expressions to extract the addresses from all

<a href="(ADDRESS_IS_HERE)">.

Here is the solution I would use:

wget -q http://example.com -O - | \
    tr "\t\r\n'" '   "' | \
    grep -i -o '<a[^>]\+href[ ]*=[ \t]*"\(ht\|f\)tps\?:[^"]\+"' | \
    sed -e 's/^.*"\([^"]\+\)".*$/\1/g'

This will output all http, https, ftp, and ftps links from a webpage. It will not give you relative urls, only full urls.

Explanation regarding the options used in the series of piped commands:

wget -q makes it not have excessive output (quiet mode). wget -O - makes it so that the downloaded file is echoed to stdout, rather than saved to disk.

tr is the unix character translator, used in this example to translate newlines and tabs to spaces, as well as convert single quotes into double quotes so we can simplify our regular expressions.

grep -i makes the search case-insensitive grep -o makes it output only the matching portions.

sed is the Stream EDitor unix utility which allows for filtering and transformation operations.

sed -e just lets you feed it an expression.

Running this little script on "http://craigslist.org" yielded quite a long list of links:

http://blog.craigslist.org/
http://24hoursoncraigslist.com/subs/nowplaying.html
http://craigslistfoundation.org/
http://atlanta.craigslist.org/
http://austin.craigslist.org/
http://boston.craigslist.org/
http://chicago.craigslist.org/
http://cleveland.craigslist.org/
...

Very cool. But the Win32 versions of the tools are choking... Somewhere. Can you say how to modify them for Cygwin or straight Windows? — Snowy, Jul 08 '13 at 02:38
@Snowy I'm not sure what you mean by "choking". Cygwin should work fine. You could also try using `curl` instead of `wget`. — Jay Taylor, Jul 09 '13 at 00:29
I would like to point out that @Rob wanted to get all urls from a website and not from a webpage. — erdomester, Jun 15 '15 at 20:58
OK nevermind, I changed the grep command to this: `grep -i -o ']\+href[ ]*=[ \t]*"[^"]\+">[^<]*'` and removed the `sed` and it seems to do the job. Now I just need to parse these `A` tags somehow. — BarbaraKwarc, Oct 28 '16 at 23:23

Rick · Answer 3 · 2013-05-09T16:47:02.923

8

I've used a tool called xidel

xidel http://server -e '//a/@href' | 
grep -v "http" | 
sort -u | 
xargs -L1 -I {}  xidel http://server/{} -e '//a/@href' | 
grep -v "http" | sort -u

A little hackish but gets you closer! This is only the first level. Imagine packing this up into a self recursive script!

edited May 09 '13 at 16:47

answered May 09 '13 at 16:37

Rick

591
1
9
23

Thanks ... that looks perfect for scripting a workaround to my wget problem ( http://opendata.stackexchange.com/q/4851/263 ) – Joe Mar 26 '15 at 16:52

Spider a Website and Return URLs Only

3 Answers3

Linked