Get mp3/pdf files using JSoup in Groovy

Question

I am developing an application for crawling the web using crawler4j and Jsoup. I need to parse a webpage using JSoup and check if it has zip files, pdf/doc and mp3/mov file available as a resource for download.

For zip files i did the following and it works:

Elements zip = doc.select("a[href\$=.zip]")
        println "No of zip files is " + zip.size()

This code correctly tells me how many zip files are there in a page. I am not sure how to count all audio files or document files using JSoup. Any help is appreciated. Thanks.

score 2 · Accepted Answer · answered Jul 07 '14 at 15:45

2

Using the same approach I suspect it would be something like this:

Elements docs = doc.select("a[href\$=.doc]")
        println "No of doc files is " + docs.size() 

Elements mp3s = doc.select("a[href\$=.mp3]")
        println "No of mp3 files is " + mp3s.size()

Really it's just a selector where the href attribute ends in some file extension.

answered Jul 07 '14 at 15:45

Joshua Moore

24,706
6
50
73

thanks . I thought so but the fact that different webpages uses different approaches to embed audio files bothers me. For example, I checked this page : http://mp3.com/top-downloads/ and it has tag for embedding mp3 files. do you think this would not cause incorrect counts? – clever_bassi Jul 07 '14 at 15:48
Yep, it sure would. You would need a different selector for that a[relSong\$=.mp3] for instance. – Joshua Moore Jul 07 '14 at 15:51
In that way there are several ways people embed mp3s and doc files. Is there any generic way using Jsoup where I can extract the correct information without delving into individual embedding styles? – clever_bassi Jul 07 '14 at 15:53

Get mp3/pdf files using JSoup in Groovy

1 Answers1