How do I sequentially extract English text from different web pages of a Tamil website?

Question

The Naalayira Divya Prabandham is a 4000-verse collection of Hindu poems written in the Tamil language. The website http://dravidaveda.org has a web page for each of the 4000 verses. Each verse page gives the Tamil verse, a word-by-word Tamil commentary on the verse, and an English translation. For instance, here is the web page for verse 1008.

My question, is there any way I can extract the English translations of all 4000 verses in order, so I can have a complete English translation of the Naalayira Divya Prabandham in a single document? For instance, in the web page I linked to above, I want to extract "Singavel-Kundram is the place where the pure Lord came as a man-lion,-while the world stood awe-struck,-and tore the Asura Hiranya’s chest with his claws. Red eyed lions offer worship by heaping elephant tusks at his feet with reverence." along with the number 1008, and I want to put it in the 1008th place in my document.

So how would I go about doing that? I assume this would this might require some kind of programming, but I don't have much of a technical background, so can someone tell me what I would need to do? Note that the article ID's, for instance the number 1379 in the URL "dravidaveda.org/index.php?option=com_content&view=article&id=1379&ml=1", don't go sequentially by verse, so that may pose a bit of a problem from a programming standpoint.

Answer 1

You can do with software/commands that dump content of webpages into terminal or console. e.g lynx, w3m, links, etc. (Though it is also possible with wget, curl, aria2 etc). Visit manual pages of respective commands for further information.

Here I'm providing sample example by using lynx:

#!/bin/bash
for i in {47..4568}
 do
 {
 lynx -dump "http://dravidaveda.org/index.php?option=com_content&view=article&id=$i&ml=1" | head -n 1 >> ndp.txt
 echo -e "\n" >> ndp.txt
 lynx -dump "http://dravidaveda.org/index.php?option=com_content&view=article&id=$i&ml=1" | grep 'English Translation' -A 10 >> ndp.txt
 echo -e "\n\n" >> ndp.txt
 }
 done;

Here {47..4598} will auto expand into 47,48,....,4568 sequentially. (I've found Nalayira Divya Prabandham can be fetched from this range)

1^stlynx command will write no. of verse e.g (1008) in the file named npd.txt

2^ndlynx command will write "English Translation" for that verse in npd.txt

Hence, with the help of for loop and depending on the range provided, you'll get no. of verses with Engish Translation in the file npd.txt.

Note that as you've mentioned that page id doesn't go subsequently, it is difficult to predict the ids to skip while coding. Anyway, I think you'll easily remove those lines from undesirable page ids from npd.txt after having it.

However, if you want you can skip dumping of those pages by using verification like:

if [[ $(lynx -dump ""http://dravidaveda.org/index.php?option=com_content&view=article&id=$i&ml=1" | head -c 1) = "(" ]]
then
[Your commands here]
fi

Here expression given in the condition of if will check whether first character of page we're going to dump is "(" or not.

So, Following command may work depending on the contents from web pages:

#!/bin/bash
for i in {47..4568}
 do
 {
   if [[ $(lynx -dump "http://dravidaveda.org/index.php?option=com_content&view=article&id=$i&ml=1" | head -c 1) = "(" ]]
   then 
     {
     lynx -dump "http://dravidaveda.org/index.php?option=com_content&view=article&id=$i&ml=1" | head -n 1 >> ndp.txt
     echo -e "\n" >> ndp.txt
     lynx -dump "http://dravidaveda.org/index.php?option=com_content&view=article&id=$i&ml=1" | grep 'English Translation' -A 10 >> ndp.txt
     echo -e "\n\n" >> ndp.txt
     } 
   fi
 }
 done;

I've checked and above script runs fine on my PC.

---

Update/Improvement:

The file ndp.txt is having verses in non-sequential order since we're getting verses in non-sequential order from website. So, finally it can be sorted with the following command (Thanks to @terdon for perl code):

perl -ne 'if(/^\((\d+)\)\s*$/){$d=$1;} push @{$k{$d}},$_; END{print "@{$k{$_}}\n" for sort { $a <=> $b} keys(%k)} ' npd.txt