RSelenium: loop over all values in dropboxes

Question

I am trying to scrape for each country its table with all HIV/AIDS related NGOs using this link: https://www.unodc.org/ngo/showExtendedSearch.do

I am able to navigate towards the url and select the 'HIV/AIDS' radio button. But now I also need to extract for both the dropboxes 'region' and 'country' all the values so that I can use them inside a loop to sequentially webscrape the table for each country. How can I collect the values for both dropboxes? My code so far is below:

#load library
library(RSelenium)

#Specify remote driver
remDr <- remoteDriver(browserName='firefox')

#Initialise session 
remDr$open()

#navigate to advanced search page

url <- "https://www.unodc.org/ngo/showExtendedSearch.do"
remDr$navigate(url)

#Click 'HIV/AIDS' filter
webElem <- remDr$findElement(using = 'css', 
                         value = '#applicationArea > form > table > tbody > tr > td > table:nth-child(7) > tbody > tr:nth-child(2) > td > table > tbody > tr > td:nth-child(2) > table > tbody > tr:nth-child(3) > td:nth-child(4) > input[type="checkbox"]')

webElem$clickElement()

score 0 · Answer 1 · answered Jul 25 '16 at 23:57

Use firebug or Developer Tools to determine the xpath of the dropdown menu elements then use getElementText to retrieve the values:

 region_element <- remDr$findElement('xpath', '//*[@id="applicationArea"]/form/table/tbody/tr/td/table[2]/tbody/tr[2]/td/table/tbody/tr[1]/td[2]/select')
regions <- strsplit(region_element$getElementText()[[1]], "\n")

country_element <- remDr$findElement('xpath', '//*[@id="applicationArea"]/form/table/tbody/tr/td/table[2]/tbody/tr[2]/td/table/tbody/tr[2]/td[2]/select')
countries <- strsplit(country_element$getElementText()[[1]], "\n")

R> print(regions[[1]])
 [1] "Middle East and Northern Africa"   "Eastern Africa"                   
 [3] "Western Africa"                    "Central and Southern Africa"      
 [5] "Northern America"                  "Central America and the Caribbean"
 [7] "Latin America"                     "Central and Western Asia"         
 [9] "Southern and Eastern Asia"         "Europe"                           
[11] "Oceania"                          
R> print(head(countries[[1]]))
[1] "Afghanistan"    "Albania"        "Algeria"        "American Samoa" "Andorra"       
[6] "Angola"

RSelenium: loop over all values in dropboxes

1 Answers1