How to scrape a JSP page in R?

Question

I want to scrape the content of the following page in R: http://directoriosancionados.funcionpublica.gob.mx/SanFicTec/jsp/Ficha_Tecnica/SancionadosN.htm

However, I'm not able to locate any HTML tag or any other tool that could help me to obtain the information.

I'm interested in build a data frame with the information of the section "INHABILITADOS Y MULTADOS" like in the following images:

This is the particular option I'm trying to scrape

When this option is selected, a menu of several providers appears, each one with a particular table with the information I want to recollect.

The list of providers

The information I finally want to scrape

Answer 1

Normally, you can use GET method for requests. But for that website,you need to use POST method:

Check network tab in chrome developer mode(Press F12)

In following images, submit Form Data in POST request's body.

Find patterns in onclick: the onlick value is used for submitting forms

---

The following script should work:

library(httr)
library(rvest)
library(stringr)
library(dplyr)
my_url <- "http://directoriosancionados.funcionpublica.gob.mx/SanFicTec/jsp/Ficha_Tecnica/SancionadosN.jsp"
my_ua <- "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"


#use post method instead of get to get correct response
response <- POST(my_url,
                 user_agent(my_ua),
                 body = list(cmdsan = "INHABILITA",
                             tipoqry = "INHABILITA",
                             mostrar_msg = "SI"),
                 encode = "form")


href_nodes <- content(response) %>%
  html_node("table") %>%
  html_nodes("a")

link_text <- href_nodes %>% 
  html_text() %>% 
  as.tibble() %>%
  rename(text = value)

form_items <- href_nodes %>% 
  html_attr("onclick") %>% # extract items to POST
  str_extract("(?<=\\().*?(?=\\))") %>% # extract everything inside brackets
  str_split("\\,",simplify = T) %>%# split POST items
  as.tibble() %>%
  mutate(V1 = str_sub(V1,start = 2,end =-2))


submit_table <- bind_cols(link_text,form_items)

#using POST method to get to the page you want
#for example, if you want to go to page A Y M CONSTRUCTORA, S.A. DE C.V (row 2)
#you should:

row_num <- 2

my_url2 <- "http://directoriosancionados.funcionpublica.gob.mx/SanFicTec/jsp/Ficha_Tecnica/FichaSinTabla.jsp"

response1 <- POST(my_url2,
                 user_agent(my_ua),
                 body = list(expe = submit_table$V1[row_num],
                             tipo = submit_table$V2[row_num],
                             persona = submit_table$V3[row_num]),
                 encode = "form")

---

content in submit_table, which will be used later to make POST request to get content in each individual page.

> submit_table 
# A tibble: 1,329 x 4
text                                        V1             V2    V3   
<chr>                                       <chr>          <chr> <chr>
  1 A AND P INTERNATIONAL                       185770002/2016 1     3    
2 A Y M CONSTRUCTORA, S.A. DE C.V.            000090121/2006 1     3    
3 A Y V INDUSTRIAL Y COMERCIAL, S.A. DE C.V.  184000001/2013 1     3    
4 A+D ARQUITECTOS, S.A. DE C.V.               181640187/2006 1     3    
5 A.D.C. Consultores y Servicios, S.A de C.V. 111510007/2005 1     3    
6 AARON VERA MORALES                          006410056/2011 1     3    
7 ABASTECEDORA DE FÁRMACOS, S.A. DE C.V.      006410002/2014 1     3    
8 ABASTECEDORA EZCO, S.A. DE C.V.             000070024/2016 1     3    
9 ABEL ZURITA MAYO                            000200012/2014 1     3    
10 ABS TECNOLOGÍA, S.A. DE C.V.                090850001/2016 1     3    
# ... with 1,319 more rows

You can use functions in rvest to extract those elements using the response:

(content(response1) %>% html_nodes(".normal") %>% html_text() %>% str_trim())[3]

will return:

[1] "Publicación en el DOF: 05 DE ABRIL DE 2007Monto de la Multa: $ 72,540.00Plazo de inhabilitación: 3 MESESInicia: 06 DE ABRIL DE 2007Termina: 06 DE JULIO DE 2007"