rvest::html_nodes returns a partial list (only a few items)

Question

Using the rvest package, I am trying to scrape names of actors/actresses from IMDB page for the film JFK (https://www.imdb.com/title/tt0102138/fullcredits?ref_=tt_ql_1).

SelectorGadget says that the place I want to look to find the names is "td:nth-child(2)" for every person.

Here's the code I'm using.

        library(rvest)
        library(stringr)

        startFilm <- "tt0102138" #JFK
        personsNames <- c()
        pagePath <- paste("https://www.imdb.com/title/", startFilm, "/?ref_=nv_sr_1?ref_=nv_sr_1", sep = "")
        moviePage <- read_html(pagePath)
        personNodes <- html_nodes(moviePage, "td:nth-child(2)")
        personText <- html_text(personNodes)
        for (i in 1:length(personText)){
                actor <- (unlist(str_split(personText[i], "\n")))[2]
                personsNames[i] <- substring(actor, 2, nchar(actor))
        }
        personsNames

According to the website at https://www.imdb.com/title/tt0102138/fullcredits?ref_=tt_ql_1 this list should be fairly long.

Yet when I run the code I only get back 15 names.

[1] "Sally Kirkland"  "Anthony Ramirez" "Ray LePere"      "Steve Reed"      "Jodie Farber"    "Columbia Dubose"
[7] "Randy Means"     "Kevin Costner"   "Jay O. Sanders"  "E.J. Morris"     "Cheryl Penland"  "Jim Gough"
[13] "Perry R. Russo"  "Mike Longman"    "Edward Asner"

Why is the list of names truncated?

How should I adjust my code to get the full list of actors/actresses in the film?

Your `pagePath` URL is not correct (it's missing `fullcredits`- you also don't need those query strings). — Ritchie Sacramento, Jan 20 '20 at 04:47
That would explain why I was only getting a truncated list. Thank you. — THill3, Jan 20 '20 at 17:50

score 3 · Accepted Answer · answered Jan 20 '20 at 05:40

Here is what I did. If you just need actors/actresses, you can run the following code. I identified the specific location. In this way, you can precisely get the names of actors/actresses; no need for string manipulation.

library(rvest)
library(stringi)

read_html("https://www.imdb.com/title/tt0102138/fullcredits?ref_=tt_ql_1") %>% 
html_nodes("td.primary_photo") %>% 
html_nodes("img") %>% 
html_attr("alt")

#  [1] "Sally Kirkland"             "Anthony Ramirez"            "Ray LePere"                 "Steve Reed"                
#  [5] "Jodie Farber"               "Columbia Dubose"            "Randy Means"                "Kevin Costner"  
#[249] "Mark Edward Walters"        "Earl Warren"                "John B. Wells"              "Jim White"                 
#[253] "Phillip L. Willis"          "Rosemary Willis"            "Louis Steven Witt"          "Angus G. Wynne III"

As a bonus, if you want to create a data frame with the names and characters' names, you can try the following.

mydf <- tibble(actors = read_html("https://www.imdb.com/title/tt0102138/fullcredits?ref_=tt_ql_1") %>% 
                 html_nodes("td.primary_photo") %>% 
                 html_nodes("img") %>% 
                 html_attr("alt"),
               characters = read_html("https://www.imdb.com/title/tt0102138/fullcredits?ref_=tt_ql_1") %>% 
                 html_nodes(".character") %>% 
                 html_text() %>% 
                 stri_replace_all_regex(pattern = "\\n|\\s{2,}", replacement = ""))

#  actors          characters                             
#   <chr>           <chr>                                  
# 1 Sally Kirkland  Rose Cheramie                          
# 2 Anthony Ramirez Epileptic                              
# 3 Ray LePere      Zapruder                               
# 4 Steve Reed      John F. Kennedy - Double               
# 5 Jodie Farber    Jackie Kennedy - Double(as Jodi Farber)
# 6 Columbia Dubose Nellie Connally - Double               
# 7 Randy Means     Gov. Connally - Double                 
# 8 Kevin Costner   Jim Garrison                           
# 9 Jay O. Sanders  Lou Ivon                               
#10 E.J. Morris     Plaza Witness #1

Needed the fullcredits link. Thank you for the idea to use the photo, img tag, and html_attr("alt"). That's exactly what I was looking to scrape from the page. — THill3, Jan 20 '20 at 17:49
@THill3 You are welcome. SelectorGadget is great. I have been using it. But it is sometimes good to analyze page source. In this way, you have good chance to precisely capture information that you want. — jazzurro, Jan 20 '20 at 22:59

score 0 · Answer 2 · answered Jan 20 '20 at 04:58

You need to do some data cleaning after getting the names from html_nodes

url <- "https://www.imdb.com/title/tt0102138/fullcredits?ref_=tt_ql_1"
library(rvest)

url %>%
   read_html() %>%
   html_nodes("td:nth-child(2)") %>%
   html_text() %>%
   grep("...", ., invert = TRUE, value = TRUE, fixed = TRUE) %>%
   trimws %>%
  .[. != ""]

#  [1] "Sally Kirkland"             "Anthony Ramirez"            "Ray LePere"                
#  [4] "Steve Reed"                 "Jodie Farber"               "Columbia Dubose"      
#  [7] "Randy Means"                "Kevin Costner"              "Jay O. Sanders"   
# [10] "E.J. Morris"                "Cheryl Penland"             "Jim Gough"      
# [13] "Perry R. Russo"             "Mike Longman"               "Edward Asner"    
# [16] "Jack Lemmon"                "Vincent D'Onofrio"          "Gary Oldman"
#....

rvest::html_nodes returns a partial list (only a few items)

2 Answers2