How to retrieve search list using goquery from RottenTomatoes?

Question

I'm currently trying to get search list from RottenTomatoes for some researching purposes. I have integrated goquery in an easy way for different sites. But the situation for RottenTomatoes was not the same. I can't get the search list even though I have tried different comminutions with queries.

Testing URL: https://www.rottentomatoes.com/search/?search=test

The text I want to get: Testament

doc.Text() Output: https://pastebin.com/SsWHYXTH

Here is what i do:

func ParseSearchMovies(doc *goquery.Document) *models.SearchResponse {
    result := new(models.SearchResponse)
    finder0 := doc.Find("section > section > ul")
    finder1 := doc.Find("section.movieSection > ul")
    finder2 := doc.Find("section.movieSection > ul > li")
    finder3 := doc.Find("movieSection")
    finder4 := doc.Find("section.movieSection")
    finder5 := doc.Find("section.movieSection ul.results_ul")
    finder6 := doc.Find("div.Search.Results li.bottom_divider.clearfix")
    finder7 := doc.Find("div.search-results-root span.bold")
    finder8 := doc.Find("div.search-results-root #details")
    finder9 := doc.Find("div.search-results-root li.bottom_divider.clearfix")
    finder10 := doc.Find("#movieSection > ul > li:nth-child(1)")
    finder11 := doc.Find("#movieSection > ul")
    finder12 := doc.Find("#movieSection > ul .bottom_divider.clearfix")
    finder13 := doc.Find("#movieSection > ul > li")
    finder14 := doc.Find("section.Search.Results .movieSection")
    finder15 := doc.Find("#search-results-root section")
    finder16 := doc.Find("#search-results-root .section")
    finder17 := doc.Find("#search-results-root .section .section")
    finder18 := doc.Find("#search-results-root > section > section")
    finder19 := doc.Find("#search-results-root > section .section")

    //From hlscalon's answer
    finder20 := doc.Find("#movieSection > .results_ul > li > div.details")
    finder21 := doc.Find("#movieSection > .results_ul > li")
    finder22 := doc.Find("#movieSection > .results_ul")
    finder23 := doc.Find("#movieSection")

    fmt.Printf("%d", len(finder0.Nodes))
    fmt.Printf("%d", len(finder1.Nodes))
    fmt.Printf("%d", len(finder2.Nodes))
    fmt.Printf("%d", len(finder3.Nodes))
    fmt.Printf("%d", len(finder4.Nodes))
    fmt.Printf("%d", len(finder5.Nodes))
    fmt.Printf("%d", len(finder6.Nodes))
    fmt.Printf("%d", len(finder7.Nodes))
    fmt.Printf("%d", len(finder8.Nodes))
    fmt.Printf("%d", len(finder9.Nodes))
    fmt.Printf("%d", len(finder10.Nodes))
    fmt.Printf("%d", len(finder11.Nodes))
    fmt.Printf("%d", len(finder12.Nodes))
    fmt.Printf("%d", len(finder13.Nodes))
    fmt.Printf("%d", len(finder14.Nodes))
    fmt.Printf("%d", len(finder15.Nodes))
    fmt.Printf("%d", len(finder16.Nodes))
    fmt.Printf("%d", len(finder17.Nodes))
    fmt.Printf("%d", len(finder18.Nodes))
    fmt.Printf("%d", len(finder19.Nodes))
    fmt.Printf("%d", len(finder20.Nodes))
    fmt.Printf("%d", len(finder21.Nodes))
    fmt.Printf("%d", len(finder22.Nodes))
    fmt.Printf("%d", len(finder23.Nodes))
}

Expected: 9 (Movie count in <section id="movieSection">)
Actual: All the results are returns 0. Output: 000000000000000000000000

P.S: When I print the result to the console via doc.Text(), whole the page comes correctly.

To print all movie names in order:

if len(myFinder.Nodes) > 0 {
    doc.Find("MY QUERY").Each(func(i int, s *goquery.Selection) {
        name := s.Find("a")

        fmt.Println(name)
    })
}

score 0 · Answer 1 · edited Dec 16 '21 at 15:30

0

There are some (many) alternatives but if you want a list of all links of films this should work:

#movieSection > .results_ul > li > div.details > span.bold > a.articleLink

This will lead you directly to the links (a), so maybe you'll have to change your function to print names or stop one step before (span)

Edit

Actually, I think can't use goquery to scrape that page. It uses react, so when you download the page (ie curl "https://www.rottentomatoes.com/search/?search=test"), you will see this:

<div id="main_container" class="container ">
    <div class="col col-left-center col-full-xs">
        <div id="search-results-root"></div>
            <script>
                require(['jquery', 'globals', 'search-results', 'bootstrap'], function($, RT, mount) {
                mount($("#search-results-root").get(0), RT.PrivateApiV2FrontendHost, 'test', {"actorCount"....

When goquery is expecting html only

edited Dec 16 '21 at 15:30

DisappointedByUnaccountableMod

6,656
4
18
22

answered Dec 12 '18 at 15:50

hlscalon

7,304
4
33
40

I doesn't work. Returns 0 node. I create a loop and checked it out, but it always turns 0. – Dentrax Dec 12 '18 at 16:09
Hmm, So How can i get movies list? Should i parse `search-results-root ` as a JSON format? – Dentrax Dec 12 '18 at 18:50
You have the movies there, for sure, but I don't think you can parse them with goquery. – hlscalon Dec 12 '18 at 19:03

How to retrieve search list using goquery from RottenTomatoes?

1 Answers1