1

I'm attempting to use JSoup to retrieve the amount of wins for a team from a Sports Reference table.

Specifically, I am trying to receive the following data point highlighted below, with the html code provided

Below is what I have tried already, but I get a null pointer exception when trying to access the text of this element, telling me that my code is likely not parsing the HTML code correctly.

Element wins = document.selectFirst("td[data-stat=\"wins\"]");

What I want is for the text of this element to be 34 (or some number depending on the number of wins for the team).

invzbl3
  • 5,872
  • 9
  • 36
  • 76
bentarmann
  • 11
  • 1

1 Answers1

1

Check what your Document was able to read from page and print it. If it contains HTML content which can be dynamically added by JavaScript by browser, you need to use as tool Selenium not Jsoup.

For reading HTML source, you can write similar to: 

import java.io.IOException;
import org.jsoup.Jsoup;

public class JSoupHTMLSourceEx {
    public static void main(String[] args) throws IOException {
        String webPage = "https://www.basketball-reference.com/teams/CHI/2020.html#all_team_misc";
        String html = Jsoup.connect(webPage).get().html();
        System.out.println(html);
    }
}

Since Jsoup supports cssSelector, you can try to get an element like:

public static void main(String[] args)  {
        String webPage = "https://www.basketball-reference.com/teams/CHI/2020.html#all_team_misc";
        String html = Jsoup.connect(webPage).get().html();

Document document = Jsoup.parse(html);
    Elements tds = document.select("#team_misc > tbody > tr:nth-child(1) > td:nth-child(2)");
        for (Element e : tds) {
            System.out.println(e.text());
        }
}

But better solution is to use Selenium - a portable framework for testing web applications (more details about Selenium tool):

public static void main(String[] args) {
    String baseUrl = "https://www.basketball-reference.com/teams/CHI/2020.html#all_team_misc";
    WebDriver driver = new FirefoxDriver();

    driver.get(baseUrl);
    String innerText = driver.findElement(
        By.xpath("//*[@id="team_misc"]/tbody/tr[1]/td[1]")).getText();  
        System.out.println(innerText); 
    driver.quit();
    }
}

Also you can try instead of: 

driver.findElement(By.xpath("//*[@id="team_misc"]/tbody/tr[1]/td[1]")).getText();

in this form:

driver.findElement(By.xpath("//[@id="team_misc"]/tbody/tr[1]/td[1]")).getAttribute("innerHTML");

P.S. In the future it would be useful to add source links from where you want to get information or at least snippet of the DOM structure instead of image.

invzbl3
  • 5,872
  • 9
  • 36
  • 76
  • 1
    Selenium is not necessarily a better solution -- it simply solves a different problem. Jsoup is for parsing the raw DOM as returned from the server (without invoking JS). Selenium is for parsing DOM _after_ invoking JS. – Zack Jan 12 '20 at 05:09