HtmlUnit and decrypting a span element

Question

I am trying to scrape a telephone number from a website.

When i inspect the telephone number from the second entry, the inspector in Chrome gives me the following result:

    <span class="nummer">(012) 34 56 78</span>
    <span class="suffix encode_me telSelector129112728843_1306868" data-telselector="telSelector129112728843_1306868" data-telsuffix="IDEw"> 90</span>

However, Htmlunit (and Chrome, if I click "show source") show the following:

    <span class="nummer">(012) 34 56 78</span>
    <span class="suffix encode_me telSelector129112728843_1306868" data-telselector="telSelector129112728843_1306868" data-telsuffix="IDEw"></span>

Any way to get this last block of the telephonenumber with Htmlunit?

Perhaps they insert the data dynamically, precisely to prevent static scraping of the page...? — Andy Turner, Nov 13 '15 at 14:28

score 0 · Accepted Answer · answered Nov 13 '15 at 16:17

With latest version, I am getting it:

    try (final WebClient webClient = new WebClient(BrowserVersion.CHROME)) {
        String url = "http://www.gelbeseiten.de/schneider/hamburg";
        HtmlPage htmlPage = webClient.getPage(url);
        for (Object o : htmlPage.getByXPath("//span[@class='teilnehmertelefon']")) {
            System.out.println(((HtmlElement) o).asXml());
        }
    }

which prints an entry with:

<span class="teilnehmertelefon">
  <span class="text nummer_ganz">
    <span class="nummer">
      (040) 78 80 89
    </span>
    <span class="suffix encode_me telSelector129112728843_3662885" data-telselector="telSelector129112728843_3662885" data-telsuffix="IDEw">
       10
    </span>
  </span>
</span>

gotcha. I had JavaScript disabled. Enabling it did the trick. Thanks anyway! — , Nov 16 '15 at 09:36

HtmlUnit and decrypting a span element

1 Answers1