0

I'm using Java with the Selenium Library to scrape a webpage. When I use Firebug on the page in Firefox, I can see that the page's source contains the following HTML structure:

<div>
    <div>
        <table>
            <caption />
            <thead />
            <tbody />
        </table>
    </div>
</div>

However, when I programatically download the page's source using HtmlUnitDriver, then use driver.getPageSource(), I see that the corresponding HTML structure has changed to:

<div>
    <table>
        <caption />
        <tbody />
    </table>
</div>
  1. Why does the HtmlUnitDriver's report differ to that given by Firebug?
  2. Can I set up firebug so that I can inspect the HTML structure according to how the HtmlUnitDriver will report it?
danger mouse
  • 1,457
  • 1
  • 18
  • 31

1 Answers1

1

Note that Firebug does not adjust the HTML structure that way, i.e. the integrated developer tools should show you the same.

I assume the second wrapping <div> and the <thead> get added by some JavaScript running on the page.

You can check that by disabling JavaScript, e.g. by going to about:config and setting javascript.enabled to false or via an add-on like NoScript or Ghostery.

Sebastian Zartner
  • 18,808
  • 10
  • 90
  • 132
  • Sorry for the late comment - I was away from my computer for a couple of days. I disabled the JavaScript, then inspected the page's HTML using Firebug, which then turned out to be the same as the HtmlUnitDriver's report. Many thanks! – danger mouse Aug 31 '16 at 15:39