Python: how to get hidden html contents from a HTML page

Question

i'm trying to make a program that can reissue the books i've taken from the library using robobrowser so,for that i was supposed to

1) login to my ID
2) tick on the checkboxes of the respective books
3) click on submit

after i did the login and printed the response of the page, it did indeed printed the sourcecode of the page EXCEPT the source code of the form in which all the tick boxes and name of the books are present..

it was supposed to show the source code of the form as well, i.e

     <tr>
        <td valign="top">   














    <form action="./selectedBooks" method="get" id="form1" target="nid">
    <table border="0" class="briefListTbl" cellspacing="0" cellpadding="0" 
    valign="top">
    <tbody><tr><td>
    <table valign="top" border="0" cellspacing="0" cellpadding="0" height="100%" 
    width="100%">
       <tbody class="briefListHead">
    <tr align="middle">


            <td width="10%">
                AccNo.
            </td>

            <td width="40%">
                Title
            </td>

            <td width="15%">
                Author
            </td>

            <td width="15%">
                Due date
            </td>

            <td width="10%">
                Reserved
            </td>

            <td>Renew</td> 

    </tr>
    </tbody></table>                                        
             </td></tr>
        <tr>
        <td class="briefListBody">

           <div class="scrollbarTbl" id="divTblScroll" style="overflow: auto; 
    height: 178px; width: 1176px;">
           <table class="briefListTbl" border="0" cellspacing="2" 
    cellpadding="0" width="100%" height="100%">
           <tbody valign="top">
            <tr class="briefListRow1" id="checkoutsRow1">

                                <td width="10%">50884</td>

                        <td width="40%" align="left">Programming with C (005.133 
    GOT;  42)</td>
                        <td width="15%" align="left">Gottfried, Byron</td>
                        <td width="15%" align="left">24/03/2017</td>
                        <td width="10%" align="left">-</td>
                                <td align="center"><input type="checkbox" 
    name="selectedforRenewal" value="0"></td>

                        </tr>

                        <tr class="briefListRow2" id="checkoutsRow2">

                                <td width="10%">51203</td>

                        <td width="40%" align="left">Engineering physics 
    (621PRA;  147)</td>
                    <td width="15%" align="left"> dr.psk formy</td>
                    <td width="15%" align="left">24/03/2017</td>
                 `   <td width="10%" align="left">-</td>
                                <td align="center"><input type="checkbox" 
    name="selectedforRenewal" value="1"></td>

                    </tr>

                    <tr class="briefListRow1" id="checkoutsRow3">

                            <td width="10%">20810</td>

                    <td width="40%" align="left">Objective Mathematics (511 SHA 17)</td>
                    <td width="15%" align="left">SHA(R.D)</td>
                    <td width="15%" align="left">30/03/2017</td>
                    <td width="10%" align="left">-</td>
                            <td align="center"><input type="checkbox" name="selectedforRenewal" value="2"></td>

                    </tr>

                    <tr class="briefListRow2" id="checkoutsRow4">

                            <td width="10%">22455</td>

                    <td width="40%" align="left">Elements of Mechanical Engineering (620 PRA 19)</td>
                    <td width="15%" align="left">holydon</td>
                    <td width="15%" align="left">03/04/2017</td>
                    <td width="10%" align="left">-</td>
                            <td align="center"><input type="checkbox" name="selectedforRenewal" value="3"></td>

                    </tr>

                    <tr height="100%"><td><input type="hidden" name="searchOrBrowse" value="checkouts"></td></tr>
           </tbody>
           </table>
    </div>


    </td>
</tr>
</tbody>
<tbody><tr>


                <td class="briefListFoot">

                    <table border="0" width="100%" class="briefListFoot" cellspacing="0" cellpadding="0">
                    <tbody><tr>
                        <td align="left" valign="top" width="0*" height="0*" nowrap="">

                        </td>
                        <td align="middle" valign="center" height="0*">

                                <a class="briefListHREFFoot" href="javascript:showpopupFrame('./renew');"><b>Renew</b></a>

                            </td>

                           <td align="right" valign="top" width="0*" height="0*" nowrap="">

                                <a class="briefListHREFFoot">Next<img border="0" src="../images/button/nill.GIF" alt=""></a>

                        </td>
                    </tr>
                </tbody></table></td>



</tr>


</tbody></table>



                </form></td></tr>

but it is only printing

<tr><td valign="top">
</td></tr>
</table>
</td>
</tr>
</table>

and my code..

browser = RoboBrowser(parser='lxml', user_agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11', history=True, session=start)


browser.open('http://61.12.27.181:8080/opac/html/checkouts')
sign_in = browser.get_form(action='./memberlogin')
sign_in['txtmemberid'].value = 'MyId'
browser.submit_form(sign_in, submit='  Go  ')
box = browser.get_form(action='./selectedBooks')
print(browser.parsed)

the whole td tag is missing , i dont think it is because of robobrowser because i'm passing session as well, then why is it happening
any help?

try to use a different parser, like html5 parser. It is what I had to use to get full html with beautiful soup — chaos, Mar 23 '17 at 12:15
@chaos it is showing a white space now . whiltespace instead of the content — P.hunter, Mar 23 '17 at 12:31

score 0 · Accepted Answer · answered Mar 24 '17 at 12:28

this question can be solved by adding script URL from the network tab by inspecting the element i.e.

1) Go to the network tab in inspect element tool.
2) get the url of js or css which have ok responses and use the browser.session.get(url_of_js_or_css) in Robobrowser and requests.get(url) in requests

good luck!

Python: how to get hidden html contents from a HTML page

1 Answers1