Click website button using Python

Question

I want to automate a process that searches for a key word in a search bar using Python on my server. However, searching first returns a disclaimer and once accepted, the result is actually shown. The disclaimer site is the following:

http://emma.msrb.org/Disclaimer.aspx

I have already inspected the website and this is my preliminary code:

browser = RoboBrowser(history=True)
browser.open(tradeinfo) 
# tradeinfo = link to search result, i.e. http://emma.msrb.org/SecurityDetails/TradeActivity/ABF6AA894523779EDC8B751EB99959934
form = browser.get_forms()[0]
print form

<RoboForm __VIEWSTATE=somevalue, __VIEWSTATEGENERATOR=somevalue, __EVENTVALIDATION=somevalue, 
ctl00$Masthead$UserAcct1$NextAction=, 
ctl00$Masthead$UserAcct1$isPilotUser=False, ctl00$Masthead$searchTextBox=Enter CUSIP or Name,
ctl00$Masthead$headerSearchButton=, 
ctl00$mainContentArea$disclaimerContent$yesButton=, 
ctl00$mainContentArea$disclaimerContent$noButton=>
# Somevalue represents a string of characters (deleted for visual clarity)

Then I do the following:

form['ctl00$mainContentArea$disclaimerContent$yesButton'].value ='Yes'
browser.submit_form(form)
browser.select('body')

But the body still doesn't contain the data. I also tried mechanize, with the same outcome. Is there a way to do this without Selenium, using just mechanize or RoboBrowser? If so, what I am I missing in my script?

P.S: html script for the "Accept" button:

<input type="image" 
name="ctl00$mainContentArea$disclaimerContent$yesButton" 
id="ctl00_mainContentArea_disclaimerContent_yesButton" 
data-jquerymobilestyle="false" 
src="images/btnAccept.gif" alt="Accept" style="border-width:0px;">

Selenium requires a firefox window (either on screen or headless), both is currently not working on the Unix server I have available. So I need to find another way. — calpyte, May 25 '16 at 01:28

score 0 · Answer 1 · answered May 23 '16 at 22:30

You just have to post the data, an example using requests to post and bs4 to parse what is returned:

In [6]: from bs4 import BeautifulSoup

In [7]: with requests.Session() as s:
   ...:         r = s.post("http://emma.msrb.org/Main/QuickSearch", data={"quickSearchText":"G-12"})
   ...:         soup = BeautifulSoup(r.content)
   ...:         print(soup.select_one("#ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView"))
   ...:     
<table border="0" cellspacing="0" class="grid1" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView" width="100%">
<tr>
<th scope="col"><a href="javascript:__doPostBack('ctl00$mainContentArea$SearchResultsControl1$searchResultsGridView','Sort$IssuerName')">Issuer Name *</a></th><th scope="col"><a href="javascript:__doPostBack('ctl00$mainContentArea$SearchResultsControl1$searchResultsGridView','Sort$IssueName')">Issue Description *</a></th><th scope="col"><a href="javascript:__doPostBack('ctl00$mainContentArea$SearchResultsControl1$searchResultsGridView','Sort$DatedDate')">Dated Date</a></th><th scope="col"><a href="javascript:__doPostBack('ctl00$mainContentArea$SearchResultsControl1$searchResultsGridView','Sort$MaturityDateRange')">Maturity Dates</a></th>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=006449" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl02_issuerNameHyperLink">ADAM 12 FIVE STAR SCHOOLS, CO</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=EP347630" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl02_IssueWithImageControl1_issueLink">G.O. REFUNDING BONDS, SERIES 2011</a>
</td><td>09/20/2011</td><td>2011 to 2017</td>
</tr><tr class="RowAlternate">
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=EP1603" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl03_issuerNameHyperLink">AQUA MAINE, INC WATER SER 94G 12/21/94</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=EP369008" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl03_IssueWithImageControl1_issueLink">AQUA MAINE, INC FIRST MORTGAGE BOND SER G DUE 12/15/24 DTD 12/21/94 $9,000,000</a>
</td><td> </td><td>Unavailable</td>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=072024" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl04_issuerNameHyperLink">BAY AREA TOLL AUTHORITY</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=EA343972" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl04_IssueWithImageControl1_issueLink">SAN FRANCISCO BAY AREA TOLL BRIDGE REVENUE BONDS, SIFMA INDEX RATE BONDS, 2008 SERIES G-1 (REOFFERING)</a>
</td><td>06/05/2008</td><td>04/01/2045</td>
</tr><tr class="RowAlternate">
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=13034U" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl05_issuerNameHyperLink">CALIFORNIA INFRASTRUCTURE &amp; ECONOMIC DEV BK REV IAM COML PAPER 3/A2 SER G</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=E81A86D5BE1D9CCEFEA5BE542F8B8F9B" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl05_IssueWithImageControl1_issueLink">COML PAPER IAM 12</a>
</td><td>01/01/1930</td><td>Unavailable</td>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=13066Y" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl06_issuerNameHyperLink">CALIFORNIA ST DEPT WTR RES PWR SUPPLY REV</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=MS286087" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl06_IssueWithImageControl1_issueLink">SER G-11-RMKT 12/01/08</a>
</td><td>12/01/2005</td><td>05/01/2018</td>
</tr><tr class="RowAlternate">
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=13066Y" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl07_issuerNameHyperLink">CALIFORNIA ST DEPT WTR RES PWR SUPPLY REV</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=MS250013" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl07_IssueWithImageControl1_issueLink">SER G-4-RMKT 12/01/08</a>
</td><td>12/01/2005</td><td>05/01/2016</td>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=13066Y" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl08_issuerNameHyperLink">CALIFORNIA ST DEPT WTR RES PWR SUPPLY REV</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=MS142918" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl08_IssueWithImageControl1_issueLink">VAR-SUBSER G-12</a>
</td><td>12/01/2005</td><td>05/01/2018</td>
</tr><tr class="RowAlternate">
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=16757L" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl09_issuerNameHyperLink">CHICAGO ILL MIDWAY ARPT REV TAXABLE IAM COML PAPER 3/A2 SER G</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=67F0CEF2677E837E18B4B370D84E2D21" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl09_IssueWithImageControl1_issueLink">COML PAPER IAM 12</a>
</td><td>01/01/1930</td><td>Unavailable</td>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=167486" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl10_issuerNameHyperLink">CITY OF CHICAGO</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=ER349383" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl10_IssueWithImageControl1_issueLink">GENERAL OBLIGATION VARIABLE RATE DEMAND BONDS, REFUNDING SERIES 2007G</a>
</td><td>11/08/2007</td><td>01/01/2042</td>
</tr><tr class="RowAlternate">
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=20775B" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl11_issuerNameHyperLink">CONNECTICUT HOUSING FINANCE AUTHORITY</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=EP358373" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl11_IssueWithImageControl1_issueLink">HOUSING MORTGAGE FINANCE PROGRAM BONDS
2010 SERIES G</a>
</td><td>12/01/2010</td><td>2013 to 2031</td>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=246018" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl12_issuerNameHyperLink">DELAWARE CNTY PA INDL DEV AUTH REV</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=41B742BA34F00163F429A1B4A608E07C" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl12_IssueWithImageControl1_issueLink">ANNUAL-REF-G-RMKT-12/2/96</a>
</td><td>06/01/1993</td><td>12/01/2013</td>
</tr><tr class="RowAlternate">
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=291380" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl13_issuerNameHyperLink">EMMAUS GENERAL AUTHORITY
(PENNSYLVANIA)</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=MS60546" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl13_IssueWithImageControl1_issueLink">LOCAL GOVERNMENT REVENUE BONDS
(BOND POOL PROGRAM)
$2,400,000 SERIES 1989 G (SUBSERIES G-19)</a>
</td><td>03/16/1989</td><td>Unavailable</td>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=291380" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl14_issuerNameHyperLink">EMMAUS PA GEN AUTH REV</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=MS86489" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl14_IssueWithImageControl1_issueLink">LOC GOVT-G-11-RMKT 12/01/1998</a>
</td><td>03/16/1989</td><td>Unavailable</td>
</tr><tr class="RowAlternate">
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=291380" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl15_issuerNameHyperLink">EMMAUS PA GEN AUTH REV</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=937017C6899D3EE65A761735286F701F" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl15_IssueWithImageControl1_issueLink">VAR-LOC GOVT-G-11-RMK 12/01/98</a>
</td><td>03/16/1989</td><td>03/01/2024</td>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=291380" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl16_issuerNameHyperLink">EMMAUS PA GEN AUTH REV</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=MS159398" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl16_IssueWithImageControl1_issueLink">VAR-LOC GOVT-G-12-RMKTD 2/1/99</a>
</td><td>03/16/1989</td><td>03/01/2024</td>
</tr><tr class="RowAlternate">
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=291380" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl17_issuerNameHyperLink">EMMAUS PA GEN AUTH REV</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=851B472A6038290736FE10877813DC5C" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl17_IssueWithImageControl1_issueLink">VAR-SUBSER G-6-RMK 12/1/95</a>
</td><td>03/15/1989</td><td>03/01/2024</td>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=402175" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl18_issuerNameHyperLink">GULF BREEZE FLA REV</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=MS285974" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl18_IssueWithImageControl1_issueLink">FLTG-LOC GOVT-G-RMKT-12/01/08</a>
</td><td>12/30/1985</td><td>12/01/2015</td>
</tr><tr class="RowAlternate">
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=44241A" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl19_issuerNameHyperLink">HOUSTON TEX IAM COML PAPER NTS 3/A2 SER G-1</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=9FE04F6C8E02213549422A9A68E2F021" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl19_IssueWithImageControl1_issueLink">COML PAPER IAM 12</a>
</td><td>01/01/1930</td><td>Unavailable</td>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=44241B" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl20_issuerNameHyperLink">HOUSTON TEX IAM COML PAPER NTS 3/A2 SER G-2</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=AF16C39A167072D4704A509F1B4A1084" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl20_IssueWithImageControl1_issueLink">COML PAPER IAM 12</a>
</td><td>05/01/2012</td><td>05/22/2012</td>
</tr><tr class="RowAlternate">
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=491322" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl21_issuerNameHyperLink">KENTUCKY INTERLOCAL SCH TRANSN ASSN EQUIP LEASE REV</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=MS124785" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl21_IssueWithImageControl1_issueLink">LEASE PG-SER G-6-RMKT 12/1/99</a>
</td><td>07/23/1997</td><td>2002 to 2019</td>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=639671" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl22_issuerNameHyperLink">NEBRASKA INVT FIN AUTH</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=MS162444" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl22_IssueWithImageControl1_issueLink">SER G O-12</a>
</td><td>04/03/2001</td><td>03/01/2011</td>
</tr><tr class="RowAlternate">
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=71783E" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl23_issuerNameHyperLink">PHILADELPHIA PA GAS WKS REV IAM COML PAPER NTS 3/A2 SER G-1</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=F4D5EB17078B2AA2BA5280443137E116" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl23_IssueWithImageControl1_issueLink">COML PAPER IAM 12</a>
</td><td>01/01/1930</td><td>Unavailable</td>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=71783F" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl24_issuerNameHyperLink">PHILADELPHIA PA GAS WKS REV IAM COML PAPER NTS 3/A2 SER G-2</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=38C8B7A389D05523CFE6D97E21F195C0" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl24_IssueWithImageControl1_issueLink">COML PAPER IAM 12</a>
</td><td>01/01/1930</td><td>Unavailable</td>
</tr><tr class="RowAlternate">
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=917436" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl25_issuerNameHyperLink">UTAH HSG CORP SINGLE FAMILY MTG REV</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=507587B811D1853256E8049ECAB08C0E" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl25_IssueWithImageControl1_issueLink">VAR-SER G-2-CL I-RMKT 12/30/09</a>
</td><td>12/17/2002</td><td>01/01/2034</td>
</tr><tr>
<td>
<a href="http://emma.msrb.org/IssuerView/IssuerDetails.aspx?cusip=917436" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl26_issuerNameHyperLink">UTAH HSG CORP SINGLE FAMILY MTG REV</a>
</td><td>
<a href="http://emma.msrb.org/IssueView/IssueDetails.aspx?id=FD5E486975764638149DB3D4750980BA" id="ctl00_mainContentArea_SearchResultsControl1_searchResultsGridView_ctl26_IssueWithImageControl1_issueLink">VAR-SER G-CL I-RMKT 12/30/09</a>
</td><td>10/14/2004</td><td>01/01/2036</td>
</tr>
</table>

Click website button using Python

1 Answers1