Python - Requests: Correctly Using Params?

Question

Before I begin, may I just say, I am very new to general communication with the web in code. With that said, could anyone assist me in getting these parameters,

        'a': stMonth,
        'b': stDate,
        'c': stYear,
        'd': enMonth,
        'e': enDate,
        'f': enYear,
        'submit': 'submit'

meant for the "Set Date Range" box on this page, http://finance.yahoo.com/q/hp?s=gspc&a=00&b=3&c=1951&d=11&e=29&f=2014&g=d&z=66&y=0

,working in my Python code. It currently includes this:

def getHistoricData(symbol, stMonth, stDate, stYear, enMonth, enDate, enYear):  
    url = 'http://finance.yahoo.com/q/hp?s=%s&a=00&b=3&c=1951&d=11&e=29&f=2014&g=d&z=66&y=0' % symbol    
    params = {
        'a': stMonth,
        'b': stDate,
        'c': stYear,
        'd': enMonth,
        'e': enDate,
        'f': enYear,
        'submit': 'submit',
    }  
    response = requests.get(url, params=params)  
    tree = html.document_fromstring(response.content)

symbol = raw_input("Symbol: ")
getHistoricData(symbol, '00', '11', '2010', '00', '13', '2010')

I believe something may be wrong with either the names or values of the parameters, but I can't be sure. Thanks in advance - any and all help is much appreciated! (including criticism, as long as it's at least somewhat constructive!)

alecxe · Answer 1 · 2015-01-07T04:00:19.880

You just don't need the submit parameter, but need a g. Here d means daily:

def getHistoricData(symbol, stMonth, stDate, stYear, enMonth, enDate, enYear):
    url = 'http://finance.yahoo.com/q/hp'
    params = {
        's': symbol,
        'a': stMonth,
        'b': stDate,
        'c': stYear,
        'd': enMonth,
        'e': enDate,
        'f': enYear,
        'g': 'd'
    }  
    response = requests.get(url, params=params)  
    tree = html.document_fromstring(response.content)
    print tree.xpath('.//table[@class="yfnc_datamodoutline1"]//tr/td[1]/text()')

As an example, if you call:

getHistoricData('^GSPC', '02', '3', '1950', '10', '30', '2014')

the following is printed (dates form the first column):

[
    'Nov 28, 2014', 
    'Nov 26, 2014', 
    'Nov 25, 2014', 
    'Nov 24, 2014',
    ...
]

score 2 · Answer 2 · answered Dec 30 '14 at 08:42

I think you don't need to use params. Simply format a URL will be enough. Like this:

# -*- coding: utf-8 -*-
#!/usr/bin/python

import requests

symbol = raw_input("Symbol: ")
params = (symbol, '00', '11', '2010', '00', '13', '2010')

url = 'http://finance.yahoo.com/q/hp?s=%s&a=%s&b=%s&c=%s&d=%s&e=%s&f=%s&g=d' % params     
response = requests.get(url)
# you will get 200 OK here 
print response
# and page info is in response.text

7stud · Accepted Answer · 2014-12-30T22:25:12.743

The <input> elements with the name attributes equal to:

a, b, c, d, e, f, g(the radio button Daily/Weekly/Monthly)

are inside a <form> tag, which has this hidden form field:

<input type="hidden" name="s" value="^GSPC" data-rapid_p="11">

That will send a name/value pair to the server just like a regular <input> element. You need to include that name/value pair in your request, so that the server side program will know which stock you are requesting data for.

The submit button in a form also sends a name/value pair to the server, but it is rarely important, and in this case you can omit it:

import requests

url = 'http://finance.yahoo.com/q/hp'

params = {
    's': '^GSPC', #<input type="hidden" name="s" value="^GSPC" data-rapid_p="11">
    'a': 1, #stMonth,
    'b': 16, #stDate,
    'c': 2014, #stYear,
    'd': 1, #enMonth,
    'e': 18, #enDate,
    'f': 2014, #enYear,
    'g': 'd', #daily/weekly/monthly
}  

resp = requests.get(url, params=params) 
print resp.text
print resp.url

resp.url is actually the url that the request was sent to, and you can examine it by printing it:

http://finance.yahoo.com/q/hp?a=1&c=2014&b=16&e=18&d=1&g=d&f=2014&s=%5EGSPC

If you copy that into your browser's address bar, you will see the results. The resp.text is the html markup for the page that contains your results. You have to know how to search the html to find the specific results. To search html with python, check out:

BeautifulSoup
lxml

Python - Requests: Correctly Using Params?

3 Answers3