Flurry scraping using python3 requests.Session()

Question

This seems really straight forward, but for some reason this isn't connecting to flurry correctly and I unable to scrape the data.

    loginurl = "https://dev.flurry.com/secure/loginPage.do"
    csvurl = "https://dev.flurry.com/eventdata"

    session = requests.Session()
    login = session.post(loginurl, data={'loginEmail': 'user', 'loginPassword': 'pass'})
    data = session.get(csvurl)

Every time I try to use this, I get redirected back to the login screen (loginurl) without fetching the new data. Has anyone been able to connect to flurry like this successfully before?

Any and all help would be greatly appreciated, thanks.

Padraic Cunningham · Accepted Answer · 2016-07-30T18:27:50.897

1

There are two more form fields to be populated struts.token.name and the value from struts.token.name i.e token, you also have to post to loginAction.do:

You can do an initial get and parse the values using bs4 then post the data:

from bs4 import BeautifulSoup
import requests 

loginurl = "https://dev.flurry.com/secure/loginAction.do"
csvurl = "https://dev.flurry.com/eventdata"#
data = {'loginEmail': 'user', 'loginPassword': 'pass'}

with requests.Session() as session:
    session.headers.update({
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36"})

    soup = BeautifulSoup(session.get(loginurl).content)
    name = soup.select_one("input[name=struts.token.name]")["value"]
    data["struts.token.name"] = name
    data[name] = soup.select_one("input[name={}]".format(name))["value"]
    login = session.post(loginurl, data=data)

edited Jul 30 '16 at 18:27

answered Jul 30 '16 at 11:47

Padraic Cunningham

176,452
29
245
321

Thanks for the reply -- the data{} seems to populate correctly {'token': '4M9BFNE1NVFK89CZTP8CQO3MKUHWLQYZ', 'struts.token.name': 'token', 'loginPassword': 'mypass', 'loginEmail': 'myemail'} ... but im still redirected back to the login page – Nefariis Jul 30 '16 at 16:01
@Nefariis, you also had the wrong post url! See the update, now `print(login.history`) should show you `[302, 302]` and you will be logged in – Padraic Cunningham Jul 30 '16 at 18:28
You are absolutely right about the post URL, everything is working like a champ - thank you. – Nefariis Jul 30 '16 at 19:04
Hey, sorry to resurrect a solved thread but everything has been working great up until a few days ago and I cant figure out what has changed with their login process. ` print(login.history) ` shows ` [, , , , ] ` but i end back at the login page. Any chance you could take a look? None of the code has been changed since it was implemented a month ago. – Nefariis Aug 24 '16 at 17:13

Flurry scraping using python3 requests.Session()

1 Answers1

Linked