Scrape with requests page with CSRF token

Question

I'm trying to scrape results from this booking website. In the POST request in addition to the trip codes and date, there is also _csrf parameter which I assume is a CSRF token. I've tried to get the session cookie and extract the CSRF token following this answer but the cookie I get back has no CSRF token

<RequestsCookieJar[<Cookie JSESSIONID=W33kBCH5zFsyVhTyQL9L4Ibyq-KLGTBSD4h_IUNA.aru-270545 for www.booking.alilaurogruson.it/booking>]>

def get_session_cookie():
    url = 'https://www.booking.alilaurogruson.it/booking/welcome/home'
    s = requests.Session()
    s.get(url, verify=False)
    print(s.cookies)
    if 'csrftoken' in s.cookies:
        # Django 1.6 and up
        csrftoken = s.cookies['csrftoken']
    else:
        # older versions
        csrftoken = s.cookies['csrf']
    return csrftoken

How can I successfully scrape the search results?

If you had succes with this, it would be very nice to see the code you mdae in the end, because I am struggling with a similar issue, and don't get past csrf verification, despite having the csrf token. — Willem van Houten, Mar 03 '23 at 10:06

score 1 · Accepted Answer · answered Mar 22 '21 at 16:45

1

_csrf is provided on that page as a form parameter you would have to parse out with something like BeautifulSoup.

<input type="hidden" name="_csrf" value="666363ca-ffff-ffff-ffff-41a61e158e0f" />

answered Mar 22 '21 at 16:45

CasualDemon

5,790
2
21
39

Scrape with requests page with CSRF token

1 Answers1