Unable to log in to a site requiring unconventional payload to be sent with post requests

Question

I'm trying to log in to a website using requests module. While creating a script to do so, I could notice that the payload used in there is completely different from the conventional approach. This is exactly how the payload +åEMAIL"PASSWORD(0 looks like. This is the content type parameters content-type: application/grpc-web+proto.

The following is what I see in dev tools when I log in to that site manually:

General
--------------------------------------------------------
Request URL: https://grips-web.aboutyou.com/checkout.CheckoutV1/logInWithEmail
Request Method: POST
Status Code: 200 
Remote Address: 104.18.9.228:443

Response Headers
--------------------------------------------------------
Referrer Policy: strict-origin-when-cross-origin
access-control-allow-credentials: true
access-control-allow-origin: https://www.aboutyou.cz
access-control-expose-headers: Content-Encoding, Vary, Access-Control-Allow-Origin, Access-Control-Allow-Credentials, Date, Content-Type, grpc-status, grpc-message
cf-cache-status: DYNAMIC
cf-ray: 67d009674f604a4d-SIN
content-encoding: gzip
content-type: application/grpc-web+proto
date: Wed, 11 Aug 2021 08:19:04 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
set-cookie: __cf_bm=a45185d4acac45725b46236884673503104a9473-1628669944-1800-Ab2Aos6ocz7q8B8v53oEsSK5QiImY/zqlTba/Y0FqpdsaQt2c10FJylcwTacmdovm6tjGd8hLdy/LidfFCtOj70=; path=/; expires=Wed, 11-Aug-21 08:49:04 GMT; domain=.aboutyou.com; HttpOnly; Secure; SameSite=None
vary: Origin


Request Headers
--------------------------------------------------------
:authority: grips-web.aboutyou.com
:method: POST
:path: /checkout.CheckoutV1/logInWithEmail
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: no-cache
content-length: 48
content-type: application/grpc-web+proto
origin: https://www.aboutyou.cz
pragma: no-cache
referer: https://www.aboutyou.cz/
sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"
sec-ch-ua-mobile: ?0
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: cross-site
user-agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36
x-grpc-web: 1

Request Payload
--------------------------------------------------------
+åEMAIL"PASSWORD(0

This is what I've created so far (can't find any way to fill in the payload):

import requests
from bs4 import BeautifulSoup

start_url = 'https://www.aboutyou.cz/'
post_link = 'https://grips-web.aboutyou.com/checkout.CheckoutV1/logInWithEmail'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.3',
    'content-type': 'application/grpc-web+proto',
    'origin': 'https://www.aboutyou.cz',
    'referer': 'https://www.aboutyou.cz/',
    'x-grpc-web': '1'
}
payload = {
    
}

with requests.Session() as s:
    s.headers.update(headers)
    r = s.post(post_link,data=payload)
    print(r.status_code)
    print(r.url)

Steps to log in to that site manually:

Go to this site
This is how to get the login form
Login form looks like this

How can I log in to that site using requests module?

I always use Postman to generate Python code for me to do things like this. I first right-click on a request in Firefox's network tab and click "Copy" -> "Copy as cURL". Then in Postman, I click on "Import" and paste in "Raw text". Then under the request that was created, I click on "Code" and then "Python - Requests". — D Malan, Aug 16 '21 at 20:48

Life is complex · Accepted Answer · 2021-08-18T12:50:18.007

I don't think that you'll be able to use Python Requests to login to your target site.

Your post_link url:

post_link = 'https://grips-web.aboutyou.com/checkout.CheckoutV1/logInWithEmail'

states that it is: gRPC requires HTTP/2 and Python Requests send HTTP/1.1 requests only.

Additionally, I noted that the target site also uses CloudFlare, which is difficult to bypass with Python, especially when using Python Requests

'Set-Cookie': '__cf_bm=11d867459fe0951da4157b475cf88eb3ab7658fb-1629229293-1800-AeFomlmROcmUYcRosxxcSnoJkGOW/WXjUe1WxK6SkM2eXIbnAqXRlpwOkpvOfONrbApJd4Qwj+a8+kOzLAfpHIE=; path=/; expires=Tue, 17-Aug-21 20:11:33 GMT; domain=.aboutyou.com; HttpOnly; Secure; SameSite=None', 'Vary': 'Accept-Encoding', 'Server': 'cloudflare', 'CF-RAY': '6805616b8facf1b2-ATL', 'Content-Encoding': 'gzip'}

Here are previous Stack Overflow questions on Python Requests with gRPC

I looked through the GitHub repository for Python Requests and saw that HTTP/2 has been a requested feature for almost 7 years.

During my research, I discovered HTTPX, which is a HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2. The documentation states that the package is stable, but is still considered a beta at this point. I would recommend trying HTTPX to see if it solves your issue with logging into your target site.

Unable to log in to a site requiring unconventional payload to be sent with post requests

1 Answers1