2

I am running mitmproxy with an upstream to remote proxy.

mitmweb --set mode=upstream:http://proxyIp:proxyPort --set ssl_insecure=true

The application flow is:

  1. Make a HTTP request in Python and use mitmproxy server as proxies argument
  2. Intercept the call in mitmproxy, and do an upstream to another proxy.
  3. Return the results.

The HTTP request is made to the external API (I don't have access to it) protected by CloudFlare.

headers = {
    'User-Agent': 'PostmanRuntime/7.26.7',
    'Accept': 'application/json'
}

r = requests.get("https://api.website.com/",
                 headers=headers,
                 verify=False,
                 proxies={'http': 'http://127.0.0.1:8080', 'https': 'http://127.0.0.1:8080'})

Running this request will result in a 403 response from https://api.website.com/. The result is the same if I skip the mitmproxy part and connect to the end proxy directly from Python. How ever, I tried using Fiddler as a Gateway and it worked good (It's certainly modifying the request in a background).

<span>Error</span><span>1020</span>

If I run the same request with curl the result will be good (200 OK)

curl -x 127.0.0.1:8080 -k 'https://api.website.com/' -H 'user-agent: PostmanRuntime/7.26.7' -H 'accept: application/json'

Those two requests seem identical, yet the Python one returns 403.

Am I missing something in the Python config? Setting some protocol or headers?

NOTE:

I tried running the curl by directly connecting to the end proxy (skipping the mitmproxy), and the request is also failing with a 403 response.

curl -x proxyIp:proxyPort -k 'https://api.website.com/' -H 'user-agent: PostmanRuntime/7.26.7'

Then I tried by using the curl-openssl/bin/curl and it worked, how ever I had to add --tlsv1.3 to it.

/usr/local/opt/curl-openssl/bin/curl --tlsv1.3 -x proxyIp:proxyPort -k 'https://api.website.com/' -H 'user-agent: PostmanRuntime/7.26.7'
Dino
  • 7,779
  • 12
  • 46
  • 85
  • 1
    If the same request works in Fiddler but does not work in Python this indicates that CloudFlare performs client finger printing (e.g. based on TLS handshake and further data) and therefore rejects certain requests. I would recommend to look at the requests in Wireshark to see the differences of the TLS handshake. – Robert Nov 02 '20 at 11:16

0 Answers0