11

I am currently testing out GET requests with Postman. I just want to see the kind of information I can get back. Simply plugging in the url for a website works for the most part. However, there is a particular website that is giving me trouble. When I try to initiate the GET request, I am given the following error:

<title>Access denied | <url> used Cloudflare to restrict access</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css"
    media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->

The website works fine when launched directly in Chrome. However, every time I try to sent the GET in Postman, I get the Response 403. I've tried a few solutions already:

  • Copy and paste the User-Agent displayed in the Chrome Inspect tools (Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36).
  • Set up Postman interceptor system with status "INTERCEPTOR CONNECTED". Also installed the plug-in in my Chrome extensions.
  • Enabled the cookies for the website.

Anyway, if I'm missing some glaringly obvious problem or you require more information, please reach out! I'm trying not to share the direct link to the website but I will if there's no other way.

Sean
  • 592
  • 4
  • 9
  • 19

2 Answers2

6

The website is using Cloudflare which is a security and content delivery product.

When Cloudflare is used, all traffic to the website goes through Cloudflare first, where various security checks are made. This is an industry leading system which has many sophisticated checks, most of which can be configured by the customer for their particular website. It helps to prevent things like malicious bots and attacks on the website.

A 403 error means you have been blocked from accessing the resource. This is probably because the firewall at Cloudflare has decided the request from Postman is malicious or unwanted, so it's blocking you. The website works fine when you visit in your browser, because this is considered a "normal" request.

BadHorsie
  • 14,135
  • 30
  • 117
  • 191
  • But is there a bypass for that. I am basically trying to scrape a wordpress page and am constantly denied access. I use Beautiful soup and adding headers only triggers the captcha. any workarounds? – Surya Palaniswamy Jun 01 '21 at 07:22
  • 1
    @SuryaPalaniswamy If you're trying to scrape someone's site, that's probably precisely why they want to use Cloudflare to deny you. – BadHorsie Sep 06 '21 at 15:33
3

Use this code to simulate a browser,

header = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'}
Derek Caramella
  • 47
  • 1
  • You should probably explain why would changing the user agent help, and what is the user agent you set. – A. Abramov May 04 '21 at 17:25
  • https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent The User-Agent request header is a characteristic string that lets servers and network peers identify the application, operating system, vendor, and/or version of the requesting user agent. – Gustavo Contreiras Oct 22 '21 at 10:39
  • 4
    This doesn't work, cloudflare protection is not so easy to bypass by just using a custom header – Shekhar Samanta Dec 12 '21 at 18:41