0

I've been trying to migrate a simple web scraping script to NodeJS using request-promise package but I'm always getting the following error as output

403 This IP has been automatically blocked

However, if I use my browser or Postman to trigger the request it works perfectly (and the IP is not blocked at all)

Here's the code I'm using for NodeJS

const request = require('request-promise');
const main = async () => {
    const options = {
        url: 'https://sfbay.craigslist.org/d/software-qa-dba-etc/search/sof',
        headers: {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36',
            'Accept-Language':'en-US,en;q=0.5',
            'Cache-Control': 'no-cache'
        }
    }
    try {
        const htmlResult = await request.get(options);
        console.log(htmlResult);
    } catch (e) {
        console.log(e);
    }

}

main();

I've also tried with Axios, but the output is the same. Any thoughts?

WSD
  • 3,243
  • 26
  • 38
  • 1
    Take a look at the red box at the top of https://www.npmjs.com/package/request-promise (not the cause of your problem, but something better solved now rather than after you've done any more work with that module). – Quentin Mar 22 '21 at 10:07
  • You should use PROXY service like [BrightData](https://brightdata.com/) – s4suryapal Apr 04 '21 at 18:41

0 Answers0