Nodejs unable to request url with request-promise

Question

I am unable to fetch https from the OpenTable website using the following code. However, when I try to fetch data from other sites e.g. Instagram using the same code, it works. As I just started learning web scraping with node js, could you let me know what could be the issue?

const request = require('request-promise');



(async () => {

    const BASE_URL = 'https://www.opentable.co.uk/s?dateTime=2021-05-30T19%3A00%3A00&covers=2&latitude=51.525225&longitude=-0.079615'


    let response = await request(BASE_URL);

    let $ = cheerio.load(response);
    
    console.log(response);

  
  })();

You probably want to set some headers. Also these days you should probably be using axios. — pguardiario, May 24 '21 at 00:38

score 1 · Accepted Answer · answered Jul 08 '21 at 08:10

Most likely your code does not work because JavaScript is used to build a DOM on the site. Or the site is using anti-scraping protection. Anyway, I would suggest you something like Puppeteer:

const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");

puppeteer.use(StealthPlugin());

async function scrapeOpenTable() {
  const BASE_URL =
    "https://www.opentable.co.uk/s?dateTime=2021-05-30T19%3A00%3A00&covers=2&latitude=51.525225&longitude=-0.079615";

  const browser = await puppeteer.launch({
    headless: false,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });
  const page = await browser.newPage();
  await page.goto(BASE_URL);
  // More commands here...
  /* You may view the docs at:
       https://pptr.dev/
     And more magic at:
       https://www.npmjs.com/package/puppeteer
     Github:
       https://github.com/puppeteer/puppeteer
  */

  // await browser.close();
}

scrapeOpenTable();

Nodejs unable to request url with request-promise

1 Answers1