7

I'm trying to send a request to this endpoint: https://9gag.com/v1/group-posts/group/default/type/trending to get 9gag posts data.

Works fine on postman and chrome, but when I tried using axios from Node.js, it responded with 403 (and reading the returned DOM, it believe that it's asking me to enter captcha), and when I tried to send a request using node-fetch, it responded with 200 with posts data (which is the correct one).

Question is, how come node-fetch works fine but axios doesn't?

Here's the code I used for testing:

import axios from "axios";
import fetch from "node-fetch";

const URL = "https://9gag.com/v1/group-posts/group/default/type/trending";

// Inside async function
await fetch(URL); // Responded with 200 and json data
await axios(URL); // Responded with 403 and HTML DOM, axios.get() also gives the same result
sideshowbarker
  • 81,827
  • 26
  • 193
  • 197
Owl
  • 6,337
  • 3
  • 16
  • 30
  • Maybe they send different `user-agent` headers, which is a factor in the decision to serve a captcha? – Bergi Oct 05 '20 at 12:29
  • I tried sending the header with same `user-agent` value, but axios still doesn't work. Also somehow axios works fine in [repl.it I made](https://repl.it/repls/TurboLuminousAcrobat) – Owl Oct 05 '20 at 12:39
  • 2
    Look at the actual http requests getting sent to find the difference. – Bergi Oct 05 '20 at 12:52
  • Any update? I'm seeing the same thing where a site works on `fetch` but 403s on `axios` – user2402616 Jun 08 '22 at 02:41

1 Answers1

1

If you take a look at https://9gag.com/robots.txt, you might notice Mediapartners-Google. It's a User-Agent you have to specify in headers to bypass the Cloudflare captcha.

Ximaz
  • 198
  • 2
  • 9
  • Lots of other posts led to misleading info. This was the simplest and it worked for me. Seems some sites use CF to block popular things like axios, puppeteer, etc.. Simply spoofing headers to use another user-agent worked for me. (I used Mediapartners-Google). Seems like such a simple fix though. Surely..CF must have a more robust mechanism to prevent against user-agent spoofing? – user2402616 Jun 08 '22 at 18:38