nodejs fetch does not return url data

Question

I'm trying to get the json that the url returns. But with this call it returns a response without the data and a status of 200

url: https://www.instagram.com/explore/tags/jeep/?a=1

Code:

const express = require("express");
const fetch = require("node-fetch")
const router = express.Router();

const response = await fetch('https://www.instagram.com/explore/tags/jeep/?__a=1');
const data = await response;
console.log(data)

module.exports = router;

What is shown in the console:

Response {
  size: 0,
  timeout: 0,
  [Symbol(Body internals)]: {
    body: Gunzip {
      _writeState: [Uint32Array],
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 5,
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: true,
      bytesWritten: 0,
      _handle: [Zlib],
      _outBuffer: <Buffer 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 01 e2 33 19 87 00 00 00 40 00 00 00 00 00 00 00 41 e2 33 19 87 00 00 00 40 00 00 00 00 00 00 00 81 e2 ... 16334 more bytes>,
      _outOffset: 0,
      _chunkSize: 16384,
      _defaultFlushFlag: 2,
      _finishFlushFlag: 2,
      _defaultFullFlushFlag: 3,
      _info: undefined,
      _maxOutputLength: 4294967296,
      _level: -1,
      _strategy: 0,
      [Symbol(kCapture)]: false,
      [Symbol(kCallback)]: null,
      [Symbol(kError)]: null
    },
    disturbed: false,
    error: null
  },
  [Symbol(Response internals)]: {
    url: 'https://www.instagram.com/accounts/login/?next=/explore/tags/jeep/',
    status: 200,
    statusText: 'OK',
    headers: Headers { [Symbol(map)]: [Object: null prototype] },
    counter: 1
  }
}

How do I get the normal json with all the hashtag data?

are you trying to send a POST or GET request? If you are sending a POST request you need to modify the fetch request accordingly I guess — besjon_c, May 29 '22 at 21:40

jfriend00 · Answer 1 · 2022-05-30T00:01:38.743

1

Assuming you're tring to get a response that is application/json, then change from this:

const data = await response;

to this:

const data = await response.json();

But, the specific request you show returns HTML not JSON so you can't parse the HTML as JSON anyway.

You could fetch the HTML itself with:

const data = await response.text();

But, that will just get you a page of HTML which would have to be parsed in order to do something useful with it.

You're going to have to back up a few steps and rethink what you're trying to do and how to get that data. You could use an HTML engine like Cheerio or Puppeteer to parse the HTML for you.

edited May 30 '22 at 00:01

answered May 29 '22 at 21:52

jfriend00

683,504
96
985
979

I removed the ".json()", but even with it it doesn't work – Gabriel Edu May 29 '22 at 23:46
1

@GabrielEdu - You have to ADD the `.json()`, not remove it. And, please don't just say "it doesn't work". If you have a specific problem, describe exactly what you observed and logged. That's the only way we can help you further. – jfriend00 May 29 '22 at 23:51
@GabrielEdu - Of course, this fix requires that the content-type of your response is `application/json`. If it's not, then `.json()` won't work and you apparently don't have JSON data in the response anyway. – jfriend00 May 29 '22 at 23:52
@GabrielEdu - Your specific request returns HTML, not JSON so you cannot parse that as JSON. You can get the HTML with `const data = await response.text()`, but that will just get you the text of an HTML page that has Javascript data in it, but it's in an HTML page. – jfriend00 May 29 '22 at 23:56
@GabrielEdu - FYI, a little error handling (try/catch around your await) would show you the error that `await response.json()` was throwing. – jfriend00 May 30 '22 at 00:02

nodejs fetch does not return url data

1 Answers1