How to make sure headers that im passing to node-fetch request are correct?

Question

Im trying to rewrite one function which does the get request with request-promise library to be based on node-fetch:

Here is method in request-promise:

import { default as request } from 'request-promise';

async function sendGet(cookies, url) {
  const options = {
    method: 'GET',
    uri: url,
    headers: {
      Accept: 'text/html,application/xhtml+xml,application/xml',
      'Upgrade-Insecure-Requests': '1',
      'Accept-Encoding': 'gzip, deflate, sdch, br',
      'Accept-Language': 'en-US,en;q=0.8',
      Host: 'some.url.com',
      Connection: 'keep-alive',
      cookie: cookies,
    },
    json: true,
    resolveWithFullResponse: true,
    simple: false,
    followRedirect: false,
  };
  try {
    const response = await request(getNextOptions);
    return {
      cookies: response.headers['set-cookie'],
      location: response.headers['location'],
    };
  } catch (e) {
    throw e;
  }
}

When changed to node-fetch it gives following error in response:

"The requested URL /session/auth/failure was not found on this server. Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request."

import fetch from 'node-fetch';

async function sendGet(cookies, url) {
  const headers = {
    'Upgrade-Insecure-Requests': '1',
    'Accept-Language': 'en-US,en;q=0.8',
    Host: 'some.url.com',
    Connection: 'keep-alive',
    cookie: cookies,
  }
  try {
    const result = await fetch(url, {
      headers,
      compress: true
    });
    const body = await result.json();

    return {
      cookies: body.headers.raw()['set-cookie'],
      location: body.headers.get('location'),
    };
  } catch (e) {
    throw e;
  }

I think there might be something wrong with headers that Im passing but after couple of days of research I was unable to get it to work. Some explanation in difference of headers:

Im not using Accept header because node-fetch by default adds Accept: */*
Instead of 'Accept-Encoding': 'gzip, deflate, sdch, br' im using compression: true

Thanks in advance for help!

score 2 · Accepted Answer · answered Jan 24 '20 at 12:49

2

You can use something like https://requestbin.com/ to test your request.

Basically, you create new Request BIN, and then use the BIN url instead of your real url, and see what you are requesting there

answered Jan 24 '20 at 12:49

dkasipovic

5,930
1
19
25

Thanks a lot for this tip! this allowed me to compare requests but without host in headers because it wouldn't let me to send request to requestbin. This showed that requests are exactly the same which made me start thinking more about host in headers. After removing it I received error "maximum redirect reached" following that I noticed option on request-promise version {followRedirect: false} so only thing left to do was to add option to my fetch {redirect: 'manual'} this solved problem. Thanks again! – Piotr Lee Jan 24 '20 at 15:09
I wish I had known request bin :) – nerkn Mar 20 '21 at 11:06

score 0 · Answer 2 · answered Mar 20 '23 at 08:43

You can start listening to a local server, using netcat, like, nc -l 8080 and make the request to it. For example:

1st terminal:

$ nc -l 19990
GET /session/auth/failure HTTP/1.1
Host: localhost:19990
User-Agent: curl/7.68.0
a:b
content-type: application/json
Accept: text/html,application/xhtml+xml,application/xml
Upgrade-Insecure-Requests: 1
Accept-Encoding: gzip, deflate, sdch, br

2nd terminal:

$ curl -H "a:b" \
       -H "content-type: application/json" \
       -H "Accept: text/html,application/xhtml+xml,application/xml" \
       -H "Upgrade-Insecure-Requests: 1" \
       -H "Accept-Encoding: gzip, deflate, sdch, br" \ 
       http://localhost:19990/session/auth/failure

How to make sure headers that im passing to node-fetch request are correct?

2 Answers2