Authentication POST works in 'request' but not with 'node-fetch'

Question

I've written an authentication request that works with the request module but fails to return the desired 'authentication token' cookie with the node-fetch module that I'm trying to move towards since 'request' has been deprecated.

Here is the working code using 'request'

  var callback1 = function(err, httpResponse, body){
    console.log("Correctly prints all the cookies we want: ");
    console.log(httpResponse.headers["set-cookie"]);
    if (err){console.log("here it is!"); throw err;}
    else {
      //do more with response
    }
  };
 var callback0 = function(err, httpResponse0, body){

    console.log("Check that sent cookies are identical, from request:");
    console.log(httpResponse0.headers["set-cookie"][0].substr(0,httpResponse0.headers["set-cookie"][0].indexOf(";")));
    if (err){throw err;}
    else {
    var options1 = {
      url: urlString1
      ,headers:{
      'Host': hostName
      ,'User-Agent': myUserAgent
      ,'Accept': myAccept 
      ,'Accept-Language':myAcceptLanguage 
      ,'Accept-Encoding': myAcceptEncoding 
      ,'Referer': "https://"+hostName+'/login'
      ,'Cookie': httpResponse0.headers["set-cookie"][0].substr(0,httpResponse0.headers["set-cookie"][0].indexOf(";"))
      ,'Content-Type':  "application/x-www-form-urlencoded"
      ,'Content-Length': Buffer.byteLength(querystring.stringify(postData))
      ,'Connection': "keep-alive"
      ,'Upgrade-Insecure-Requests': "1"
      ,'DNT': "1"
      ,'TE': "Trailers"
      }
      ,form: {email: myEmail, password: myPassword}
    };
    request.post(options1, callback1);
    }
  }
 var options0 = {
    url: urlString0
    ,headers:{
    'Host': hostName
    ,'User-Agent': myUserAgent
    ,'Accept': myAccept 
    ,'Accept-Language':myAcceptLanguage 
    ,'Accept-Encoding': myAcceptEncoding 
    ,'Referer': "https://"+hostName
    ,'Connection': "keep-alive"
    ,'Upgrade-Insecure-Requests': "1"
    }
  };
  request(options0, callback0);

And here is the code in node-fetch that's failing to properly return the auth_token cookie:

  const fetchOptions0 = {
      method: 'GET', 
      headers:{
      'Host': hostName
      ,'User-Agent': myUserAgent
      ,'Accept': myAccept 
      ,'Accept-Language':myAcceptLanguage 
      ,'Accept-Encoding': myAcceptEncoding 
      ,'Referer': "https://"+hostName
      ,'Connection': "keep-alive"
      ,'Upgrade-Insecure-Requests': "1"
      }
  }
  fetch(urlString0, fetchOptions0)
  .then(res0 => {
    console.log("Check that sent cookies are identical, from fetch:");
    console.log(res0.headers.raw()['set-cookie'][0].substr(0, res0.headers.raw()['set-cookie'][0].indexOf(";")));
    const FormData = require('form-data');
    const myForm = new FormData();
    myForm.append('email', myEmail);
    myForm.append('password', myPassword);
    var fetchOptions1 = {
      method: 'POST'
      ,headers:{
      'Host': hostName
      ,'User-Agent': myUserAgent
      ,'Accept': myAccept 
      ,'Accept-Language':myAcceptLanguage 
      ,'Accept-Encoding': myAcceptEncoding 
      ,'Referer': "https://"+hostName+'/login'
      ,'Cookie': res0.headers.raw()['set-cookie'][0].substr(0, res0.headers.raw()['set-cookie'][0].indexOf(";"))
      ,'Content-Type':  "application/x-www-form-urlencoded"
      ,'Content-Length': Buffer.byteLength(querystring.stringify(postData))
      ,'Connection': "keep-alive"
      ,'Upgrade-Insecure-Requests': "1"
      ,'DNT': "1"
      ,'TE': "Trailers"
    }
    ,body:myForm
    };
    fetch(urlString1, fetchOptions1)
    .then(res1=> {console.log("Incorrect cookie, missing auth:"); console.log(res1.headers.raw()['set-cookie']);});
    });

I have tried writing the form data using JSON.stringify as suggested in this answer, but that didn't help.

Can you show the console.logs of both the request and node-request implementations. — jeeves, Feb 24 '20 at 21:09

x00 · Answer 1 · 2020-02-23T08:06:57.903

0

There is

,form: {email: myEmail, password: myPassword}

in request version of code. It is application/x-www-form-urlencoded
See https://www.npmjs.com/package/request#applicationx-www-form-urlencoded-url-encoded-forms)

But there is

body: myForm

in fetch version of code. And it is multipart/form-data
See

But still

'Content-Type': "application/x-www-form-urlencoded"

in the fetch version

If your API endpoint doesn't accept multipart/form-data you probably should switch from FormData to URLSearchParams.
See https://www.npmjs.com/package/node-fetch#post-with-form-parameters

edited Feb 23 '20 at 08:06

answered Feb 22 '20 at 21:18

x00

13,643
3
16
40

I've tried using UrlSearchParams, too, but unfortunately the cookie still doesn't come through. – garson Feb 23 '20 at 19:31
Then, please, add request and response headers from both versions of your code – x00 Feb 23 '20 at 20:29
and by the way - what is `postData`. I do not see it's being posted. – x00 Feb 23 '20 at 20:32
Maybe `fetchOptions1.body` size does not match `Content-Length`. And why would you set it manually, anyway? – x00 Feb 23 '20 at 20:43
I had set it manually because I was trying to send the exact same request, line for line, that I saw in Dev Tools, and I wasn't sure it would automatically set otherwise. I have tried removing that and it still doesn't return the right cookie. – garson Feb 23 '20 at 21:43
1) remove headers from `request` version one by one while `request` still works 2) remove same headers from `fetch` version 3) use `UrlSearchParams` with `fetch` instead of `FormData`. If you'd do all these steps it should work... if not then in DevTools you should be able to right-click on a request done by `request` version and select `Copy/Copy as fetch`, maybe that would do the trick. Any way, update you question with requests and responses. – x00 Feb 24 '20 at 19:21
You also can try `body: querystring.stringify(postData)`. I still have no clue what the `postData` is. But the code in the question is definitely not the full story, so this advice can still work for you – x00 Feb 24 '20 at 19:31
Strangely it still doesn't work. I was able to get the main "request" down to no headers at all! It seems like the problem is with the submission of the "form." Sorry I didn't post postData, it's just the object {email: myEmail, password: myPassword} – garson Feb 26 '20 at 03:50

score 0 · Answer 2 · answered Feb 25 '20 at 16:14

To send an application/x-www-form-urlencoded payload into the body, simply pass a URLSearchParams object

const params = new URLSearchParams();
params.append("email", myEmail);
params.append("password", myPassword);

fetch(urlString1, {
  headers: {
    // your headers
    // omit content-length and content-type because node-fetch
    // will handle that for you
  },
  body: params
}).then(response => {
  console.log(response.headers.raw());
})

In your example, you seem to be using FormData, which results in node-fetch writing multipart/formdata as the header's content-type.

Also you might want to check your authorization server code to see if it sends the set-cookie headers correctly.

Tested with:

node-fetch v2.6.0
nodejs v12.14.0

As noted below, this doesn't work :( is it any hint that URLSearchParams() isn't blue when I type it? — garson, Feb 26 '20 at 04:46
You might be using old nodejs where URLSearchParams must be required first using `const { URLSearchParams } = require("url");` — Alan Darmasaputra, Feb 27 '20 at 16:52

Authentication POST works in 'request' but not with 'node-fetch'

2 Answers2