Node-Fetch POST returning different result from Firefox POST

Question

I am able to successfully log into a website using Firefox but having trouble replicating that log in using node-fetch. There are three stages to the log-in process:

go to /login and website responds with sessionToken (this part is working OK)
enter email and password alongside sessionToken and website responds with sessionToken and authToken (this is the part I am having trouble with)
request with sessionToken and authToken to /portal and website responds with the HTML I am trying to access

in Firefox Dev Tools, I can see both the working headers and working request body. When I click on "Request" in Dev Tools, I see a 'Form data' heading with the proper values of 'email' and 'password'. When I click on "Headers" in Dev Tools, here is the successful Firefox request:

POST /login/action HTTP/2
Host: www.website.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded
Content-Length: 46
Origin: https://www.website.com
Connection: keep-alive
Referer: https://www.website.com/login
Cookie: _sessiontoken=sessionTokenHere
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
TE: trailers

On Firefox, that returns a 302 (redirect) response with these headers:

HTTP/2 302 Found
date: Sun, 31 Oct 2021 17:25:03 GMT
content-type: text/html; charset=utf-8
cache-control: no-cache
location: https://www.website.com/portal
x-runtime: 9
set-cookie: auth_token=authtokenHere; path=/
set-cookie: 
set-cookie: _sessiontoken=sessionTokenHere; path=/; HttpOnly
cf-cache-status: DYNAMIC
[omitted for brevity: expect-ct, report-to, nel, server, cf-ray, alt-svc, X-Firefox-Spdy]

Now, here are the POST options I am trying to use with node-fetch in node.js:

{
method: 'POST', headers: {
Host: 'www.website.com'
,'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0'
,Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'Content-Length': 46,
Origin: 'https://www.website.com',
Connection: 'keep-alive',
Referer: 'https://www.website.com/login',
Cookie: '_sessiontoken=sessionTokenHere'
'Upgrade-Insecure-Requests': '1',
'Sec-Fetch-Dest': 'document',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-Site': 'same-origin',
'Sec-Fetch-User': '?1',
TE: 'trailers',
body: URLSearchParams { 'email' => 'myEmail','password' => 'myPassword'}

However, when using node-fetch with the above options it returns a 200, not a 302. Here are the headers of that response (printed via console.log() within the fetch() function):

 [Object: null prototype] {
date: [ 'Sun, 31 Oct 2021 18:12:01 GMT' ],
'content-type': [ 'text/html; charset=utf-8' ],
'transfer-encoding': [ 'chunked' ],
connection: [ 'keep-alive' ],
vary: [ 'Accept-Encoding' ],
'x-runtime': [ '5' ],
'cache-control': [ 'private, max-age=0, must-revalidate' ],
 'set-cookie': [
'_sessionToken=sessionTokenHere; path=/; HttpOnly'
],
'cf-cache-status': [ 'DYNAMIC' ],
'content-encoding': [ 'br' ],
[omitted for brevity: expect-ct, report-to, nel, server, cf-ray, alt-svc]
}

I am puzzled why it works in Firefox but not with node-fetch. A few notes:

I suspect I am somehow including the email and password incorrectly, but I can't figure out how
The working Firefox request includes "X-Firefox-Spdy" in the response header; the nonworking node-fetch request does not.
I am not including "Content-Type" in my request per suggestion from this post
The Firefox request seems to be using HTTP/2. Could this be part of the problem?

The session token comes from a prior request. I don't know if this makes sense but: the login process works in 3 sections. The first one seems to be working fine, this is the middle one that provides auth_token that isn't working. I edited the post to clarify this. — garson, Oct 31 '21 at 19:10
At first glance it looks like you need to add back ```Content-Type: application/x-www-form-urlencoded```, since you're sending ```URLSearchParams``` in the body of the ```node-fetch```request. Was that not working? Also, what is in the body of the request sent by Firefox? — Carlton Lindsay, Nov 04 '21 at 07:07
Right, that's not working either, same result. When I click on "Request" in Dev Tools, I see a 'Form data' heading with the proper values of 'email' and 'password'. Let me know if there's other information about the Firefox request body that would be helpful. — garson, Nov 04 '21 at 13:48

Bertrand Martel · Accepted Answer · 2021-11-06T23:20:06.707

node-fetch automatically follows redirects. It will automatically go to the next redirected url (Location http header value).

If you want to catch the 301 or 302 call, you can set redirect property to manual in the options like redirect: "manual", this way you manage the redirection yourself

For example the following code will catch the 301 call (it'll do the same for your 302 in theory, implemented here):

const fetch = require("node-fetch");

(async () => {
  const response = await fetch("https://stackoverflow.com/users/2614364", {
    redirect: "manual",
  });
  console.log(response.status);
  console.log(response.headers.get("set-cookie"));
})();

I suppose you want to get the cookies using response.headers.get("set-cookie")

Node-Fetch POST returning different result from Firefox POST

1 Answers1