Node fetch implementation seems to differ from Deno and Bun causing a website to return no response?

Question

I am trying to fetch the html for my universities login page. When using the built-in fetch in node I get the html response as expected. when using Deno and Bun however the response is nothing. Deno throws an EOF error. Bun prints nothing. Python, Golang and wget all manage to get the html as well as node. My guess is the server is closing the connection prematurely not sure if there is any work around for this.

const r = await fetch(url, {redirect: "follow"});
console.log(await r.text())

I tried the above code it works in node version 18 but not in Deno or Bun both latest versions.

Could be a difference in default headers sent with the request. You could set up a proxy to send the request through so you can see exactly what is different between the http requests that apparently the target host cares about. — jfriend00, Jul 08 '23 at 04:27
@jfriend00 tried by setting the headers to an empty object same response also tried using insomnia came back with "Error: Failure when receiving data from the peer" — user12957229, Jul 08 '23 at 04:32
Setting headers to an empty object won't necessarily change what a given library sets for default headers. When it comes right down to it, you need to create a situation where you can see exactly what the different http requests look like so you can see what's different. — jfriend00, Jul 08 '23 at 04:36

score 0 · Accepted Answer · answered Jul 08 '23 at 15:11

Because you mentioned that this problem occurs in multiple environments (Deno, Bun, Insomnia, etc.), it seems likely that the response is not valid, but some parsers handle it more gracefully (e.g. Chrome, Node, etc.). Note: I can't verify this without seeing the exact response that you are — and you haven't included the binary response data in the question.

In any case, you can create an abstraction to gracefully handle an unexpected end of file scenario during parsing — you can check that the collected data meets your criteria in your exception-handling code, and recover at that point if things seem ok. Here's an example that I tested in Deno and Node:

main.mjs:

async function parseHtmlBody(response) {
  const decoder = new TextDecoder();
  let html = "";

  try {
    for await (const u8Arr of response.body) {
      const str = decoder.decode(u8Arr, { stream: true });
      html += str;
    }
    return html;
  } catch (cause) {
    if (
      // The exception is an error:
      cause instanceof Error
      // And it includes the expected message text:
      && cause.message.toLowerCase().includes("unexpected end of file")
      // And it looks like the end of the document was received:
      && html.trimEnd().endsWith("</html>")
    ) return html;
    // Else, rethrow:
    throw cause;
  }
}

async function main() {
  // The URL in your question details:
  const url = "https://www.uvic.ca/cas/login";
  const response = await fetch(url, { redirect: "follow" });
  const html = await parseHtmlBody(response);
  // Print the beginning and end of the HTML document and its length:
  console.log(html.slice(0, 100));
  console.log("…");
  console.log(html.slice(-100));
  console.log("Total length:", html.length);
}

main();

In the terminal:

% node --version
v18.16.1

% node main.mjs
<!DOCTYPE html><html lang="en">

<head>
    <meta charset="UTF-8" /><meta name="viewport" content="w
…
t src="https://kit.fontawesome.com/32795de2d2.js" crossorigin="anonymous"></script>
</body>
</html>

Total length: 7236

% deno --version
deno 1.35.0 (release, aarch64-apple-darwin)
v8 11.6.189.7
typescript 5.1.6

% deno run --allow-net main.mjs
<!DOCTYPE html><html lang="en">

<head>
    <meta charset="UTF-8" /><meta name="viewport" content="w
…
t src="https://kit.fontawesome.com/32795de2d2.js" crossorigin="anonymous"></script>
</body>
</html>

Total length: 7236

Both outputs look the same. You can modify the code above to meet your needs.

Node fetch implementation seems to differ from Deno and Bun causing a website to return no response?

1 Answers1