Node - Piping remote http large video file stream is slower than serving remote file url directly

Question

I have a simple node server that I want to use to act as a proxy to stream a large video file from an external server (e.g. google drive, dropbox, etc.) which is set up as so:

var http = require("http");
var request = require("request");

var hostname = "localhost";
var port = 3000;

var decode = (encodedString) =>
  Buffer.from(encodedString, "base64").toString("ascii");

var server = http.createServer((req, res) => {
  var rid = parseInt(Math.random() * 10000);

  var url = new URL(`http://${hostname}:${port}${req.url}`);

  if (url.pathname == "/download" && url.searchParams.get("file")) {
    var remoteFileUrl = decode(url.searchParams.get("file"));

    var headers = {};
    if (req.headers.range) headers.range = req.headers.range;

    var fileReq = request(remoteFileUrl, { headers });
    fileReq.pipe(res).on("close", () => {
      fileReq.destroy();
    });
  } else {
    res.statusCode = 200;
    res.setHeader("Content-Type", "text/plain");
    res.end(":)\n");
  }
});

server.listen(port, "0.0.0.0", () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

Now I have this node app running on my own private VPS which has a high network speed of about 5000 mbits/s and 3GB RAM, yet when I try to play this video file by url (http://my-vps-ip/download?file=remoteFileUrlEncodedInBase64) on a media player such as VLC, it lags compared to playing the remotefile directly from the origin server.

I also tested if the node server is able to serve the file to its full bandwidth by downloading the file from another seperate VPS (different cloud) and it was able to download the file from my server at top speed.

For any files that I try to stream that are below 40GB, there doesnt seem to be lag, but for files greater than 60gb, the buffering is there while there is none when playing the file directly from remote origin server

Is there anything I can do to improve performance here? Or why my serving the stream through my server may not be responsive?