How to get raw file data from node-fetch calls?

Question

Supposing I have an endpoint like https://someurl.com/myfile.mp4. I want to fetch the mp4 file and insert its raw data into a string in order to calculate the sha256 of the entire file content. How to do this using the node-fetch library? All the examples I found on the internet assumes that an json object is returned from the query, is it always the case and I should use another library? Here's an example code:

let urlFinal = 'https://someurl.com/myfile.mp4';
let videoFile = await fetch(urlFinal, {method: 'GET'}).then(res => { return res.blob() });
console.log('videofile: ' + videoFile) // prints out: videofile: [object Blob]
console.log('videofile: ' + JSON.stringify(videoFile)) // prints out: videofile: {}
calculateHash(videoFile)

I also didn't found anything about the blob() function and object in the documentation. So what is the simplest way to reach the file data in this case? thanks in advance

score 1 · Accepted Answer · edited Jan 22 '22 at 13:12

Download video raw content in memory like this:

const downloadNew = () => {
    http.get(urlFinal, (res) => {
        const { statusCode } = res;
        const contentType = res.headers['content-type'];

        let error;
        if (statusCode !== 200) {
            error = new Error('Request Failed.\n' +
                `Status Code: ${statusCode}`);
        } 
        
        if (error) {
            console.error(error.message);
            res.resume();
            return;
        }

        res.setEncoding('utf8');
        let rawData = '';
        res.on('data', (chunk) => { rawData += chunk; });
        res.on('end', () => {
            try {
                console.log(rawData);
            } catch (e) {
                console.error(e.message);
            }
        });
    }).on('error', (e) => {
        console.error(`Got error: ${e.message}`);
    });
}

You can do it using node-fetch as well as with nodejs native code. I'll demonstrate both ways of downloading content in Nodejs (with/without using third party Libs):

Download using Node-Fetch library:

const http = require('http');
const fs = require('fs');
const urlFinal = 'http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4';
const util = require("util");
const fetch = require("node-fetch");
const newFilePath = "/home/amulyakashyap/Desktop/download/BigBuckBunny.mp4";

const downloadUsingFetch => ()=>{
  fetch(urlFinal)
      .then(res => {
          const dest = fs.createWriteStream(newFilePath);
          res.body.pipe(dest);
      }).then(()=>console.log("finished"))
      .catch(console.error);
  }

Download using Native Nodejs (without third party lib support):

const downloadUsingNative = function(cb) {
  let file = fs.createWriteStream(newFilePath);
  let request = http.get(urlFinal, function(response) {
    response.pipe(file);
    file.on('finish', function() {
     console.log("done")
      file.close(cb);

    });
  }).on('error', function(err) {
    fs.unlink(dest);
    if (cb) cb(err.message);
  });
};

const downloadUsingNativePromisified = util.promisify(downloadUsingNative);

downloadUsingNativePromisified().then(console.log).catch(console.error);

but i don't need to write the file to disk, i just need to get its data as string, in memory. isn't another way where I dont need to create a file with `fs`? — Fnr, Apr 08 '21 at 19:24
@Fabiotk I have updated the answer, please check the very first approach in the answer. I hope it works for you. Also refer to this link : https://stackoverflow.com/questions/61354719/in-node-js-how-can-i-write-to-file-chunks-uploaded-from-a-video — Amulya Kashyap, Apr 08 '21 at 20:38

How to get raw file data from node-fetch calls?

1 Answers1