0

I'm doing my best to upload a file directly to google cloud storage, using koa.js and multer. I carefully read through the other SO questions, google api documentation and several tutorials and was able to upload the file successfully, but I have a question.

From what I understand the sequence I made with my code is:
FILE >>> Upload to backend (memorystorage) >>> upload to Google Cloud Storage.

I would like the file to start uploading to google "directly", without first being fully read by the backend and loaded into memory.

I tried and wrote two different approaches (see code below) but in my opinion neither of them works as I would like, and having to upload even large files I don't want to risk creating performance problems for the server during uploads.

Can you help me better understand how to do it and also explain the concrete difference of the two codes I wrote?

import Koa from 'koa';
import KoaRouter from '@koa/router';
import { Storage } from '@google-cloud/storage';
import multer from '@koa/multer';

const kr = new KoaRouter();

const upload = multer({
  storage: multer.memoryStorage(),
},

kr.post('/upload', multer.single('uploadedFile'), async (ctx, next) => {

  const fileToUpload = ctx.request.file.buffer

  // google cloud storage bucket and file (omitting auth part)
  const storage = new Storage();
  const bucket = storage.bucket('testbucket');
  const newFile = bucket.file('testfile');

  // SOLUTION A: Create a pass through stream
  await new Promise((resolve, reject) => {
    console.log(`File upload START: `, new Date());
    const passthroughStream = new Stream.PassThrough();
    passthroughStream.write(fileToUpload);
    passthroughStream.end();
    passthroughStream
      .pipe(newFile.createWriteStream({ resumable: false, gzip: true, contentType: fileMimeType }))
      .on('finish', () => {
        console.log(`File upload END: `, new Date());
        resolve(true);
      });
  });

  // SOLUTION B: Create a readable stream
  await new Promise((resolve, reject) => {
    console.log(`File upload START: `, new Date());
    const stream = Readable.from(fileToUpload);
    stream
      .pipe(newFile.createWriteStream({ resumable: false, gzip: true, contentType: fileMimeType }))
      .on('finish', () => {
        console.log(`File upload END: `, new Date());
        resolve(true);
      });
  });

  console.log(`File successfully uploaded`);
})
crivella
  • 514
  • 5
  • 19

0 Answers0