How to send file from a multipart form request into another multipart form request

Question

I currently have a gateway server which receives multipart form requests where each request contains a file and some fields (user id, name of file, etc). After the server does some validation and a couple other things, I would like to send the original file with a few different fields to a separate file storage server of mine. In other words, I change the fields, keep the file the same, and send it to a 2nd server.

I currently have this working about 80% of the time, but the rest of the time I get an error on the storage server saying, Error: Part terminated early due to unexpected end of multipart data. (Happens at the comment 'ERROR PRINTED HERE'). Again, this bug only happens sometimes.

Gateway Server

const asyncBusboy = require( 'async-busboy' );
const request = reqwuire( 'request-promise' );

async function saveFile( context )
{
    //parses the multipart form request. 'files' is an array of Readable Streams
    const { files, fields } = await asyncBusboy( context.req ); //NEVER breaks here

    if ( files.length == 0 )
        return;

    let form_data =
    {
        directory: 'my/files/directory',
        filename: 'filename',
        readstream: files[0]
        //etc, etc, etc
    };

    let options =
    {
        method: 'POST',
        uri: file_storage_server_url + '/song',
        headers: { 'content-type': 'multipart/form-data' },
        formData: form_data
    };

    await request( options ); //send to storage server
}

File Storage Server

const asyncBusboy = require( 'async-busboy' );

async function saveFile( context )
{
    const { files, fields } = await asyncBusboy( context.req ); //BREAKS HERE

    //more stuff
}

AsyncBusboy

What asyncBusboy does internally is it creates a temporary file in the /tmp directory, pipes the incoming file's read stream to the temporary file, creates a read stream to the temporary file, and pushes it onto that 'files' array above. However, I sometimes receive the error I mentioned during the piping.

function onFile(files, fieldname, file, filename, encoding, mimetype) 
{
  const tmpName = file.tmpName = new Date().getTime()  + fieldname  + filename;
  const saveTo = path.join(os.tmpdir(), path.basename(tmpName));

  file.on('end', () => {
    const readStream = fs.createReadStream(saveTo);
    readStream.fieldname = fieldname;
    readStream.filename = filename;
    readStream.transferEncoding = readStream.encoding = encoding;
    readStream.mimeType = readStream.mime = mimetype;
    files.push(readStream);
  });

  const writeStream = fs.createWriteStream(saveTo);

  writeStream.on('open', () => {
    console.log( "onFile onOpen" ); //Always prints
    file.pipe( fs.createWriteStream(saveTo) );
  });

  //******* ERROR PRINTED HERE ********* 
  file.on( 'error', (error) => { console.log( "Readstream error: " + error ); } );
}

What I THINK is happening

I think the storage server is reading from the tmp file faster than it's being written to causing it to reach the end of the file and trigger the error, Part terminated early due to unexpected end of multipart data.

Questions

Is the overall approach I'm taking wrong?

Is the way async-busboy does things internally fundamentally wrong?

How can I resolve this bug?

How to send file from a multipart form request into another multipart form request

0 Answers0