Trigger HTTP fileupload by reading a file from another multi part HTTP request

Question

An upload end-point receives data from a client in multipart file. This file is again sent as a parameter to new HTTP call made from the same service.

//Client uploading a file code

form(enctype="multipart/form-data", action="/data/upload", method="post")
  input(type="file", name="data")

I want to read that file and upload that to another Java end-point that accepts the file.

I tried getting the file stream via busBoy and converted it into a string and tried converting the string back into the stream and passed it to the new request. But this fails from Java service.

import axios from 'axios';

const FormData = require('form-data');
const { Readable } = require('stream');

const fileBufferChunks = [];

busBoyStream.on('file', (fieldname, file, filename, encoding, mimetype) => {
  file.on('data', (data) => {
    fileBufferChunks.push(data);
  });
  file.on('end', () => {
    console.log(` ${filename} file parsed from req`);
    const fileString = Buffer.concat(fileBufferChunks).toString('base64');
    const data = new FormData();
    const readableStream = Readable.from([fileString]);
    data.append('file_param', readableStream);
    const headers = data.getHeaders();
    const axiosConfig = {
      method: 'post',
      url: URL,
      headers: {
        ...headers,
      },
      data,
    };

    axios(axiosConfig)
      .then((response) => {
        console.log('job trigger success');
      });
  });
});

But if I store the file in local and pass the same file stream via fs.createReadStream it works.


import axios from 'axios';

const FormData = require('form-data');
const fs = require('fs');

const fileBufferChunks = [];

busBoyStream.on('file', (fieldname: any, file: any, filename: any, encoding: any, mimetype: any) => {
  file.on('end', () => {
    const data = new FormData();
    const saveTo = '/dummypath/name.csv';
    file.pipe(fs.createWriteStream(saveTo));
    data.append('file_param', fs.createReadStream('/dummypath/name.csv'));
    const headers = data.getHeaders();
    const axiosConfig = {
      method: 'post',
      url: URL,
      headers: {
        ...headers,
      },
      data,
    };

    axios(axiosConfig)
      .then((response) => {
        console.log('job trigger success');
      });
  });
});

I want to understand what is the difference between the stream which is created converting the file to string and creating a stream from that file string and a stream created using fs.createReadStream(pathToFile)

Can anyone explain why there is a difference in behavior and what is the recommended way of sending file upload from one HTTP request to another request...?

score 0 · Answer 1 · answered Nov 26 '20 at 14:49

The issue was with the handling file properties with library form-data which converts streams to formdata specific payload.

form-data library auto populates the contentType information of file if the stream is fs generated file stream. So it was working fine with fs file streams. In my case the stream is Readable file stream from stream library so i need to manually provide the contentType information like below example.

data.append(uploadFile.fieldname, uploadFile.file, {
   filename: 'filename.csv', // ... or:
   //   filepath: 
   contentType: 'text/csv',
   // knownLength: 5,
 });

form-data library for handling files requires many file properties like this to be provides for custom stream objects. Full list can be found in their documentation.

Trigger HTTP fileupload by reading a file from another multi part HTTP request

1 Answers1