2

I'm trying to upload a file to S3 using the node-formidable method fileWriteStreamHandler.Before the upload to S3 I want to create a hash of file. This means implementing a stream pipe that first pass the data through a hash then passes that data to the S3 upload.

When trying to implement the pipe I kept running into issues. So below is a simplified function that more or less represents what I want to do.

formHandler.js

const form = formidable({
  encoding: 'utf-8',
  keepExtensions: true,
  allowEmptyFiles: false,
  maxFiles, maxFileSize, maxTotalFileSize,
  maxFields, maxFieldsSize, minFileSize,
  multiples: true,
  fileWriteStreamHandler: streamUploadImage,
});

streamUploadImage.js

function streamUploadImage() {
  const firstStream = new PassThrough();
  const lastStream = new PassThrough();

  const hash = createHash('SHA2-256');
  hash.setEncoding('hex');
  const transform = new Transform({
    transform(chunk, encoding, cb) {
      hash.write(chunk);
      cb();
    },
    flush(cb) {
      hash.end();
      console.log('all done', hash.read());
      cb();
    }
  });

  firstStream.on('data', () => console.log('first'));
  lastStream.on('data', () => console.log('last'));

  return first.pipe(transform).pipe(last);
};

When using the above streamUploadImage only the lastStream is called. firstStream & transform are never called.

Why is that? Is the pipeline not implemented correctly? Does the formidable fileWriteStreamHandler not work with pipes?

using formidable@3.2.1

UPDATE: see below for a quick reproduction of my issue:

var server = http.createServer(function (req, res) {
  if (req.url == '/') {
      res.writeHead(200, { 'Content-Type': 'text/html' });
      res.end(`
        <form action="/upload" enctype="multipart/form-data" method="post">
          <label>file name<input type="text" name="file_name" autofocus /></label><br />
          <label>single file<input type="file" name="file_single" /></label><br />
          <label>multiple files<input type="file" name="filearray_with_multiple[]" multiple /></label><br />
          <br />
          <button>Upload</button>
        </form>
      `);

      res.end();
  } else if (req.url === '/upload') {
    const form = formidable({
      encoding: 'utf-8',
      keepExtensions: true,
      allowEmptyFiles: false,
      multiples: true,
      fileWriteStreamHandler: streamUploadImage,
    });

    form.parse(req, (err, fields, files) => {
      if (err) throw err;

      console.log('parsed file upload');
      console.log({ fields, files });
      res.writeHead(201, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ err, fields, files}, null, 2))
    })
  }
});

function streamUploadImage() {
  const firstStream = new PassThrough();
  const lastStream = new PassThrough();

  const hash = createHash('SHA2-256');
  hash.setEncoding('hex');
  const transform = new Transform({
    transform(chunk, encoding, cb) {
      hash.write(chunk);
      cb();
    },
    flush(cb) {
      hash.end();
      console.log('all done', hash.read());
      cb();
    }
  });

  firstStream.on('data', () => console.log('first'));
  lastStream.on('data', () => console.log('last'));

  return firstStream.pipe(transform).pipe(lastStream);
};

server.listen(5000);
M.Holmes
  • 403
  • 1
  • 7
  • 22
  • You could use [`stream.pipeline()`](https://nodejs.org/docs/latest-v17.x/api/stream.html#streampipelinesource-transforms-destination-callback) – 2pichar Jan 18 '22 at 19:17
  • You shouldn't need to wrap `hash` in a `Transform`. `crypto.Hash` extends `Transform` – 2pichar Jan 18 '22 at 19:32
  • I got the exact opposite. Only `first` was being called, and not `last`. What is your full code? – 2pichar Jan 18 '22 at 19:57
  • @2pichar added some code to reproduce – M.Holmes Jan 18 '22 at 20:12
  • @2pichar i would like to use `stream.pipeline()` but I think this is more of an issue with formidable being able to handle that – M.Holmes Jan 18 '22 at 21:21
  • Does your code actually use `PassThrough`, or is it just for testing? formidable's docs says that `fileWriteStreamHandler` should return a `Writable`. – 2pichar Jan 19 '22 at 00:29
  • I think the `Passthrough` is your problem. When I use `process.stdin` and `process.stdout`, it works correctly and hashes the values – 2pichar Jan 19 '22 at 01:29
  • if you swap out `PassThrough` for some generic `Transform`, I'm still unable to get this to work. Do you mind posting an example of what is working for you? – M.Holmes Jan 19 '22 at 01:38
  • I'm using `process.stdin` for the input and `process.stdout` for the output – 2pichar Jan 19 '22 at 01:41
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/241190/discussion-between-m-holmes-and-2pichar). – M.Holmes Jan 19 '22 at 01:59

1 Answers1

2

stream.pipe() returns the destination stream to allow for chaining.

You need to return the head of the pipeline from streamUploadImage() (firstStream in your example), rather than the tail.

function streamUploadImage() {
  const firstStream = new PassThrough();
  const lastStream = new PassThrough();

  // *snip*

  // Wire up the pipeline
  firstStream.pipe(transform).pipe(lastStream);
  // Return the head of the pipeline
  return firstStream;
};
teppic
  • 7,051
  • 1
  • 29
  • 35