event driven pattern for writing chunks to a file - JS

Question

I'm trying to transfer a file over WebRTC, and I'm struggling to figure out a good pattern for writing data as it's coming in. Since file chunks will be coming in at an unknown rate, I need to be able to write each chunk as it becomes available; this means two things:

1. if data is coming in too fast, then we need to queue each chunk to be written later after the current write task is finished
2. if data is coming in too slow, then we need to wait for a chunk to become available

Ideally, I'd like to avoid having to rely on setTimeout() for waiting on chunks to arrive. What I have so far is close, but not exactly what I'm looking for:

// container to hold chunks as they come in
var chunkCollection = {};

// callback function for RTCDataChannel messages
function onData(data) {

  chunkCollection[data.chunkIndex] = data.chunk;
  writeToFile(data.chunkIndex);
}

function writeToFile(chunkIndexToWrite) {

  // if we have the next chunk, write it to the file
  if (chunkCollection[chunkIndexToWrite]) {
    var chunk = chunkCollection[chunkIndexToWrite];
    delete chunkCollection[chunkIndexToWrite];
    fileWriter.seek(chunk.offset);
    fileWriter.write(chunk.blob);
  }
  else {
    // we don't have the next chunk, so we have to wait
    setTimeout(function() {
      writeToFile(chunkIndexToWrite);
    }, 100);
  }
}

The problem with this is that if chunks come in too fast, fileWriter will not be ready to write the next chunk and will throw an exception. However, if chunks come in too slowly, setting the correct timeout will be extremely tricky.

It seems like an event-driven approach will work best here. One event is the data message coming in from RTCDataChannel. Another event is that fileWriter has finished writing, and is ready to write the next chunk. But the piece that I'm getting stuck on is how to properly wait for a chunk to come in...

If the browser is not busy writing to the file, and is just sitting waiting for a chunk to come in, then the browser should start writing the chunk as soon as the chunk is available. But I can't figure out how to do this without an ugly setTimeout loop; I can't figure out how to publish an event that says we don't need to wait anymore, and can continue writing.

What's a good pattern for doing this?

Answer 1

There is a way to do that without timeout, although it uses flag to indicate that writer is available, it still should be better than timeout loop.

I would create queue where data chunks will be stored when it arrives and use async fileWriter (one that does callback when it finishes writing to file).

Like that:

var readyToWrite = true;
var chunks = [];

var onData = function(data) {
    chunks.push(data);
    if (readyToWrite) {
         writeChunksToFile(chunks);
    }
};

var writeChunksToFile = function() {
    if (chunks) {
        readyToWrite = false;
        fileWriter.write(chunks.shift(), function() {
            writeChunksToFile(chunks);
        });
    } else {
        readyToWrite = true;
    }
};

That way writer is writing until chunks will become empty, then it notifies data receiver (using flag) that it's available, so data receiver (onData) knows that it has to call it again when next chunk is received. If data receiver gets data and writer is still writing to file, then it won't call it, it will just push chunk to our queue array, and writer will get that chunk when it finishes his writing, because it's called recursively.

This is the simplest code we could wrote with that approach, but you can make it "more OOP" with ObserverPattern or "more functional" using Monad.