4

I have a scenario where I need to upload a series of files to a web server sequentially, with chunking involved for each file. The upload process is quite simple when implemented procedurally, and looks approximately like:

foreach file in files:
  const result = await createMultipartUpload();

  do {
    const chunk = takeChunk();
    const result = await uploadChunk(chunk);

    chunksRemain = doChunksRemain();
  } while (chunksRemain);

  await completeUpload();

That is, for each sequential upload process, a single creation request occurs, multiple chunks are uploaded sequentially—but I don't know how many chunks will be uploaded—and then a single completion request occurs.

I need to convert this into reactive code with RxJs. What I have actually works for single-chunk uploads, where only one chunk is needed. But I haven't figured out how dynamically generate the upload chunk observables and push them onto some kind of array that they can then be retrieved from.

Here's what I have so far:

  // Contains single request. 
  public createMultipartUpload(file: File): Observable<File> {
    return this.filesHttp.store(file);
  }

  // Contains two HTTP requests. Generates a presigned URL, then uploads the chunk with the presaged URL.
  public uploadChunk(file: File): Observable<File> {
    let chunk: Blob;

    // Determine the bytes of the file to take and assign to chunk.
    if (file.originalFile.size < this.chunkSize) {
      chunk = file.originalFile;
    } else {
      chunk = file.originalFile.slice(file.currentByteOffset, file.currentByteOffset + this.chunkSize);
    }

    return this.filesHttp.getPresignedUrlForUpload(file)
      .pipe(
        concatMap(res => {
          return this.filesHttp.upload(res.uri, chunk);
        }),
        map(f => {
          // Set next chunk to take
          f.currentByteOffset += this.chunkSize;
          return f;
        })
      );
  }

  public completeUpload(file: File): Observable<MultipartCompletionResponse> {
    return this.filesHttp.complete(file);
  }

  from(this.files)
    .pipe(
      concatMap(f => this.createMultipartUpload(f)),
      switchMap(f => this.uploadChunk(f)),
      concatMap(f => this.completeUpload(f))
    ).subscribe(output => {
      // done?
    });

How can I sequentially upload all chunks, and not just the first one, given this scenario?

marked-down
  • 9,958
  • 22
  • 87
  • 150

1 Answers1

3

Use expand to process a dynamic amount of Observables. expand recursively maps to an Observable and receives its output as the next input. End this recursion by returning EMPTY.

// Upload mutiple files one after another with concat
concat(...this.files.map(file => uploadFile(file))).subscribe(console.log);

// Upload single file in multiple chunks with expand
uploadFile(file): Observable<any> {
  return createMultipartUpload(file).pipe(   
    // add this for a do...while logic where doChunksRemain() shouldn't be called for 
    // the first uploadChunk()
    //switchMap(f => uploadChunk(f)),

    // Upload next chunks until doChunksRemain() returns false
    expand(f => doChunksRemain() ? uploadChunk(f) : EMPTY),
    // only take the result of the last uploaded chunk
    last(),
    // map to completeUpload when we receive the last result
    switchMap(f => completeUpload())
  );
}

https://stackblitz.com/edit/rxjs-d5vpyf

frido
  • 13,065
  • 5
  • 42
  • 56