ConcatMap with CombineLatest creating more calls than expected

Question

I have an ngrx action/effect that takes a FileList and uploads them sequentially to the server. After the first File is uploaded, an ID will be generated that needs to be used by the second File as argument, so it can be later related to the first one in the back. The way I setup the component-store is that I have a State that holds the ID (initially null and the files).

The problem arises when I try to update the ID in the store after the first file is uploaded. The concatMap that I am using to keep the sequential order is still working on the files and its only after all the files are actually uploaded that a new stream comes along the way with the ID that I set after the first call. So, at the end I have more https calls than expected and none of the files are using the ID that was created initially.

The real problem is that I don't know how to get the latest value of the ID from the store at the moment that the concatMap is shooting the files. I am guessing that combineLatest is not in the right place.

Here's what I have tried so far:

readonly uploadFiles = this.effect((files: Observable<FileList>) => {
    return combineLatest([
        files.pipe(
            mergeMap(list => Array.from(list).map(i => ({file: i, length: list.length})) )
        ),
        //selector holding the ID that will be set after the first file is uploaded
        this.uploadId$,
    ]).pipe(
        concatMap(([tuple, uploadId], index) => 
            this._dataSvc.upload(tuple.file, index, tuple.length, uploadId.ID).pipe(
                tap({
                    next: (event) => {
                        switch (event.type) {
                          case HttpEventType.UploadProgress:
                            this.uploadProgress([
                                Math.round((event.loaded / event.total) * 100),
                                index
                            ]);
                            break;
                          case HttpEventType.Response:
                            if(uploadId.ID === null) {
                              //calls the reducer with the ID of the current set of uploads
                              this.setUploadId(event.body.ID);
                            }
                        }
                    },
                }),
                catchError(() => EMPTY) //for now
            )
        )
    )
  });

Thanks

Here are my changes after @maxime1992 suggestion:

readonly uploadFiles = this.effect((files$: Observable<FileList>) => {
    return files$.pipe(
        concatMap((files) =>
            of({
                ID: null,
            }).pipe(
                expand((ids, index) =>{
                    const idx = index/5;
                    return files.length === idx || index % 5 !== 0 ? EMPTY :
                    this._dataSvc.upload(files[idx], idx, files.length, ids.ID).pipe(
                        tap({
                            next: (event) => {
                                switch (event.type) {
                                  case HttpEventType.UploadProgress:
                                    this.uploadProgress([
                                        Math.round((event.loaded / event.total) * 100),
                                        idx
                                    ]);
                                    break;
                                  case HttpEventType.Response:
                                    this.uploadCompleted(idx);
                                }
                            },
                        }),
                        map((event, index) => 
                            event.type === HttpEventType.Response ? 
                            ({ ID: event.body.ID }) : ({ ID: null })),
                        catchError(() => EMPTY) //for now
                    )   
                })
            )
        )
    );

Answer 1

Assuming I understood your use case correctly, you want to provide for each file upload the previous upload ID. The closest use case for this in general terms is pagination, when you move on you need to know the previous page number or page ID.

For such use case, the most appropriate method is expand, which as the name suggests, let you "expand" and have recursive calls.

In order to get a demo working, I've tweaked the code to work outside of ngrx/effects context and mocked the backend but my code is down to this:

const uploadFiles = (files$: Observable<FileList>) =>
  files$.pipe(
    switchMap((files) =>
      of(null).pipe(
        expand((uploadId, index) =>
          files.length === index
            ? EMPTY
            : _mockDataSvc
                .upload(files[index], index, files.length, uploadId)
                .pipe(map((res) => res.ID))
        )
      )
    )
  );

I've mocked the backend with the following:

let mockResponseId = 0;

// mocking the API call in order to have a working demo
const _mockDataSvc = {
  upload: (file: File, index: number, length: number, uploadId: string) => {
    console.log(
      `[Mocked backend]: Call for uploading a file that is the index ${index}, has length ${length} and with a previous upload ID of ${uploadId}`
    );
    return of({ ID: `mock-response-ID-${mockResponseId++}` }).pipe(delay(500));
  },
};

When I run this, here's the output:

[Mocked backend]: Call for uploading a file that is the index 0, has length 3 and with a previous upload ID of null
[Mocked backend]: Call for uploading a file that is the index 1, has length 3 and with a previous upload ID of mock-response-ID-0
[Mocked backend]: Call for uploading a file that is the index 2, has length 3 and with a previous upload ID of mock-response-ID-1

You can see that the previous ID for the first request is null (and this request returns ID 0, then next request passes the ID received previously and so on.

I think you've got a good base here to adapt that back to your effect.

Here's the full code with a live demo.