RxJS `repeatWhen` with notifier repeats rapidly

Question

I am trying to make a network retry-er using RxJS repeatWhen operator. The idea is that when a new request receives to the scheduler, then I try the request directly and if it results in a network failure result, I add it to a pool to be retried on sometime later. So the entering point of my scheduler is the queue function which does the job like this:

queue({ operation, body }) {
    const behaviourSubject = new BehaviorSubject();
    const task = {
        operation,
        body,
        behaviourSubject,
    };
    this.doTask(task).subscribe({
        error: _ => this.notifier.next({ tasks: [task], remove: false }),
        complete: () => console.log('Task successfully done on first try: ', task),
    });
    return behaviourSubject;
}

and this.notifier is a Subject which is used as the notifier of the worker. So the worker itself is like this:

    const rawWorker = new Observable(subscriber => {
        const doneTasks = [];
        const jobs = [];
        for (const task of this.tasks) {
            jobs.push(
                this.doTask(task).pipe(
                    tap(_ => {
                        doneTasks.push(task);
                    }),
                    catchError(error => { task.behaviourSubject.next(error); return of(error); }),
                )
            );
        }

        if (jobs.length > 0) {
            forkJoin(...jobs).subscribe(_ => {
                this.notifier.next({ tasks: doneTasks, remove: true });
                subscriber.next(`One cycle of worker done. #${doneTasks.length} task(s) done and #${this.tasks.length} remaining.`);
                // subscriber.complete();
                if (this.tasks.length > 0) {
                    this.notifier.next();
                }
            })
        } else {
            subscriber.complete();
        }
    });
    const scheduledWorker = rawWorker.pipe( // TODO: delay should be added to retry and repeat routines
        retry(),
        repeatWhen(_ => this.notifier.pipe(
            filter(_ => this.tasks.length > 0),
        )),
    );

and also the notifier keeps track of all undone requests in an array like this:

    this.notifierSubscription = this.notifier
        .pipe(
            filter(data => data && data.tasks)
        )
        .subscribe({
            next: ({ tasks = [], remove = false }) => {
                if (remove) {
                    console.log('removing tasks: ', tasks);
                    this.tasks = this.tasks.filter(task => !tasks.some(tsk => task === tsk));
                } else {
                    console.log('inserting: ', tasks);
                    this.tasks.push.apply(
                        this.tasks,
                        tasks,
                    );
                }
                console.log('new tasks array: ', this.tasks);
            }
        });

As I know if a cycle of the worker is not completed then repeatWhen has nothing to do. For example if I remove the part:

    else {
            subscriber.complete();
        }

from the worker, on the first try of the worker (an empty cycle) the Observable does not complete and repeatWhen won't do anything. But on the other hand as you see I have commented // subscriber.complete(); when there exist jobs but the repeat is occurring. And the worst part of the problem is that many different instances of the worker run in parallel which makes many duplicate requests.

I have spent a lot of time on this problem but do not have any clue to trace.