How to make BackgroundTasks manager?

Question

I have FastAPI endpoint to upload csv files to mongodb:

Router endpoint:

@UploadRouter.post(
    "/",
    response_model=CsvUploadResponse,
    status_code=status.HTTP_201_CREATED
)
async def upload(
    bg:BackgroundTasks,
    file : UploadFile = File(...),
    uploadService : UploadService = Depends()
):
    return uploadService.add_to_queue(file=file,bg=bg)

Service methods:

def add_to_queue(self, file: File, bg):
    content_type = file.content_type
    if content_type != "text/csv":
        raise raise_with_log(status_code=415,detail='Only csv files are supported!')
    name = uuid4().hex
    self.statusRepository.create(name)
    bg.add_task(self.upload,file,name)
    return CsvUploadResponse(status=True, msg='Csv is in uploading queue now', id=name)


async def upload(self,file,name):
    t1=time.time()
    self.statusRepository.update(name,'Uploading')
    temp_file = BytesIO()
    while chunk:=await file.read(10*1024*1024):
        temp_file.write(chunk)
    try:
        csv=polars.read_csv(temp_file).to_dict()
    except polars.ComputeError:
        self.statusRepository.update(name,'Failed')
        return
    csv_data = (dict(zip(csv.keys(), row)) for row in zip(*csv.values()))
    await self.csvRepository.create(csv_data,name)
    self.statusRepository.update(name,'Uploaded')
    t2=time.time()
    print('ALL TIME IS: ',t2-t1)

But, unfortunately, every request i do, it appears new object of BackgroundTasks and tasks are completing parallel, so it`s make motor driver to upload csvs in parallel. I don't know what to do exactly to make requests in queue and not to block endpoints!!!

How to make BackgroundTasks manager?

0 Answers0