Python async iterator (httpx) and tqdm.asyncio (download progress bar)

Question

I am using httpx as AsyncClient() (called http) and want to display the progress of a download.

async with self.http.stream(method='GET', url=download_url) as res:

                file_out = open(file_path, 'wb')

                async for chunk in tqdm.asyncio.tqdm(iterable=res.aiter_bytes(), 
                                   desc=name, unit='iB',unit_scale=True, unit_divisor=1024, total=size):
                    file_out.write(chunk)
                
                file_out.close()

The download works fine and the progress bar does show some progress but it is not related to the scale provided.

Result:

test.mov:   0%|                                     | 169/2.52M [00:07<32:42:46, 21.4iB/s]

The correct total size is displayed but apparently the unit differs.

If using a specific chunk size, the progress is also not correctly displayed:

async with self.http.stream(method='GET', url=download_url) as res:

                file_out = open(file_path, 'wb')

                async for chunk in tqdm.asyncio.tqdm(iterable=res.aiter_bytes(chunksize), 
                                   desc=name, unit='iB',unit_scale=True, unit_divisor=1024, total=size):
                    file_out.write(chunk)
                
                file_out.close()

The progress bar will then iterate over the chunks (chunk count) but the scale set for bytes does not work, e.g. for a 10 MB file:

test.mov:   0%|                                 | 2.00/10.0M [00:35<51795:37:19, 17.8s/iB

The closest result to a byte stream is omitting the chunk size but the unit is off.

Any idea on how to display the correct progress?

Thanks!

Answer 1

Solved with passing 1 as unit (byte):

async with self.http.stream(method='GET', url=download_url) as res:
    file_out = open(file_path, 'wb')
    async for chunk in tqdm.asyncio.tqdm(iterable=res.aiter_bytes(1), 
            desc=name, unit='iB',unit_scale=True, unit_divisor=1024, total=size):
        file_out.write(chunk)
    file_out.close()