11

I am writing a program in which an operation takes a few seconds (loading a large file). Luckily, it always takes the same amount of time. So, for the sake of the user, I want to make a progress bar. However, tqdm seems to be designed for loops. Let's say that sleep(10) is the process that takes time. How do I work this out?

I want to make a progress bar for a long process that is not a loop.

from time import time, sleep
from tqdm import tqdm

for i in tqdm([1]):
    sleep(10)

The problem with this code is that the progress bar will stay at zero, then jump to 100% at the end of the process. I want a progress bar that evolves consistently over the course of 10 seconds.

Nicolas Gervais
  • 33,817
  • 13
  • 115
  • 143
  • but you already have a loop there. I cant see the problem, could you develop a little bit more? – adam Feb 20 '20 at 13:42
  • With `tqdm`, a loop with one operation like this will wait at zero, then jump at 100% at the end of the process. I want a progress bar that develops progressively over the course of 10 seconds. – Nicolas Gervais Feb 20 '20 at 13:44
  • 1
    I think tqdm is meant for long loops, not short loops that takes a lot of time. That is because tqdm estimates the ETA based on the average time it took a cycle to complete, so it wont be that useful. – adam Feb 20 '20 at 13:48
  • There is a way to manually [create and update the progress bar](https://pypi.org/project/tqdm/#manual), however you would have to configure the updates directly inside your process and I don't know how accurate the progress estimate would be. – A Co Feb 20 '20 at 13:50
  • Related: [Is it possible to have a progress bar without a loop?](https://stackoverflow.com/q/47500953/7851470) – Georgy Feb 20 '20 at 13:51

4 Answers4

8

Manually set the progress value for tqdm:

#Manually set p_bar
p_bar = tqdm(range(10))
p_bar.update(5)
p_bar.refresh()
time.sleep(1)
#another way to do it
p_bar.n = 7
p_bar.refresh()
Jeremiah Horstick
  • 111
  • 1
  • 2
4

You can define a chunk size. A bar that lasts 10 seconds assumes you already know the time it takes to load the file...

Try this

from tqdm import tqdm_notebook as tqdm

chunk_size = 1000
total_chunks=nrows/chunk_size

chunks = pd.read_sql_query(query, connection, index_col='index_name',chunksize= chunk_size)
raw_train_data=pd.DataFrame()

with tqdm(total=total_chunks) as pbar:

    for chunk in chunks:
        raw_train_data = pd.concat([raw_train_data, chunk])
        pbar.update(1)
CarlosSR
  • 1,145
  • 1
  • 10
  • 22
3
for _ in tqdm(range(10),desc="waiting..."):
    time.sleep(1)
Jose Antonio Martin H
  • 1,453
  • 1
  • 11
  • 10
  • 3
    Please don't post only code as answer, but also provide an explanation what your code does and how it solves the problem of the question. Answers with an explanation are usually more helpful and of better quality, and are more likely to attract upvotes – Boken Dec 01 '20 at 13:28
2

You can use a thread to load the file, and loop through tqdm(range(10)) until the file is loaded like this:

import logging
import threading
import time
from tqdm import tqdm


def thread_function(name):
    logging.info("Thread %s: starting", name)
    logging.info("Main    : file loading...")
    time.sleep(5)
    logging.info("Thread %s: finishing", name)


if __name__ == "__main__":
    logging.basicConfig(format="%(asctime)s: %(message)s", level=logging.INFO,
                        datefmt="%H:%M:%S")

    x = threading.Thread(target=thread_function, args=(1,))
    x.start()
    for i in tqdm(range(10)):
        if not x.is_alive():
            break
        time.sleep(1)
    x.join()
    logging.info("Main    : end.")
DavidBu
  • 478
  • 4
  • 6
  • That's fantastic. Could you perhaps guide me on how to control the width of that? It extends beyond 1 line – Nicolas Gervais Feb 20 '20 at 14:29
  • You just need to remove any logging.info or print() call. – DavidBu Feb 20 '20 at 16:38
  • The "problem" here is that the `n` in `range(n)` has to be large enough to accommodate the run time of `thread_function` in order to show progress. But, then, the bar will always stop early and never reach 100%. If it's the other way around, the bar will reach 100% but then just sit there and not show progress anymore. Neither is great, but I guess there's no way around it as we do not know (by design) how long it takes to run the function. – NichtJens Apr 28 '21 at 10:47