Iterate over a list and remove their elements multiple times

Question

I have the following script in Python, each 30 seconds is checking all the tasks in a list, when a task is completed (FAILED, CANCEL or COMPLETED), the task is removed from the original task and then will track again the other tasks. The code is working good in this way.

The problem is that I'm not sure if each iteration of the while loop is creating a new list, because the entire process could take several hours, so I don't want to create unnecessary data in memory.

def download():
    while tasks:

        for task in tasks[:]:
            success = 0
            file_name = task[1]
            task_index = tasks.index(task)
            task_state = ee.data.getTaskStatus(task[0])[0]['state']

            print(task, task_state)

            if  task_state == FAILED:
                tasks.remove(tasks[task_index])

            elif task_state in [CANCEL_REQUESTED, CANCELLED]:
                tasks.remove(tasks[task_index])

            elif task_state == COMPLETED:
                tasks.remove(tasks[task_index])
                success = 1

            if success == 1:

                do_something()

        if tasks:
            time.sleep(30)
        else:
            print('Done!')
download()

The original tasks come from a file, previous of that I read the file and store the tasks with the proper format in a list. — dguerrero, Apr 11 '20 at 09:54
"I'm not sure if each iteration creates a new list" – No, there are no new lists created in this code. Except you mean `tasks[:]` to which there are no references and it will be garbage collected. — mkrieger1, Apr 11 '20 at 09:57
@muyustan no, using `tasks[:]` is correct, removing items from a list while iterating over it would lead to undesired results. — mkrieger1, Apr 11 '20 at 09:59
It seems to me as if this would be better asked on [Code Review](https://codereview.stackexchange.com), unless you make this question here more specific. I don’t quite understand what you want to know. — mkrieger1, Apr 11 '20 at 10:03
@muyustan, this is not redundant, the copy of the list is important because if not the current loop would be changing the original list, and the indexes in the current loop would be wrong. — dguerrero, Apr 11 '20 at 10:04
@mkrieger1 what I actually want to know if for each loop, the `tasks[:]` is storing in memory a new list, I'm not so sure about that, because when I run the code for long time, the seems to be higher. — dguerrero, Apr 11 '20 at 10:08

score 1 · Answer 1 · answered Apr 11 '20 at 10:06

There is no new list created in your code. But you might want to shorten it a bit:

def download():
while tasks:

    for task in tasks[:]:
        success = 0
        file_name = task[1]
        task_index = tasks.index(task)
        task_state = ee.data.getTaskStatus(task[0])[0]['state']

        print(task, task_state)

        if task_state in [CANCEL_REQUESTED, CANCELLED, FAILED, COMPLETED]:
            tasks.remove(tasks[task_index])
        if task_state == COMPLETED:
            do_something()

    if tasks:
        time.sleep(30)
    else:
        print('Done!')
download()

I think this code is good to go ;)

kederrac · Accepted Answer · 2020-04-11T12:45:50.940

1

the code that you show is not creating another list, also you could improve a bit:

to_remove_states = {CANCEL_REQUESTED, CANCELLED, FAILED, COMPLETED}

def my_filter(taks):
    state = ee.data.getTaskStatus(task[0])[0]['state']
    if state in to_remove_states:
        if state == COMPLETED:
            do_something()   # should not be dependent on your tasks form download function
        return False
    return True

def download():
    while tasks:
        time.sleep(30)
        tasks  = list(filter(my_filter, tasks))
    print('Done!')

download()

edited Apr 11 '20 at 12:45

answered Apr 11 '20 at 10:14

kederrac

16,819
6
32
55

I gave you the answer, because answered my question "there is not a new list". btw your code is working good, but the returns from the filter functions should be in the opposite way, when the task is _ended_ it should be removed from the list. – dguerrero Apr 11 '20 at 10:57

Iterate over a list and remove their elements multiple times

2 Answers2