Running lots of processes modifying the same list with Python Multiprocessing results in OSError: Too many open files

Question

I can't get the following code to execute successfully, always ending up with the following error:

OSError: [Errno 24] Too many open files
with the most recent call on p.start()

As you can see I already tried to execute the processes in chunks of 500, as they run fine when only 500 are executed, however after the second loop I still receive the above error. I guess that the processes are not properly closed after execution, however I could not figure out how to check and properly close them...

here is my code:

import multiprocessing

manager = multiprocessing.Manager()
manager_lists = manager.list()


def multiprocessing_lists(a_list):
    thread_lists = []
    for r in range(400):
        thread_lists.append(a_list)
    manager_lists.extend(thread_lists)


manager = multiprocessing.Manager()
manager_lists = manager.list()
processes = []
counter = 0
chunk_size = 500
all_processes = 4000


for i in range(0, all_processes, chunk_size):

    for j in range(chunk_size):
        a_list = [1,2,3,4,5,6,7,8,9]
        p = multiprocessing.Process(target=multiprocessing_lists, args=(a_list,))
        processes.append(p)
        p.start()

        counter += 1
    for process in processes:
        process.join()
        process.terminate()


print(len(manager_lists))