-1

suppose i have a dictionary such as 

dict = {A:[1,2,3], B:[4,5,6], C:[7,8,9], ......}

I want to process each element of a particular key's list one after other but the individual keys could be processed in parallel using concurrent.futures.ThreadPoolExecutor

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as ex:
  ex.submit(process_element, contents of A)
  ex.submit(process_element, contents of B)
  ex.submit(process_element, contents of C)
  .
  .
  .

So the output should be 

result of process_element A[0]
result of process_element B[0]
result of process_element C[0]

.
.
.

result of process_element A[1]

but not necessary in that order

what is wrong with the above method ?

omkar joglekar
  • 35
  • 7

1 Answers1

2

It doesn't look like you're passing the list, but instead a variable named 'A'. I'm not sure when you run your code why your not getting an error. What you want is below

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as ex:
  ex.submit(process_element, dict['A'])
  ex.submit(process_element, dict['B'])
  ex.submit(process_element, dict['C'])

To process each element one by one, then you could simply use a list construction:

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as ex:
  procsA = [ex.submit(process_element, elem) for elem in dict['A']
  ...
  ...
Richard
  • 3,024
  • 2
  • 17
  • 40
  • if i use this all the elements of dict['A'] are passed into the funtion process_element at once.. i want to pass them one at a time – omkar joglekar Dec 10 '19 at 09:29
  • Do you need them to specifically be in order 0, 1, 2 etc? – Richard Dec 10 '19 at 19:44
  • So the elements of A,B,C can be processed concurrently but no two elements in A or B or C can be processed at the same time. The order is not important – omkar joglekar Dec 10 '19 at 19:49
  • Process_elements is a function that has a variable delay depending on the response of the API depending on the elements in A, B, C – omkar joglekar Dec 10 '19 at 19:50