0

I am executing a function using concurrent.futures.ThreadPoolExecutor() where each thread loads a module from site-packages. Here is how the code goes,

def execFn(r):
    try:
        module = sys.modules['xxx.abc']
    except KeyError:
        module = import_module('xxx.abc') #this will find and load values in sys.modules
    ....

result = []
rlist = [1,2,3,4,5]
with concurrent.futures.ThreadPoolExecutor(10) as executor:
    futureList = {executor.submit(execFn, r):r for r in rlist}
    for future in concurrent.futures.as_completed(futureList):
            result.append(future.result())

While one of the two thread is able to load the module named xxx.abc present in site-packages, the other is not able to. Is it because one thread locks the module? How can I resolve this issue?

Mahalakshmi Suresh Kumar
  • 35
  • 8
  • What exactly happens? – Michael Butscher Dec 21 '17 at 14:05
  • `sys.modules['xxx.abc']` will never exist, only `sys.modules['xxx']` will after a successful import. The submodule can be accessed as `sys.modules['xxx'].abc` – mata Dec 21 '17 at 14:11
  • @MichaelButscher One thread successfully returns the module(checked with dir(module) and all its methods were present) while the other thread does not have. – Mahalakshmi Suresh Kumar Dec 21 '17 at 14:30
  • @mata I am basically using Arista's pyeapi and trying to load one of its module while is present like this - pyeapi->api->bgp.py . So the module name is pyeapi.api.bgp – Mahalakshmi Suresh Kumar Dec 21 '17 at 14:31
  • Why not just use `import_module` in any case. Thy python import machinery should do the right thing. Otherwise you might get the base module at a point where not all submodules have been imported. Or use a lock around the importing code. – mata Dec 21 '17 at 14:37

0 Answers0