Got great help on concurrency here and actually got it working, or so I thought: How to fix multithreading/multiprocessing with dictionaries? ...got pulled into some other tasks and just now had a chance to come back to this. It works great at 50,000, 70,000 and 71,000 iterations but then for some reason it gives a 'NoneType' error, right around 72,000??? Everytime, and the information does seem to be there... it worked without concurrency it just took hours.
pool = ThreadPoolExecutor(max_workers = 25)
results = list(tqdm(pool.map(get_ips_from_sysinfo, urls), total=len(urls)))
and then a little of the get_ips_from_sysinfo
def get_ips_from_sysinfo(urls):
sysinfo = hx_request(urls)
ip_dict =[]
sysinfo = sysinfo["data"]
hostname = sysinfo.get("hostname")
print(hostname)
network_array = sysinfo.get("networkArray", {})
network_info = network_array.get("networkInfo", [])
Works at 69,000 70,000 71,000 I don't understand why it works up until around 72,000???
Traceback (most recent call last):
File "RiskReportV2_concurrent.py", line 282, in <module>
concurrency(appliance)
File "RiskReportV2_concurrent.py", line 267, in concurrency
results = list(tqdm(pool.map(get_ips_from_sysinfo, urls), total=len(urls)))
File "/mnt/c/Users/money/Documents/riskreport/src/tqdm/tqdm/_tqdm.py", line 1022, in __iter__
for obj in iterable:
File "/usr/lib/python3.5/concurrent/futures/_base.py", line 556, in result_iterator
yield future.result()
File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
return self.__get_result()
File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
raise self._exception
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "RiskReportV2_concurrent.py", line 135, in get_ips_from_sysinfo
sysinfo = sysinfo["data"]
TypeError: 'NoneType' object is not subscriptable
Finally had a chance to look at this again and I have no idea if I did this right or not but switching the function around to this, works
def get_ips_from_sysinfo(urls):
sysinfo = hx_request(urls)
ip_dict =[]
if not sysinfo:
print("None")
else:
sysinfo = sysinfo["data"]
...hope it helps someone!
