0

I'm trying to use pool.starmap() to make many concurrent post requests and URL retrievals. The function that I've built, let's call it "download", takes 4 arguments, 1 of them being a JSON payload. This function downloads a file and returns the path to that file.

Right now I have those arguments as a list of tuples, but pool.starmap() isn't playing nice with the JSON payload, which is just a dictionary. it seems to try to pass in items of that dictionary as arguments, so in my case raises this exception:

"download() takes 4 positional arguments but 27 were given"


dataList = [(downloadFolder, jsonPayload, arg3, arg4), (downloadFolder, jsonPayload, arg3, arg4)]

if __name__ == "__main__":
    with Pool(3) as pool:
        filePaths = [pool.starmap(download, i) for i in dataList]

Does anybody know away of passing in a dictionary as one argument of many? I've already thought about changing my request to requests.post(URL, data = x) instead of requests.post(URL, json = x) but this function is used in a lot of other places already so I'd rather not do that.

itsrainingschmen
  • 1
  • 2

1 Answers1

0

You don't need the list comprehension. starmap does that iteration for you.

filePaths = pool.starmap(download, dataList)

Your code is unpacking the dictionaries into separate arguments to download.

Barmar
  • 741,623
  • 53
  • 500
  • 612