How can i use multithreading (or multiproccessing?) for faster data upload?

Question

I have a list of issues (jira issues):

listOfKeys = [id1,id2,id3,id4,id5...id30000]

I want to get worklogs of this issues, for this I used jira-python library and this code:

listOfWorklogs=pd.DataFrame()                 (I used pandas (pd) lib)
lst={}                                       #dictionary for help, where the worklogs will be stored
for i in range(len(listOfKeys)):
    worklogs=jira.worklogs(listOfKeys[i])    #getting list of worklogs
    if(len(worklogs)) == 0:
        i+=1
    else:
        for j in range(len(worklogs)):
            lst = {
                    'self': worklogs[j].self,  
                    'author': worklogs[j].author,
                    'started': worklogs[j].started,
                    'created': worklogs[j].created,
                    'updated': worklogs[j].updated,
                    'timespent': worklogs[j].timeSpentSeconds
                }
            listOfWorklogs = listOfWorklogs.append(lst, ignore_index=True)
########### Below there is the recording to the .xlsx file ################

so I simply go into the worklog of each issue in a simple loop, which is equivalent to referring to the link: https://jira.mycompany.com/rest/api/2/issue/issueid/worklogs and retrieving information from this link

The problem is that there are more than 30,000 such issues. and the loop is sooo slow (approximately 3 sec for 1 issue) Can I somehow start multiple loops / processes / threads in parallel to speed up the process of getting worklogs (maybe without jira-python library)?

Answer 1

I recycled a piece of code I made into your code, I hope it helps:

from multiprocessing import Manager, Process, cpu_count

def insert_into_list(worklog, queue):
    lst = {
        'self': worklog.self,  
        'author': worklog.author,
        'started': worklog.started,
        'created': worklog.created,
        'updated': worklog.updated,
        'timespent': worklog.timeSpentSeconds
    }
    queue.put(lst)
    return

# Number of cpus in the pc
num_cpus = cpu_count()
index = 0

# Manager and queue to hold the results
manager = Manager()

# The queue has controlled insertion, so processes don't step on each other
queue = manager.Queue()

listOfWorklogs=pd.DataFrame()
lst={}                                       
for i in range(len(listOfKeys)):
    worklogs=jira.worklogs(listOfKeys[i])    #getting list of worklogs
if(len(worklogs)) == 0:
    i+=1
else:

    # This loop replaces your "for j in range(len(worklogs))" loop
    while index < len(worklogs):
        processes = []
        elements = min(num_cpus, len(worklogs) - index)

        # Create a process for each cpu
        for i in range(elements):
            process = Process(target=insert_into_list, args=(worklogs[i+index], queue))
            processes.append(process)

        # Run the processes
        for i in range(elements):
            processes[i].start()

        # Wait for them to finish
        for i in range(elements):
            processes[i].join(timeout=10)

        index += num_cpus

    # Dump the queue into the dataframe
    while queue.qsize() != 0:
        listOfWorklogs.append(q.get(), ignore_index=True)

This should work and reduce the time by a factor of little less than the number of CPUs in your machine. You can try and change that number manually for better performance. In any case I find it very strange that it takes about 3 seconds per operation.

PS: I couldn't try the code because I have no examples, it probably has some bugs

Answer 2

I have some troubles((

1) indents in the code where the first "for" loop appears and the first "if" instruction begins (this instruction and everything below should be included in the loop, right?)

for i in range(len(listOfKeys)-99):
    worklogs=jira.worklogs(listOfKeys[i])    #getting list of worklogs
    if(len(worklogs)) == 0:
    ....

2) cmd, conda prompt and Spyder did not allow your code to work for a reason: Python Multiprocessing error: AttributeError: module '__ main__' has no attribute 'spec' After researching in the google, I had to set a bit higher in the code: spec = None (but I'm not sure if this is correct) and this error disappeared. By the way, the code in Jupyter Notebook worked without this error, but listOfWorklogs is empty and this is not right.

3) when I corrected indents and set __spec __ = None, a new error occurred in this place: processes[i].start () error like this: "PicklingError: Can't pickle : attribute lookup PropertyHolder on jira.resources failed"

if I remove the parentheses from the start and join methods, the code will work, but I will not have any entries in the listOfWorklogs(((

I ask again for your help!)

Answer 3

How about thinking about it not from a technical standpoint but a logical one? You know your code works, but at a rate of 3sec per 1 issue which means it would take 25 hours to complete. If you have the ability to split up the # of Jira issues that are passed into the script (maybe use date or issue key, etc) you could create multiple different .py files with basically the same code, you would just be passing each one a different list of Jira tickets. So you could just run say 4 of them at the same time and you would reduce your time to 6.25 hours each.