How to apply multiprocessing in Python for event listener and function?

Question

So basically I have set up an event listener that is constantly checking Microsoft Outlook to see if any new emails have come in to my inbox. Based on the emails coming in, I run certain processes to check whether the specific email is the one we are looking for (which does not take too much time), but if the email fits our criteria, we run a certain function on it that takes about a minute or two to complete.

My problem is, say I am getting an email at the same exact time (auto-generated), with criteria that I need and I want to run certain functions on it. When my program picks up on the first email, it goes into the process one that email and it takes about a minute or so to complete, at which point the second email had already come in, and my event listener has missed it. This is a problem because the second email is also important for me to capture and run my process on.

I assume that multiprocessing it the way to tackle this situation - how do I go about setting up a multiprocessing structure, for which my event listener does not stop running while the functions - to check if the email is valid, perform a process on the email - can proceed while the listener is still moving and capturing the next email, and then running the process on it again.

Any help or tips would be greatly appreciated. I am building a metrics/analytics database, for which many reports come in to my inbox, and I automate the reporting process.

Thank you!

import win32com.client
import pythoncom
import time
import os

class Handler_Class(object):
    def OnNewMailEx(self, receivedItemsIDs):
        # RecrivedItemIDs is a collection of mail IDs separated by a ",".
        # You know, sometimes more than 1 mail is received at the same moment.

        for ID in receivedItemsIDs.split(","):
            print('')
            print('Running scan...')
            mail = outlook.Session.GetItemFromID(ID)
            email_date = mail.SentOn.strftime("%m-%d-%Y" + " at " + "%I:%M:%S %p")
            email_date_stamp = mail.SentOn.strftime('%m-%d-%Y_at_%I-%M-%S-%p')
            email_message = mail.Body
            email_subject = mail.Subject
            email_sender = mail.SenderEmailAddress
            email_attachments = mail.Attachments

            print('From: ' + email_sender)
            print('Subject: ' + email_subject)
            print('Date: ' + email_date)

            try:
                if check_correct_subject(email_subject) == True:

                        if email_attachments.Count > 0:
                            print(str(email_attachments.Count) + ' attachments found.')
                            for i in range(email_attachments.Count):
                                email_attachment = email_attachments.Item(i + 1)

                                report_name = email_date_stamp + '_' + email_attachment.FileName
                                print(report_name)
                                print('Pushing attachment - ' + report_name + ' - to check_correct_email() function.')

                                if check_correct_attachment(email_attachment) == True:
                                    save_incoming_report(email_attachment, report_name, get_report_directory(email_subject))

                                else:
                                    print('Not the attachment we are looking for.')
                                    # add error logging here
                                    break

                        else:  # ***********add error logging here**************
                            print('No attachment found.')

            except: #add any error logging here#
                pass


def check_correct_subject(email_subject):


def check_correct_attachment(email_attachment):


def get_report_directory(email_subject):


def save_incoming_report(email_attachment, report_name, directory):


def push_email_attachment(email_attachment, report_name, directory):


def security_risk_report(email_attachment, report_name, directory):


def broker_risk_report(email_attachment, report_name, directory):


def forex_data_report():



def create_html_security_ldw(df1, df2):


def send_outlook_email(html_text):



if __name__ == '__main__':
    outlook = win32com.client.DispatchWithEvents("Outlook.Application", Handler_Class)
    #and then an infinite loop that waits from events.
    pythoncom.PumpMessages()

Problem seems moslty related to the way you catch emails, not the way you treat them after. Show/explain us the code that catch emails. Your event listener shouldn't drop events — Cédric Julien, Aug 03 '17 at 14:31
Just updated the structure and event listener @CédricJulien — sgerbhctim, Aug 03 '17 at 14:36

score 1 · Answer 1 · answered Aug 03 '17 at 14:56

1

You can find great examples of using the multiprocessing module and Queue module from Python to accomplish multithreading here:

Python multiprocessing pool.map for multiple arguments

The comments in that post provide multiple examples/code structures you can adapt to your needs. Also, this post explains why you might use multiprocessing vs threads:

Multiprocessing vs Threading Python

Finally, the official library documentation for multiprocessing here gives you all the info you need:

https://docs.python.org/2/library/multiprocessing.html

answered Aug 03 '17 at 14:56

amlaanb

11
3

I was looking for something a little more related to my exact situation. Any chance you can help me? – sgerbhctim Aug 03 '17 at 15:07
Well, you can visualize it as: **import threading def funcToRun(x, y): # event listener code # if any e-mail comes in: t = threading.Thread(target=funcToRun(), args=(x, y)) t.start() # starts the thread without interrupting main fn** but this is really really basic. A more specific example resembling your problem can be found here: http://www.bogotobogo.com/python/Multithread/python_multithreading_Event_Objects_between_Threads.php Hope that helps! – amlaanb Aug 03 '17 at 15:31
Do you think my problem is meant for multiprocessing or multithreading? – sgerbhctim Aug 03 '17 at 15:33
Multiprocessing and multithreading do the same thing EXCEPT the former uses diff processes while the latter uses diff threads. I hope you know the diff between using the two. In your scenario, I'm sure multithreading makes more sense. But there aren't any restrictions. – amlaanb Aug 03 '17 at 15:48

score 0 · Answer 2 · answered Jan 25 '18 at 13:41

how do I go about setting up a multiprocessing structure, for which my event listener does not stop running while the functions

I think a sensible approach here would be to use BaseManager, which would act as a server to which your event listener would send Outlook messages to. Then all of your other scripts would connect to that server and retrieve any messages that need to be processed. How to do it is neatly explained here.

How to apply multiprocessing in Python for event listener and function?

2 Answers2