0

I am watching a folder being_watched using the pyinotify module in python. But the files are being created faster than they can be processed by the Event Handler of the pyinotify. Is there a way around this problem, where all the new files created can be lined up or taken up later by the Event Handler?

Code snippet:

class EventHandler(pyinotify.ProcessEvent):
    def process_IN_CREATE(self, event):
        print "Creating:", event.pathname
        #start_time=time.time()
        #reading the data 
        with open(str(event.pathname)) as ofile:
            d = json.loads(ofile.read())
            for _id in d:
                if d[_id]["_comment"]>0:
                    inputdata.update({_id:{"la":d[message_id]["la"],"lo":d[_id]["lo"]}})


        try:
            getcdata(inputdata,outfile)
        except RuntimeError as e:
            logger.error('Caught a Rutime exception in the getcdatadriver . '+str(e.value))

def main():
    logger = logging.getLogger('mainloop')
    logger.setLevel(logging.DEBUG)
    fh = logging.FileHandler('getcdatadriver.log')
    logger.addHandler(fh)

    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    fh.setFormatter(formatter)
    sys.excepthook=Notify(logger,"Exception caught getcdatadriver mainloop").my_excepthook
    try:
        while True:
                watch_dir="./output_watchdir"
                wm = pyinotify.WatchManager()  # Watch Manager
                mask = pyinotify.IN_CREATE  # watched events
                handler = EventHandler()
                notifier = pyinotify.Notifier(wm, handler)
                wdd = wm.add_watch(watch_dir, mask, rec=True)
                notifier.loop()
    except (KeyboardInterrupt, SystemExit):
        print '\n! Received keyboard interrupt, quitting.\n'
        Notify(logger,"getcdatadriver stopped").send_exception_email('Received keyboard interrupt, quitting.')
    except RuntimeError as e:
        print 'Caught a Rutime exception in the getcdatadriver mainloop. '+str(e.value)
        Notify(logger,"Runtime Exception getcdatadriver mainloop").send_exception_email(str(e.value))
a'-
  • 251
  • 5
  • 14
  • Akanksha, please clarify your question. Be more explicit. Typically it helps to add snippets of code and any errors you are receiving. – mmann1123 Nov 18 '15 at 19:06
  • Yeah, inotify can loose data and its own docs suggest you try to clean up inconsistencies. I assume your watch is as narrow as possible. Keep event processing to a minimum.... maybe queue it to another thread for any actual work. – tdelaney Nov 18 '15 at 19:26
  • Haven't received any error. However the following func is called only on few files being created in the watched directory: – a'- Nov 18 '15 at 19:29
  • When I used `pyinotify` several years ago, [http://www.serpentine.com/blog/2008/01/04/why-you-should-not-use-pyinotify/](http://www.serpentine.com/blog/2008/01/04/why-you-should-not-use-pyinotify/) was very helpful. `pyinotify` was massively non-performant and I ended up hacking my own version. I don't know if its still the case today. – tdelaney Nov 18 '15 at 19:46
  • You shouldn't do that level of work in the event handler. Have this code run as a separate thread and queue to a worker. Also, you run the risk that the file has not been fully written and closed in `process_IN_CREATE`. You need to also grab the close events and do the work after they are done. Also, I'm not sure what happens if your event raises an exception (like the file hasn't been written so json fails). Add an exception handler and see if it catches something. – tdelaney Nov 18 '15 at 19:49

0 Answers0