I'm using Python's Watchdog to monitor a given directory for new files being created. When a file is created, some code runs that spawns a subprocess shell command to run different code to process this file. This should run for every new file that is created. I've tested this out when one file is created, and things work great, but am having trouble getting it working when multiple files are created, either at the same time, or one after another.
My current problem is this... the processing code run in the shell takes a while to run and will not finish before a new file is created in the directory. There's nothing I can do about that. While this code is running, watchdog will not recognize that a new file has been created, and will not proceed with the code.
So I think I need to spawn a new process for each new file, or do something get things to run concurrently, and not wait until one file is done before processing the next one.
So my questions are:
1.) In reality I will have 4 files, in different series, created at the same time, in one directory. What's the best way get watchdog to run the code on file creation for all 4 files at once?
2.) When the code is running for one file, how do I get watchdog to begin processing the next file in the same series without waiting until processing for the previous file has completed. This is necessary because the files are particular and I need to pause the processing of one file until another file is finished, but the order in which they are created may vary.
Do I need to combine my watchdog with multiprocessing or threading somehow? Or do I need to implement multiple observers? I'm kind of at a loss. Thanks for any help.
class MonitorFiles(FileSystemEventHandler):
'''Sub-class of watchdog event handler'''
def __init__(self, config=None, log=None):
self.log = log
self.config = config
def on_created(self, event):
file = os.path.basename(event.src_path)
self.log.info('Created file {0}'.format(event.src_path))
dosWatch.go(event.src_path, self.config, self.log)
def on_modified(self, event):
file = os.path.basename(event.src_path)
ext = os.path.splitext(file)[1]
if ext == '.fits':
self.log.warning('Modifying a FITS file is not allowed')
return
def on_deleted(self, event):
self.log.critical('Nothing should ever be deleted from here!')
return
Main Monitoring
def monitor(config, log):
'''Uses the Watchdog package to monitor the data directory for new files.
See the MonitorFiles class in dosClasses for actual monitoring code'''
event_handler = dosclass.MonitorFiles(config, log)
# add logging the the event handler
log_handler = LoggingEventHandler()
# set up observer
observer = Observer()
observer.schedule(event_handler, path=config.fitsDir, recursive=False)
observer.schedule(log_handler, config.fitsDir, recursive=False)
observer.start()
log.info('Begin MaNGA DOS!')
log.info('Start watching directory {0} for new files ...'.format(config.fitsDir))
# monitor
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.unschedule_all()
observer.stop()
log.info('Stop watching directory ...')
log.info('End MaNGA DOS!')
log.info('--------------------------')
log.info('')
observer.join()
In the above, my monitor method sets up watchdog to monitor the main directory. The MonitorFiles class defines what happens when a file is created. It basically calls this dosWatch.go method which eventually calls a subprocess.Popen to run a shell command.
