file watcher in python 3.5 using library watchgod

Question

Hi everyone i am trying to build a file watcher in python 3.5 using watchgod. I want to continuously watch a directory and if any file is added then i want to send a list of added files to another program which will perform a series of task. Following is my code in python :-

print("execution of main file begins !!!!")
import os
from watchgod import watch

#changes gives a set object when watch finds any kind of changes in directory
for changes in watch(r'C:\Users\Rajat.Malik\Desktop\Requests'): 

     fileStatus = [obj[0] for obj in list(changes) ]  #converting set to list which gives file status as added, changed or modified
     fileLocation = [obj[1] for obj in list(changes) ]   #similarly getting list of location of files added  
     var2 = 0

     for var1 in fileLocation: 
          if fileStatus[var2] == 1:   #if file is added then passing all files to another code which will work on the list of files added
               os.system('python split_thread_module.py '+var1)  #now this code will start executing 
               var2 = var2 + 1

So the problem i am having is that while split_thread_module.py is executing the watcher is not watching the directory. Any file which is coming at time when split_thread_module.py is executing is not reflecting in changes. How can i watch the changes in directory and pass it to the other program on the fly even when the other program is executing. I am not a python programmer. Can anyone help me in this regard ? Thanks in advance !!!!

SColvin · Answer 1 · 2018-04-06T13:46:29.690

Sorry for delayed, I'm the developer of watchgod. I've added a python-watchgod tag to your question which I'll watch (no pun intended) in future so I can answer such questions more quickly.

To answer your question, watchgod will not miss changes which occur in the filesystem while other code is running. They'll just be reported as changes next time watch iterates.

More generally the best approach would be to run the other code asynchronously so the main process can get back to watching the directory.

a few other hints for neater python

no need to call list(changes) in the comprehension
os.system is deprecated, better to use subprocess.run
since split_thread_module.py is also python, do you really need to run it in a separate process? Even if you do you might have more luck with python multiprocessing than starting a new process with the system's process initiation.

Overall you might try something like:

from concurrent.futures import ProcessPoolExecutor
from time import sleep

from watchgod import watch


def slow_job(status, location):
    print(f'status: {status}, location: {location}, starting...')
    sleep(10)
    print(f'status: {status}, location: {location}, done')


with ProcessPoolExecutor() as executor:
    for changes in watch('./tests'):
         for status, location in changes:
             executor.submit(slow_job, status, location)

file watcher in python 3.5 using library watchgod

1 Answers1