Recursive pyinotify watch stops sub-directories from being removed

Question

This is a strange one and strace is not giving me any useful information. I am using pyinotify 0.9.6 to watch a directory recursively so I can commit changes about it to a MySQL database. The problem is when I have pyinotify running (daemonized or not), i can remove files in the sub-directories but not the sub-directories themselves. rmdir exits with status 0 and everything looks right on the system level, but the sub directory is still there just chilling. I may just be being stupid as I am new to the framework, but here is an example of how I am initializing the watch:

wm = pyinotify.WatchManager()
notifier = pyinotify.Notifier(wm)
wm.add_watch('/path/to/dir', pyinotify.IN_CLOSE_WRITE, rec=True, auto_add=True,\
        proc_fun=CommitFunction() # I can supply this if you think it's relevant
notifier.loop(daemonize=True, callback=None,\
        stdout='/path/to/log.log, stderr='/path/to/log.log')

So in this example if there was a file '/path/to/dir/file.txt' I could run a rm on 'file.txt' and it gets deleted, but if there was a sub-directory '/path/to/dir/subdir' running rm -r on 'subdir' exits cleanly but the directory doesn't actually get deleted.

Furthermore my logs aren't getting written, but I am pretty sure that's my fault somewhere.

EDIT:

Here is an example CommitFunction:

class CommitFunction(pyinotify.ProcessEvent):
    def process_default(self, event):
        dir = event.path
        file = event.name
        print "%s is the new file" % os.path.join(dir, file)

EDIT2: Actually my logs probably aren't getting written because I don't call a log or print function anywhere during the commit. I just write straight to my MySQL database

EDIT3: Ok here goes. Hopefully I am not delving too deep. Here is what I try on the command line:

bash$ cd /var/www
bash$ mkdir subdir
bash$ rmdir subdir
bash$ ls
subdir

Here is the actual commit function:

class CommitFunction(pyinotify.ProcessEvent):
    def process_default(self, event):
        dir = event.path
        file = event.name
        dbaccess.commit(dir, file) # dir corresponds to a project name

I can keep going deeper if you want, but dbaccess has all my functions for committing and querying the database (nothing really touching the fs) and it further draws from a 'models' file that defines my tables. It's a flask/uwsgi app if that helps at all

I don't know that the specifics of CommitFunction are important,but having an example that we can just copy and paste and run would be helpful. — larsks, Sep 01 '15 at 18:16
Added example to original post. Hopefully I am not being too much of a nuisance. I have been using this forum for years, but finally had to create an account when I had a problem I couldn't find an answer for — aizimmer, Sep 01 '15 at 18:27

score 2 · Answer 1 · answered Sep 01 '15 at 18:27

2

Using this code:

import pyinotify
import os


class CommitFunction(pyinotify.ProcessEvent):
    def process_default(self, event):
        dir = event.path
        file = event.name
        print "%s is the new file" % os.path.join(dir, file)


wm = pyinotify.WatchManager()
notifier = pyinotify.Notifier(wm)
wm.add_watch('/tmp/testdir', pyinotify.IN_CLOSE_WRITE, rec=True,
             auto_add=True, proc_fun=CommitFunction())
notifier.loop(daemonize=False, callback=None)

I can create and delete subdirectories in the target directory without error:

bash-4.3$ pwd
/tmp/testdir
bash-4.3$ mkdir subdir
bash-4.3$ rm -r subdir
bash-4.3$ ls
bash-4.3$

In response to the above session, the code logged the following on stdout:

/tmp/testdir/subdir is the new file
/tmp/testdir/subdir/ is the new file

Can you confirm whether the above exhibits the behavior you have described in your environment? If not, can you update your question with a complete example that does demonstrate the problem?

answered Sep 01 '15 at 18:27

larsks

277,717
41
399
399

Oddly enough I got the same results as you that time. Now I am even more confused. – aizimmer Sep 01 '15 at 18:32
1

This is why people ask for [Minimal, Complete, and Verifiable examples](http://stackoverflow.com/help/mcve), because if we can't reproduce your problem it's hard to solve. This suggests a problem with your actual code, or just some sort of not-enough-coffee-yet think-o. – larsks Sep 01 '15 at 18:34
I suppose you could be right, but if I stop the pyinotify daemon (even with the uwsgi app still running) I can then successfully remove the sub-directories – aizimmer Sep 01 '15 at 18:40
Can you post a complete example that exhibits that behavior? – larsks Sep 01 '15 at 18:41
Yea let me see if I can reproduce it on a smaller scale. If I can't I'll close the topic, and just accept my own stupidity somewhere – aizimmer Sep 01 '15 at 18:45
Ok I found the problem. My commit function was checking if the "project" directory existed already and creating it if need be. That was there for the web portal's use. This watch is so I can scp stuff to the file store and have it show up in the web portal. However, I am still confused as shouldn't pyinotify only be firing on a WRITE to the directory? – aizimmer Sep 01 '15 at 18:58

score 0 · Answer 2 · answered Sep 01 '15 at 19:12

OK so the problem I was having is that IN_CLOSE_WRITE was also getting called on removal of a sub-directory, which in turn was firing a function I had to make sure a directory for the "project" existed when uploaded through the web portal.

I am still confused, but this particular question is solved

Recursive pyinotify watch stops sub-directories from being removed

2 Answers2