0

I have a "Process" directory with a process.py script in it that does some image processing and uploading to s3.

The script scans "Process" directory for sub directories and processes the files within.

My problem is I have process.py being called via launchd every few hours, but I ran into an issue where the script ran while files (multi gig sub directories) where being dropped into the "Process" directory.

How would you deal with something like this?

Sam Luther
  • 1,170
  • 3
  • 18
  • 38
  • work in a temporary directory – ergonaut Oct 29 '15 at 16:19
  • "An issue"? That's a bit vague – if you can't fix the cause of the issue, you might try to remove `wx` permissions (write and enter) from the folder while you're handling it, but that's very kludgey and the copying into will simply fail with permission denied. – Ulrich Schwarz Oct 29 '15 at 16:20
  • "the issue" is the script being run in the middle of a long copy – Sam Luther Oct 29 '15 at 17:34
  • That's not really an "issue", that's a perfectly normal mode of operation. Are you saying that something bad happened while the script ran in the middle of a long copy? If so, precisely what bad thing happened? – Robᵩ Oct 29 '15 at 17:57

1 Answers1

0

I'd choose some atomic file system operation as sort of an ownership flag.

I'd probably use os.rename(). While the files were being copied in, the subdir in question would be named, say, dir/subdir.DONTPROCESSME. When all of the data is copied in and the subdir is in a consistent state, the copy-in script would:

os.rename('dir/subdir.DONTPROCESSME', 'dir/subdir.DOPROCESSME')

Similarly, the process launched by launchd would have code like:

for subdir in os.listdir('dir'):
    if subdir.endswith('.DONTPROCESSME'): continue
Robᵩ
  • 163,533
  • 20
  • 239
  • 308
  • sounds interesting. the subdirs are not being copied into the process directory by the script, they are being dropped in by someone else on the network. How would one, or is it possible to detect whether a copy operation is complete or still in progress? – Sam Luther Oct 29 '15 at 17:51
  • It probably isn't possible to check, except if the entity that drops the subdir in cooperates -- by giving the subdir a special name when the copy completes, or by creating a sentinal file in the subdir, or by some other cooperative protocol. – Robᵩ Oct 29 '15 at 17:53