0

A file is copied from machine1/dir1 to machine2/dir2. I have added a inotify watch on the dir2 for any new files created. Now if the file is large, it might take a few seconds to be fully on the new machine. If I'm not wrong, inotify will notify the application as soon as it detects an event. So if the file status has to be checked, How should it be done ?

King
  • 1,170
  • 2
  • 16
  • 33
  • 2
    Did you try using the IN_CLOSE_WRITE inotify event? It might work if the transfer applicaion doesn't open and close the file for each data packet received. – Andrés Senac Aug 27 '12 at 08:27
  • IN_CLOSE_WRITE is a good option. will check out. – King Aug 27 '12 at 11:47

3 Answers3

3

Save downloaded file with temporary filename (or to other directory) and rename it to expected filename when file moved successfully.

Nginx for example use this method to store cached data Caching data is first written to the temporary file which is then moved to the final location in a cache directory. Cheap and atomic rename syscall is performed instead of a full file copy So it's better to use the same file system in both locations

hostmaster
  • 1,822
  • 16
  • 17
  • This is the way to go. There is no way to find out whether the file is still being written -- the best you could do is find out whether any other program has opened the file, but that is a lot of effort (it requires enumerating all open files in all existing processes), requires root permission and is difficult to get right when there are other programs (such as a virus scanner) around that had the same idea. – Simon Richter Aug 27 '12 at 08:30
  • 1
    The easiest way to achieve this would be to use `rsync` for copying. – Simon Richter Aug 27 '12 at 08:30
  • rsync do that exactly the same way i mean it creat temporary file and rename it than – hostmaster Aug 27 '12 at 09:02
  • Indeed, that's why I suggested using it. – Simon Richter Aug 27 '12 at 10:48
  • I watch on the directory level. And moreover machine2 is not downloading it but machine is uploading it to machine2. machine2 won't have much control over that transfer. – King Aug 27 '12 at 11:48
  • In you case you may upload file to temporary destination and rename it then. So new file appears in watched directory only after successful upload. – hostmaster Aug 27 '12 at 11:54
  • Actually IN_CLOSE_WRITE works for me, as I just SCP it(one time file open and close) :) But What is explained is valid still, if file will open and closed in batched write (depends on the how the upload is done) – King Aug 29 '12 at 14:35
2

There's no way to answer this because it depends on the application environment and requirements. It might do to see that the file hasn't been modified for 60 seconds. It might require checking every few seconds. It depends.

David Schwartz
  • 179,497
  • 17
  • 214
  • 278
1

Using IN_CLOSE_WRITE works if its only a scp from one machine to another. Otherwise, it depends on the way the file is uploaded from one machine to another. If its a one time open and close , IN_CLOSE_WRITE is the way to do it.

Both the answers above make sense depending on how we do it.

King
  • 1,170
  • 2
  • 16
  • 33