16

Files are being pushed to my server via FTP. I process them with PHP code in a Drupal module. O/S is Ubuntu and the FTP server is vsftp.

At regular intervals I will check for new files, process them with SimpleXML and move them to a "Done" folder. How do I avoid processing a partially uploaded file?

vsftp has lock_upload_files defaulted to yes. I thought of attempting to move the files first, expecting the move to fail on a currently uploading file. That doesn't seem to happen, at least on the command line. If I start uploading a large file and move, it just keeps growing in the new location. I guess the directory entry is not locked.

Should I try fopen with mode 'a' or 'r+' just to see if it succeeds before attempting to load into SimpleXML or is there a better way to do this? I guess I could just detect SimpleXML load failing but... that seems messy.

I don't have control of the sender. They won't do an upload and rename.

Thanks

tetranz
  • 1,932
  • 3
  • 24
  • 32
  • 2
    If checking for locks doesn't work, maybe you can look at the file's last modified time to see if it was/is still being touched. – deceze Apr 29 '11 at 01:34

7 Answers7

5

Using the lock_upload_files configuration option of vsftpd leads to locking files with the fcntl() function. This places advisory lock(s) on uploaded file(s) which are in progress. Other programs don't need to consider advisory locks, and mv for example does not. Advisory locks are in general just an advice for programs that care about such locks. You need another command line tool like lockrun which respects advisory locks.

Note: lockrun must be compiled with the WAIT_AND_LOCK(fd) macro to use the lockf() and not the flock() function in order to work with locks that are set by fcntl() under Linux. So when lockrun is compiled with using lockf() then it will cooperate with the locks set by vsftpd.

With such features (lockrun, mv, lock_upload_files) you can build a shell script or similar that moves files one by one, checking if the file is locked beforehand and holding an advisory lock on it as long as the file is moved. If the file is locked by vsftpd then lockrun can skip the call to mv so that running uploads are skipped.

Rob Hruska
  • 118,520
  • 32
  • 167
  • 192
crayor
  • 51
  • 1
  • 1
3

If locking doesn't work, I don't know of a solution as clean/simple as you'd like. You could make an educated guess by not processing files whose last modified time (which you can get with filemtime()) is within the past x minutes.

If you want a higher degree of confidence than that, you could check and store each file's size (using filesize()) in a simple database, and every x minutes check new size against its old size. If the size hasn't changed in x minutes, you can assume nothing more is being sent.

Wiseguy
  • 20,522
  • 8
  • 65
  • 81
  • Thanks. filemtime seems like the easiest solution. They are only small files that I check for once per hour so if it hasn't been written to in the last minute or so I should be OK. I tried fopen and then getting an exclusive lock with flock but it always succeeds. I guess I don't understand what vsftp does when lock_upload_files=YES or really much about locking in general. I do understand the concept of an advisory lock so I guess that's not what vsftp uses. – tetranz Apr 29 '11 at 14:06
  • [filectime()](https://www.php.net/manual/en/function.filectime.php) might be another option, as some ftp clients can set a file's modification time if allowed, which could mess up any logic you have that relies on modification time and the current time – georaldc Aug 20 '19 at 20:53
2

The lsof linux command lists opened files on your system. I suggest executing it with shell_exec() from PHP and parsing the output to see what files are still being used by your FTP server.

Sander Marechal
  • 22,978
  • 13
  • 65
  • 96
1

Picking up on the previous answer, you could copy the file over and then compare the sizes of the copied file and the original file at a fixed interval.

If the sizes match, the upload is done, delete the copy, work with the file.

If the sizes do not match, copy the file again.

repeat.

Ajoy
  • 240
  • 2
  • 8
0

Here's another idea: create a super (but hopefully not root) FTP user that can access some or all of the upload directories. Instead of your PHP code reading uploaded files right off the disk, make it connect to the local FTP server and download files. This way vsftpd handles the locking for you (assuming you leave lock_upload_files enabled). You'll only be able to download a file once vsftp releases the exclusive/write lock (once writing is complete).

You mentioned trying flock in your comment (and how it fails). It does indeed seem painful to try to match whatever locking vsftpd is doing, but dio_fcntl might be worth a shot.

Adam Monsen
  • 9,054
  • 6
  • 53
  • 82
0

I guess you've solved your problem years ago but still.

If you use some pattern to find the files you need you can ask the party uploading the file to use different name and rename the file once the upload has completed.

vitoshabg
  • 26
  • 4
0

You should check the Hidden Stores in proftp, more info here: http://www.proftpd.org/docs/directives/linked/config_ref_HiddenStores.html

M_per
  • 313
  • 2
  • 12