I'm writing a very simple pipeline for processing uploaded files and I've opted to keep the state of the pipeline on the file system instead of a database. The pipeline itself is very simple because files are processed in temporary locations and then atomically moved into a folder that indicates a specific stage of the pipeline is complete. Once the file is moved into a folder for the next stage the file from the previous stage is deleted. The pipeline is a single process and there is only one of it active at a time so there are no race conditions or crash scenarios to worry about. At worst we repeat work that was done previously and waste some disk space.
The problems are with the very first step of the pipeline. The server that is handling the file uploads moves files into a specific folder to make it available for clients to access and at the same time it creates a symlink in another folder to indicate to the pipeline process that there is work to be done. Since there is a race condition here I'm using file locks to create and delete the symlinks. The pipeline process locks the symlink, processes the file, deletes the symlink, and releases the lock. The upload process does the same thing except it does it for creating the symlink. This takes care of the race condition between deleting the symlink and creating it but there is still a bug here.
It is possible that the upload process will crash after moving the file into place but before creating the symlink to indicate there is a file that needs to be processed. I would like to know what is the best way to handle this case. I can in theory create a file at the beginning of the upload process and then delete it upon successful creation of the symlink to indicate successful creation of the symlink but this again leads to the same locking problems as before because there are multiple upload processes that need to coordinate between each other.
Is there a simpler way to handle the case of the crashed upload process?
