-1

I have a Delphi5 application which exports a file (.pdf) and a very small metadata file to a network location. The intention is that these 2 files should be processed, and then removed, by a polling .NET application.

My approach is to

  1. Write the metadata file with the extension '.part'
  2. Generate the .pdf
  3. Rename the .part file to .dat

The .NET process is looking for files with the extension '.dat' only, so I would expect there to be no conflict between the 2 reader/writers. However, the .NET process is occasionally logging the following error ...

System.IO.IOException: The process cannot access the file '\\server\Path\FileName.dat' because it is being used by another process.

(I say occasionally - we are currently testing, so when volumes increase this may become much more of an issue)

The Delphi code looks like this :

AssignFile(FTextFile, Format('%s\%s.part', [DMSPath, FullFileName])); 
try
  try
    ReWrite(FTextFile);
    Writeln(FTextFile, MetaDataString);
  finally  
    CloseFile(FTextFile);
  end;
except
  raise ELogFileException.Create( LOGFILEWRITEFAILURE );
end;

Then there is a separate method which performs the following lines of code

if FindFirst(Format('%s\*.part',[DMSPath]), faAnyFile, SearchRec) = 0 then begin
  repeat
    OldName := Format('%s\%s',[DMSPath, SearchRec.Name]);
    NewName := Format('%s\%s',[DMSPath, ChangeFileExt(SearchRec.Name, '.dat')]);
    RenameFile(OldName, NewName);
  until FindNext(SearchRec) <> 0;
  FindClose(SearchRec);
end;

I cannot see anything inherently wrong with this code and we have a couple of remedies in mind, but I have 2 questions

  1. Should I try a different technique to more reliably protect the '.dat' file until it is fully ready
  2. What circumstances could be causing this?

So far there has been one suggested cause - Antivirus software.

Any suggestions as to how the file might be produced differently? Note that my application is Delphi5; I wondered if there was a newer, more 'atomic' version of the 'MoveFileA' WinApi call I could use.

Hugh Jones
  • 2,706
  • 19
  • 30
  • 1
    Common cause of this is your AV software – David Heffernan Sep 22 '14 at 15:39
  • Thanks, @David - I hadn't thought of that – Hugh Jones Sep 22 '14 at 15:41
  • Windows Search used to cause this problem though I haven't seen it recently so it might be fixed now. If this is the cause you can work around it by setting the "don't index" attribute on files you create. – arx Sep 22 '14 at 15:45
  • 1
    [When in doubt, run Process Monitor](http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-4-Process-Monitor). Seriously: run procmon, setup some filtering, wait for the exception to occur and look what other process is using your `.dat` file. – Lieven Keersmaekers Sep 23 '14 at 05:56
  • @HughJones - np and should you find the culprit, please post the results. – Lieven Keersmaekers Sep 24 '14 at 18:04
  • @Lieven - I am unlikely to find the culprit because a solution is available; the consumer is now retrying before erroring and that has solved it for us. The most likely candidates are i) that the NAS device (where the file is being placed) is paying a part or ii) that the Anti Virus Software on the producer is 'Kicking in' momentarily on renaming the file. This second suggestion totally fits the bill. The only reason for mentioning the NAS storage is that we have experienced all sorts of odd behaviour recently since a migration – Hugh Jones Sep 25 '14 at 08:59

2 Answers2

0

In the past, we had a problem with file being locked like that. Investigation pointed to Windows' Prefetch. The file being affected were not on a network directory though.

As far as I know, Prefetch only work on process startup and/or while booting (Controlled by a registry key), so it might not apply to your current situation.

You can check the "C:\"Windows"\Prefetch\" directory. If prefetch is active, it should contains multiple *.pf files. If there is one with your executable filename, it might be worth investigating.

Ken Bourassa
  • 6,363
  • 1
  • 19
  • 28
0

Personally speaking, because there are multiple files involved, I'd create a separate lock file (eg, myfile.lck) that you write first. If the polling app sees it in the folder, then it stops looking for other files. Once that file is gone, then it dives deeper. I don't know if this would solve the problem you're encountering or not, but I'd give it a try. (Files with .dat extensions are frequently created by malicious evildoers, so they can raise spurious issues through other sources, like AV software. A lock file with 0 bytes in it is generally harmless and disregarded.)

David Schwartz
  • 1,756
  • 13
  • 18
  • 1
    Wouldn't that just delay the problem? Even if the consumer application is changed to look for a lock file first, it still needs to read the dat file, so there's still a possibility that the dat will be in use. When the consumer application says the file is still in use by another process, we don't know what process that is. I think you're assuming it's the producer application. Either way, the producer has already closed the dat file prior to renaming it. – Rob Kennedy Sep 22 '14 at 22:22
  • 1
    Don't take my word for it. This is how Unix/Linux systems have done it forever. Fundamentally, you're pointing at a concurrency issue. You need to use a flag of some kind. File create operations are generally regarded as atomic. You'll either get a return code saying the file was created, or an error code saying the file already exists. If it exists, you do not proceed. If you created it, then it's safe to proceed. Just be sure to delete it when you're done. – David Schwartz Sep 22 '14 at 23:01
  • Obviously this doesn't work if there are other apps that don't follow this protocol and access the other files anyway. Again, .dat files are frequently created by malware, so I don't know if there's much you can do about this if it's your AV software, other than going in and flagging the file as "safe", if that's even possible. – David Schwartz Sep 22 '14 at 23:04