System.Io.Directory::GetFiles() Polling from AX 2009, Only Seeing New Files Every 10s

Question

I wrote code in AX 2009 to poll a directory on a network drive, every 1 second, waiting for a response file from another system. I noticed that using a file explorer window, I could see the file appear, yet my code was not seeing and processing the file for several seconds - up to 9 seconds (and 9 polls) after the file appeared!

The AX code calls System.IO.Directory::GetFiles() using ClrInterop:

interopPerm = new InteropPermission(InteropKind::ClrInterop);
interopPerm.assert();
files = System.IO.Directory::GetFiles(#POLLDIR,'*.csv');
// etc...
CodeAccessPermission::revertAssert();

After much experimentation, it emerges that the first time in my program's lifetime, that I call ::GetFiles(), it starts a notional "ticking clock" with a period of 10 seconds. Only calls every 10 seconds find any new files that may have appeared, though they do still report files that were found on an earlier 10s "tick" since the first call to ::GetFiles().

If, when I start the program, the file is not there, then all the other calls to ::GetFiles(), 1 second after the first call, 2 seconds after, etc., up to 9 seconds after, simply do not see the file, even though it may have sitting there since 0.5s after the first call!

Then, reliably, and repeatably, the call 10s after the first call, will find the file. Then no calls from 11s to 19s will see any new file that might have appeared, yet the call 20s after the first call, will reliably see any new files. And so on, every 10 seconds.

Further investigation revealed that if the polled directory is on the AX AOS machine, this does not happen, and the file is found immediately, as one would expect, on the call after the file appears in the directory.

But this figure of 10s is reliable and repeatable, no matter what network drive I poll, no matter what server it's on.

Our network certainly doesn't have 10s of latency to see files; as I said, a file explorer window on the polled directory sees the file immediately.

What is going on?

I'm not sure if this is an Axapta phenomena or has something do to with how System.IO.Directory interacts with network drives. You may have a better chance of getting an answer if you add some tags in that regard. Also have you tried a similar program implemented in .NET, to check if it exhibits the same behavior? — FH-Inway, Feb 16 '17 at 17:23

score 2 · Answer 1 · answered Feb 16 '17 at 19:04

Sounds like your issue is due to SMB caching - from this technet page:

Name, type, and ID
Directory Cache [DWORD] DirectoryCacheLifetime

Registry key the cache setting is controlled by HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Lanmanworkstation\Parameters

This is a cache of recent directory enumerations performed by the client. Subsequent enumeration requests made by client applications as well as metadata queries for files in the directory can be satisfied from the cache. The client also uses the directory cache to determine the presence or absence of a file in the directory and uses that information to prevent clients from repeatedly attempting to open files which are known not to exist on the server. This cache is likely to affect distributed applications running on multiple computers accessing a set of files on a server – where the applications use an out of band mechanism to signal each other about modification/addition/deletion of files on the server.

In short try to set the registry key

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Lanmanworkstation\Parameters\DirectoryCacheLifetime

to 0

Thank-you! You are correct, and your solution works! I didn't even need to reboot the AOS machine. — Fleetie, Feb 17 '17 at 08:59

Fleetie · Answer 2 · 2017-02-17T12:32:43.273

Thanks to @Jan B. Kjeldsen , I have been able to solve my problem using FileSystemWatcher. Here is my implementation in X++ :

class SelTestThreadDirPolling
{
}

public server static Container SetStaticFileWatcher(str _dirPath,str _filenamePattern,int _timeoutMs)
{
    InteropPermission interopPerm;
    System.IO.FileSystemWatcher fw;
    System.IO.WatcherChangeTypes watcherChangeType;
    System.IO.WaitForChangedResult res;
    Container cont;
    str fileName;
    str oldFileName;
    str changeType;
    ;

    interopPerm = new InteropPermission(InteropKind::ClrInterop);
    interopPerm.assert();
    fw = new System.IO.FileSystemWatcher();
    fw.set_Path(_dirPath);
    fw.set_IncludeSubdirectories(false);
    fw.set_Filter(_filenamePattern);

    watcherChangeType = ClrInterop::parseClrEnum('System.IO.WatcherChangeTypes', 'Created');
    res = fw.WaitForChanged(watcherChangeType,_timeoutMs);
    if (res.get_TimedOut()) return conNull();

    fileName = res.get_Name();
    //ChangeTypeName can be: Created, Deleted, Renamed and Changed
    changeType = System.Enum::GetName(watcherChangeType.GetType(), res.get_ChangeType());

    fw.Dispose();
    CodeAccessPermission::revertAssert();

    if (changeType == 'Renamed') oldFileName = res.get_OldName();

    cont += fileName;
    cont += changeType;
    cont += oldFileName;
    return cont;
}

void waitFileSystemWatcher(str _dirPath,str _filenamePattern,int _timeoutMs)
{
    container cResult;
    str filename,changeType,oldFilename;
    ;

    cResult=SelTestThreadDirPolling::SetStaticFileWatcher(_dirPath,_filenamePattern,_timeoutMs);
    if (cResult)
    {
        [filename,changeType,oldFilename]=cResult;
        info(strfmt("filename=%1, changeType=%2, oldFilename=%3",filename,changeType,oldFilename));
    }
    else
    {
        info("TIMED OUT");
    }
}

void run()
{;
    this.waitFileSystemWatcher(@'\\myserver\mydir','filepattern*.csv',10000);
}

I should acknowledge the following for forming the basis of my X++ implementation:

https://blogs.msdn.microsoft.com/floditt/2008/09/01/how-to-implement-filesystemwatcher-with-x/

score 1 · Accepted Answer · edited May 23 '17 at 12:00

1

I would guess DAXaholic's answer is correct, but you could try other solutions like EnumerateFiles.

In your case I would rather wait for the files rather than poll for the files. Using FileSystemWatcher there will be a minimal delay from file creation till your process wakes up. It is more tricky to use, but avoiding polling is a good thing. I have never used it over a network.

edited May 23 '17 at 12:00

Community

1
1

answered Feb 16 '17 at 22:02

Jan B. Kjeldsen

17,817
5
32
50

Your reply has been most helpful! I have even been able to implement it in X++; see my post below. – Fleetie Feb 17 '17 at 11:11

System.Io.Directory::GetFiles() Polling from AX 2009, Only Seeing New Files Every 10s

3 Answers3