3

My objective is to poll the SFTP server for changes. My first thought is to check if the number of files in the dir changed. Then maybe some additional checks for changes in the dir.

Currently I'm using the following:

try {
        FileSystemOptions opts = new FileSystemOptions();
        SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(opts, "no");
        SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, true);
        SftpFileSystemConfigBuilder.getInstance().setTimeout(opts, 60000);

        FileSystemManager manager = VFS.getManager();
        FileObject remoteFile = manager.resolveFile(SFTP_URL, opts);

        FileObject[] fileObjects = remoteFile.getChildren();
        System.out.println(DateTime.now() + " --> total number of files: " + Objects.length);
        for (FileObject fileObject : fileObjects) {
            if (fileObject.getName().getBaseName().startsWith("zzzz")) {
                System.out.println("found one: " + Object.getName().getBaseName());
            }
        }
} catch (Exception e) {
        e.printStackTrace();
}

This is using apache commons vfs2 2.2.0. It works "fine", but when the server has too many files, it takes over minutes just to get the count(currently, it takes over 2 minutes to get the count for a server that has ~10k files). Any way to get the count or other changes on the server faster?

iCodeLikeImDrunk
  • 17,085
  • 35
  • 108
  • 169
  • Do you need filter files? or just count it. – Dezigo Jun 25 '15 at 19:43
  • @Dezigo my objective is to poll the sftp server and react if there is changes. so ideally i thought first way is to check the number of files in that dir. maybe you can suggest better ways of checking? – iCodeLikeImDrunk Jun 25 '15 at 19:44

2 Answers2

2

Unfortunately there's no simple way in the SFTP protocol to get the changes. If you can have some daemon running on the server OR if the source of the new files can create/update a helper file, creation of such file with the last modification time in its name or contents can be an option.

Eugene Mayevski 'Callback
  • 45,135
  • 8
  • 71
  • 121
1

I know the SFTP protocol fairly well, having developed commercial SFTP clients and an SFTP server (CompleteFTP), and as far as I know there's no way within the protocol to get a count of files in a directory without listing it. Some servers, such as ours, provide ways of adding custom commands to servers that you can invoke from the client, so it would be possible to add a custom command that returns the number of files in a directory. CompleteFTP also allows you to write custom file-systems so you could potentially write one that only shows files that have changed after a given timestamp when you do a listing, which might be another approach. Our server only runs on Windows though, so that might be show-stopper for you.

HansA
  • 1,375
  • 2
  • 11
  • 19
  • custom commands seems useful, could you provide any resource to that? – iCodeLikeImDrunk Jun 29 '15 at 14:30
  • 1
    Sorry for my very slow reply. I forgot to check my StackOverflow inbox until now. There are two ways to add custom commands by Javascript (see [here](http://enterprisedt.com/products/completeftp/doc/guide/html/customcommandscript.html)) and by .NET (see [here](http://enterprisedt.com/products/completeftp/doc/guide/html/sitecommandextensions.html)). – HansA Jul 22 '15 at 01:09
  • thanks for the references, will definitely check them out! – iCodeLikeImDrunk Jul 22 '15 at 15:35
  • 1
    By the way, in the next version (9.0) you'll be able to access CompleteFTP's virtual file-system from the Javascript extension. This way the custom command you'd need would be implemented as follows: `system.getFile(path).getFiles().length` where `path` is the path in the virtual file-system, i.e. the same that you'd use from within regular SFTP. – HansA Jul 23 '15 at 02:47