Read file attibutes for all files in the tree using as few IO operation as possible

Question

I have a lot of small files on a NFS drive (Amazon EFS in my case). Files are provided over HTTP protocol, very similar to a classical Web-Server it does. As I need to validate the last modification of the file, it takes at least a single I/O per file request. It is a case even if I already cached the file body in RAM.

Is there a way to read the last modify attribute for all the files in the tree (or at least in a single directory) using only a single I/O operation?

There it a method Files.readAttributes which reads multiple attributes of a single file as a bulk operation. I am looking for bulk operation to read a single attribute of multiple files.

UPDATE: in case of NFS this question is how to utilize NFS command READDIRPLUS. This command does exactly what I need, but it seems to be no way to use it out of Java I/O library.

score 4 · Accepted Answer · answered Oct 04 '19 at 02:37

4

I don't know of a standard Java class to list all the files and modified time in one operation, but if you are permitted to utilise the host environment and the NFS drive is mount you could adapt the following technique to suit your environment:

ProcessBuilder listFiles = new ProcessBuilder("bash", "", "ls -l");
Process p = listFiles.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
String inputLine;
List<String> filesWithAttributes = new ArrayList<String>();

while ((inputLine = reader.readLine()) != null) {
    filesWithAttributes.add(inputLine);
}

answered Oct 04 '19 at 02:37

Najinsky

628
4
12

Useful link about Java, NFS and READDIRPLUS command (slides 28 to 44): https://www.slideshare.net/GregBanks1/java-hates-linux-deal-with-it – 30thh Oct 04 '19 at 08:21
@30thh interesting link, a bit hard to digest as it looks like slides to accompany a presentation, but the gist seems to be that Files.newDirectoryStream(dir) should give better performance by delaying getdents(), as shown on page 43. – Najinsky Oct 04 '19 at 15:59
@Najinsky I don't think newDirectoryStream is somehow better regarding I/O. It uses less RAM because you don't need to store the huge collection of File-Objects. This sort of "better performance" is neglectable in context of this question. – 30thh Oct 05 '19 at 04:39

score 0 · Answer 2 · answered Oct 02 '19 at 18:47

I think this question may be a duplicate of Getting the last modified date of a file in Java. Nonetheless, I think if you use lastModified() of the File class, you probably use the least IO operation. So for this, I would use something similar to icyrock.com's answer. Which would be:

new File("/path/to/file").lastModified()

Also, the answers to the questions java - File lastModified vs reading the file might be able to give you helpful information.

Read file attibutes for all files in the tree using as few IO operation as possible

2 Answers2