3

I would like to have some sort of file handle to a file that survives file moves/renames. This does not have to be platform independent, it's fine if it only works on Java7+linux, and even fine if the files can be assumed to not change directory.

The use-case is that I want to writer a log tailing input source similar to logstash-file-input, i.e. a tool that I can run to tail a file, and even if that file is rotated (i.e. moved/renamed) when my tailing application is down, I want to be able to find that file when the tailing application starts again.

I looked at BasicFileAttributes.fileKey but I haven't found any way to go from a fileKey to a File/Path object.

Does anyone know if this can be done?

Best Regards /Thomas

Thomas Larsson
  • 697
  • 1
  • 8
  • 17
  • I am not a Linux expert, but maybe you could create a hard link to the log file in the filesystem and tail that instead of the actual file? – Philipp Feb 28 '15 at 11:47
  • You can probably use a [`WatchService`](http://docs.oracle.com/javase/8/docs/api/java/nio/file/WatchService.html) for that – fge Feb 28 '15 at 11:48
  • @Philipp. yes that might be a way, i've seen that answer to similar questions. It seems a bit cumbersome though, possibly on par with writing some JNI module for this. – Thomas Larsson Feb 28 '15 at 15:07

1 Answers1

1

This is a trivial demo, setting up a WatchService and waiting for events:

Path dir = Paths.get( "/home", "me", "experiments" );
WatchService watcher =  dir.getFileSystem().newWatchService();
dir.register( watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY );
WatchKey key = watcher.take();
for (WatchEvent<?> event: key.pollEvents()) {
    System.out.println( event.kind().name() + " " + event.context() );
}

If a mv xyz.dat abc.dat is executed in ./experiments while this program is waiting in the take() call, a WatchKey is created containing these events:

ENTRY_DELETE zyx.dat
ENTRY_CREATE xyz.dat

The context is a relative Path from the directory to the changed element.

Later It is possible to register the "identity" of a file on Linux by determining its device and inode, perhaps also adding size and the creation date. You can then, after regaining control, search for this file assuming it is the same directory.

Path dir = Paths.get( "/home", "wlaun", "Java_Experiments" );
Path file = dir.resolve( "a.log" );
Object fileKey = Files.getAttribute( file, "fileKey" );
Object creaTime = Files.getAttribute( file, "creationTime" );
System.out.println( fileKey.toString() + creaTime.toString() );

I should add that 1.8.0_20 seems to confuse creationTime and lastAccessTime.

laune
  • 31,114
  • 3
  • 29
  • 42
  • 1
    It's better to grab a `WatchService` using the `Path` itself: `dir.getFileSystem().newWatchService()`. – fge Feb 28 '15 at 13:11
  • Ah, yes - not guaranteed to have the right file system. Thanks, modified. – laune Feb 28 '15 at 13:19
  • This only works if my tailing application is running when the rename/move takes place or am I missing something? I'm asking for a way to detect this afterwards, if my java application was not running at the time the the rotation took place. – Thomas Larsson Feb 28 '15 at 15:04
  • @ThomasLarssonKron then how would you know what file to track then? This cannot be done unless you have at least a pattern of file names to observe or the like – fge Feb 28 '15 at 15:09
  • @fge Consider that my tailing app is first running and tailing a logfile A, continously writing to some other bookkeeping file the position that it is currently at in A. Then the tailing app crashes. While it is not running the logging application rotates logfile A to A.1 and creates a new logfile A that it logs to. When I start up the tailing app again I would like it to find that logfile A is now logfile A.1 and it should immediately consume all the rest of the data in that file that was written while the tailing app was down. – Thomas Larsson Feb 28 '15 at 15:22
  • @ThomasLarssonKron this scenario just cannot work; you rely on a position in a file, now what tells you that the renamed file, should you detect it, will have identical contents? You can only rely on luck. What you really want is a structured logging framework. Otherwise it's all just guess games. – fge Feb 28 '15 at 15:53
  • @ThomasLarssonKron Not sure whether looking at the file attributes can help you - see the addition to my answer. – laune Feb 28 '15 at 17:15
  • @laune Yea I think this is the way I'll go for this. I'll accept your answer. – Thomas Larsson Feb 28 '15 at 18:45
  • @fge Yes, well this is very much a best effort thing I'm aiming for here. To have more reliable logforwarding solution would mean (in my view) something like having the logging application logging directly to a message queue. Thanks for your thoughts. – Thomas Larsson Feb 28 '15 at 18:47
  • @ThomasLarssonKron Still, it's a rather shaky thing. An app that stays active throughout monitoring things might be safer, or a logging function taking care of enventualities. But it's for you to decide. – laune Feb 28 '15 at 18:50
  • @laune and fge. Yes, I agree. I think I'll skip this complexity and just accept that if the forwarder goes down, someone or something will have to react to it and restart it, accepting that logs might be missed during the downtime. Thanks again for your comments. – Thomas Larsson Feb 28 '15 at 19:07