Remake catalina.out without restart of server

Question

Is it possible to remake a deleted catalina.out on a tomcat server without restarting the server? The catalina.out got accidentally deleted but it's needed due to a funky cluster configuration (4 server, and they seem to draw data from each other that is written in catalina.out) but we can't restart the server (we'd need to restart all 4) right now. Would it be enough if we just made the file again and giving the tomcat user all access to it?

Answer 1

No, it would not be enough. The main problem here is that the file you've deleted still exists, or, to be precise, the data area the file occupies still exists. You can't access it, since you've deleted the inode which points to the file, but the tomcat process still has a file descriptor pointing to that file. From the viewpoint of Tomcat, the file still exists, and can be written to. If you create a new file, it will be a different file, not the one currently used by the tomcat process.

If you're using Linux, you might salvage the situation however (it might work for other Unix variants, I don't know).

Every open file descriptor listed in the /proc/(process_id)/fd/ directories. You have to find the PID of the Tomcat process, and look in the /proc/(tomcat_pid)/fd/ directory for a symlink pointing to /.../catalina.out (deleted).

You can make a symlink pointing to the above symlink, and the resulting filesystem entry will be the deleted catalina.out file, so you will be able to read from it. There are a few problems with this approach however:

1. This is a "hack", not a solution. It might not work at all.

2. The resulting file will be available only for the user who runs the Tomcat process (and for root).

3. After Tomcat exits, the file will disappear, and you'll end up with a dead link.

It is probably better to just restart Tomcat.

Answer 2

I had a similar situation where someone did rm -rf catalina.out instead of truncating the file, which kept the file descriptors working like:

$ ls -asl /proc/$PID/fd | grep catalin Nov 29 11:56 1 -> /opt/tomcat/logs/catalina.out (deleted) Nov 29 11:56 2 -> /opt/tomcat/logs/catalina.out (deleted)

To fix this you can use the tool "gdb". You will need to find the pid of your process, for example like:

ps faux | grep java

Then use the tool "gdb" to attach the process and alter the file descriptors. But there will be service unavailability for as long as the process is attached.

gdb attach $PID (gdb) p close(1) $1 = 0 (gdb) p close(2) $2 = 0 (gdb) p fopen("/opt/tomcat/logs/catalina-gdb.out", "w") $3 = 11226208 (gdb) p fileno($3) $4 = 1 (gdb) quit A debugging session is active.

Inferior 1 [process 9845] will be detached.

Quit anyway? (y or n) y

I did touch the file and set the correct ownership before doing this, but it should not be necessary.

I found this fix here:

http://dnaeon.github.io/remove-file-handles-with-gdb/

In some cases this is preferable to restarting tomcat, because some java applications can take more than five or ten minutes to restart...

Answer 3

Just send a SIGUSR1 signal to the tomcat.

From the documentation:

If you are using jsvc 1.0.4 or later (from Apache Commons Daemon project) to launch Tomcat, you can send SIGUSR1 signal to jsvc to get it to re-open its log files 

kill -SIGUSR1 <tomcat pid>

This will release the file lock from the deleted/moved file and create a new one.