Our enterprise application deployed in Jboss wildfly 8.2. The jboss console log is being set during the startup using an environment variable - JBOSS_CONSOLE. This ensures that any thread dump triggered using kill -QUIT is dumped to jboss_console.log. The GC statistics(collected using -XX:+PrintGCTimeStamps -XX:+PrintGCDetails) are also sent to this file.
The rotation of the log file is done by
cat /dev/null > jboss_console.log
Even though this brings down the file size, on triggering a thread dump using, the file size increases to (size of file before rotation + size of new thread dump). So, the file size doesn't come down unless we restart the jboss and explicitly remove the file.
Also, there is a clear discrepancy in output of du & ls commands.
[root@myserver logs]# du -csh jboss_console.log
28M jboss_console.log
28M total
[root@myserver logs]# ls -lrtS jboss_console.log -h
-rwxr-xr-x. 1 admin admin 1.7G Jun 22 05:28 jboss_console.log
The issue is that after a month of jboss restart, the file system operations slow down to one tenth. I.e. if I had a simple static text file of size 1 GB hosted in my apache, downloading it to my local box would be done in like say 3 mins. But after a month, the same operation takes 30 mins. On restarting the jboss, the issue clears off immediately. There is no significant cpu, memory or IO spike in the system.
Is this being caused by the jboss_console.log size?