0

Too much logging activity over the weekend led to the following error being thrown by ColdFusion:

Message:    No space left on device
StackTrace: java.io.IOException: No space left on device at
            java.io.FileOutputStream.writeBytes(Native Method) at 
            java.io.FileOutputStream.write(FileOutputStream.java:269) at 
            coldfusion.compiler.NeoTranslator. ......

By this morning pages on the ColdFusion web site were not loading at all. The disc (12Gig) was over 99% used up. We moved several files to the second disc and now it's at about 80%, much lower than where it's always been. We're going to direct logging activity to the second disc (100Gig) to prevent a repeat. Having created space on the disc we restarted apache and coldfusion but still pages are not loading.

When we run top -H it appears that java is running at close to 100% CPU. Does anyone have a clue what's going on or what info I need to provide so someone can figure it out?

The set up is AWS, ubuntu 13.04, coldfusion 10, mysql (rds).

UPDATE

I have made some really strange but hopefully helpful observations. I am still trying to locate a tool to help with getting thread dumps. Whenever I restart ColdFusion, most pages load fine and CPU usage seems normal, mostly 0.7 - 1.5% but once in a while spikes of up to 10% can be seen. But there's one particular page, which when I try to load, causes CPU uses to rise to 97% always. The pages that load fine have a simple query reading data from one table. This problematic page has an inner join, and reads data from two tables. I don't know how helpful this is but I think it is too consistent to be insignificant.

UPDATE 2

And since this issue started the following errors appeared for the first time and there're hundreds of lines of it:

[2243:140630871263104] [error] ajp_send_request::jk_ajp_common.c (1649):
(cfusion) connecting to backend failed. Tomcat is probably not started 
or is listening on the wrong port (errno=111)

// and

[2234:140630871263104] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1027): 
Failed opening socket to (127.0.0.1:8012) (errno=111)

// and 

[2756:139822377088896] [info] jk_handler::mod_jk.c (2702): No body with status=500 
for worker=cfusion

UPDATE 3 - RESOLVED

After stripping the "offending" page of all the code in it and simply replacing it with some plain text and tried to load the page several times, we realized that ColdFusion was not loading the live pages. It was loading cached compiled versions of the pages, normally found in a subfolder named <cf-root>/cfusion/wwwroot/WEB-INF/cfclasses. Removing (or renaming) the subfolder resolved the issue.

PeterKA
  • 24,158
  • 5
  • 26
  • 48
  • 1
    It depends on your application, most likely when you are restarting, it is trying to do some pending task , which is eating up all your CPU. – Juned Ahsan Feb 08 '16 at 23:04
  • Is there a way to find out what that task could be? – PeterKA Feb 08 '16 at 23:06
  • 1
    try [jstack](http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstack.html) and see what JVM is doing. – ZhongYu Feb 08 '16 at 23:06
  • 1
    a profiler should help, visualvm comes along with jdk. – Juned Ahsan Feb 08 '16 at 23:08
  • @bayou.io, tried that but jstack is not installed. The java on the server is that packaged with ColdFusion. Is there a way to install it on it's own? – PeterKA Feb 08 '16 at 23:08
  • Looks like they advise to use jstack or their own tool. See http://blogs.coldfusion.com/post.cfm/taking-thread-dumps-from-coldfusion-server-programmatically – Aaron Feb 08 '16 at 23:11
  • @Aaron, I'll give the tool a try. – PeterKA Feb 08 '16 at 23:28
  • For some reason the thread dump log file is not being created. Trying to figure out if it's a permissions issue or some setting in CF Admin. – PeterKA Feb 09 '16 at 00:14

1 Answers1

1

You would probably want to provide thread dumps, a few of them taken at a few seconds interval, and gc logs.

Thread dumps can be produced with jstack (a tool provided in your jdk's bin directory), and garbage collector logs must be activated beforehand.

Aaron
  • 24,009
  • 2
  • 33
  • 57
  • This tool does not seem to work: http://blogs.coldfusion.com/post.cfm/taking-thread-dumps-from-coldfusion-server-programmatically. I followed the instructions but looks like the functions do not return any output at all. – PeterKA Feb 09 '16 at 14:36
  • I'm sorry but I'm not familiar with ColdFusion at all. If you're not sure of your tempDirectory's location and are using a linux based system you could try searching for the threaddump file with the following command : `find / -name threaddump*`. On Oracle's JVMs you could also use `kill -3` to ouput a threaddump to the stderr of the JVM, but I can't vouch for ColdFusion – Aaron Feb 09 '16 at 15:17
  • I understand. Thank you for your time. Just to be sure I was not losing my mind I changed to script to output to the browser page instead of to a file and nothing was output to the browser page. – PeterKA Feb 09 '16 at 15:26
  • You could still try to install jstack as suggested, but I never did that either, so anyway you're on your own, sorry! Good luck! – Aaron Feb 09 '16 at 15:36