2

I just had a Tomcat instance stop responding (it started working fine after being killed and restarted), and in the log messages it looks like right before it stopped responding someone was spamming it with invalid requests. Here's an excerpt from the log file:

[WARNING] [talledLocalContainer] WARNING: Parameters: Character decoding failed. Parameter 'ì°ÅM©2a
                                                                                                      7)ýúVßiPÑUÙvPÏB%<kNíR¹YÃÁéIÿ4^ØsZ' with value '' has been ignored. Note that the name and value quoted here may corrupted due to the failed decoding. Use debug level logging to see the original, non-corrupted values.
[WARNING] [talledLocalContainer] Aug 6, 2012 5:23:16 PM org.apache.tomcat.util.http.Parameters processParameters
[WARNING] [talledLocalContainer] WARNING: Parameters: Character decoding failed. Parameter 'ì°ÅM©2a
                                                                                                      7)ýúVßiPÑUÙvPÏB%<kNíR¹YÃÁéIÿ4^ØsZ' with value '' has been ignored. Note that the name and value quoted here may corrupted due to the failed decoding. Use debug level logging to see the original, non-corrupted values.
[INFO] [talledLocalContainer] 19:58:04 [InputSanitizerFilter] WARN:  Removing potentially malicious parameter from request:  batch
[WARNING] [talledLocalContainer] Aug 6, 2012 9:11:39 PM org.apache.tomcat.util.http.Parameters processParameters
[WARNING] [talledLocalContainer] WARNING: Parameters: Character decoding failed. Parameter 'ì°ëñÉÞ
[WARNING] [talledLocalContainer] 8Rû£QÜPÖ◆D)ç£<%E◆<ìS
[WARNING] [├▒┌┌␊␍L⎺␌▒┌C⎺┼├▒␋┼␊⎼] ÎS·Éä:ìAQØRÜ]' ┬␋├␤ ┴▒┌┤␊ '' ␤▒⎽ ␉␊␊┼ ␋±┼⎺⎼␊␍. N⎺├␊ ├␤▒├ ├␤␊ ┼▒└␊ ▒┼␍ ┴▒┌┤␊ ─┤⎺├␊␍ ␤␊⎼␊ └▒≤ ␌⎺⎼⎼┤⎻├␊␍ ␍┤␊ ├⎺ ├␤␊ °▒␋┌␊␍ ␍␊␌⎺␍␋┼±. U⎽␊ ␍␊␉┤± ┌␊┴␊┌ ┌⎺±±␋┼± ├⎺ ⎽␊␊ ├␤␊ ⎺⎼␋±␋┼▒┌, ┼⎺┼-␌⎺⎼⎼┤⎻├␊␍ ┴▒┌┤␊⎽.A┤± 6, 2012 9:11:39 PM ⎺⎼±.▒⎻▒␌␤␊.├⎺└␌▒├.┤├␋┌.␤├├⎻.P▒⎼▒└␊├␊⎼⎽ ⎻⎼⎺␌␊⎽⎽P▒⎼▒└␊├␊⎼⎽

There was quite a bit of that in the log. Has anyone seen something similar before? Is this a malicious attack, or did the Tomcat instance just flip out/go crazy?

More practically, is there anything that I can do, for instance in a Valve or Filter, to detect requests like these and temporarily ban the originating IP address? I assume that because Tomcat is unable to decode the garbled parameter to begin with I can't just look for nonsensical parameter names and reject the request if I find one.

Edit

The same problem happened again, and this time I captured a thread-dump. It's quite long (far too long to include in its entirety, in fact):

[INFO] [talledLocalContainer] 2012-08-08 12:00:27
[INFO] [talledLocalContainer] Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode):
[INFO] [talledLocalContainer] 
[INFO] [talledLocalContainer] "http-80-44" daemon prio=10 tid=0x00007f789c0b7800 nid=0x2d6a in Object.wait() [0x00007f78972f1000]
[INFO] [talledLocalContainer]    java.lang.Thread.State: WAITING (on object monitor)
[INFO] [talledLocalContainer]   at java.lang.Object.wait(Native Method)
[INFO] [talledLocalContainer]   - waiting on <0x00000000f55f8da0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
[INFO] [talledLocalContainer]   at java.lang.Object.wait(Object.java:485)
[INFO] [talledLocalContainer]   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
[INFO] [talledLocalContainer]   - locked <0x00000000f55f8da0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
[INFO] [talledLocalContainer]   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
[INFO] [talledLocalContainer]   at java.lang.Thread.run(Thread.java:662)
[INFO] [talledLocalContainer] 
[INFO] [talledLocalContainer] "http-80-43" daemon prio=10 tid=0x00007f789c0b5800 nid=0x2d09 in Object.wait() [0x00007f78973f0000]
[INFO] [talledLocalContainer]    java.lang.Thread.State: WAITING (on object monitor)
[INFO] [talledLocalContainer]   at java.lang.Object.wait(Native Method)
[INFO] [talledLocalContainer]   - waiting on <0x00000000e07e96b8> (a com.mchange.v2.resourcepool.BasicResourcePool)
[INFO] [talledLocalContainer]   at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
[INFO] [talledLocalContainer]   at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
[INFO] [talledLocalContainer]   - locked <0x00000000e07e96b8> (a com.mchange.v2.resourcepool.BasicResourcePool)
[INFO] [talledLocalContainer]   at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
[INFO] [talledLocalContainer]   at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
[INFO] [talledLocalContainer]   at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
[INFO] [talledLocalContainer]   at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:35)
[INFO] [talledLocalContainer]   at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:417)
[INFO] [talledLocalContainer]   at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
[INFO] [talledLocalContainer]   at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
[INFO] [talledLocalContainer]   at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1538)
[INFO] [talledLocalContainer]   at org.hibernate.loader.Loader.doQuery(Loader.java:661)
[INFO] [talledLocalContainer]   at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224)
[INFO] [talledLocalContainer]   at org.hibernate.loader.Loader.doList(Loader.java:2211)
[INFO] [talledLocalContainer]   at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2095)
[INFO] [talledLocalContainer]   at org.hibernate.loader.Loader.list(Loader.java:2090)
[INFO] [talledLocalContainer]   at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:388)
[INFO] [talledLocalContainer]   at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)
[INFO] [talledLocalContainer]   at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172)
[INFO] [talledLocalContainer]   at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121)
[INFO] [talledLocalContainer]   at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
[INFO] [talledLocalContainer]   at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:80)
[INFO] [talledLocalContainer] 
[INFO] [talledLocalContainer] "http-80-42" daemon prio=10 tid=0x00007f789c0cb800 nid=0x2d08 in Object.wait() [0x00007f78974f3000]
[INFO] [talledLocalContainer]    java.lang.Thread.State: WAITING (on object monitor)
[INFO] [talledLocalContainer]   at java.lang.Object.wait(Native Method)
[INFO] [talledLocalContainer]   - waiting on <0x00000000f93e8070> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
[INFO] [talledLocalContainer]   at java.lang.Object.wait(Object.java:485)
[INFO] [talledLocalContainer]   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
[INFO] [talledLocalContainer]   - locked <0x00000000f93e8070> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
[INFO] [talledLocalContainer]   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
[INFO] [talledLocalContainer]   at java.lang.Thread.run(Thread.java:662)
[INFO] [talledLocalContainer] 
[INFO] [talledLocalContainer] "http-80-41" daemon prio=10 tid=0x00007f789c0c9800 nid=0x2d07 in Object.wait() [0x00007f78975f2000]
[INFO] [talledLocalContainer]    java.lang.Thread.State: WAITING (on object monitor)
[INFO] [talledLocalContainer]   at java.lang.Object.wait(Native Method)
[INFO] [talledLocalContainer]   - waiting on <0x00000000e07e96b8> (a com.mchange.v2.resourcepool.BasicResourcePool)
[INFO] [talledLocalContainer]   at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
[INFO] [talledLocalContainer]   at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
[INFO] [talledLocalContainer]   - locked <0x00000000e07e96b8> (a com.mchange.v2.resourcepool.BasicResourcePool)
[INFO] [talledLocalContainer]   at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
[INFO] [talledLocalContainer]   at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
[INFO] [talledLocalContainer]   at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
[INFO] [talledLocalContainer]   at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:35)
[INFO] [talledLocalContainer]   at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:417)
[INFO] [talledLocalContainer]   at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
[INFO] [talledLocalContainer]   at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
[INFO] [talledLocalContainer]   at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1538)
[INFO] [talledLocalContainer]   at org.hibernate.loader.Loader.doQuery(Loader.java:661)
[INFO] [talledLocalContainer]   at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224)
[INFO] [talledLocalContainer]   at org.hibernate.loader.Loader.loadEntity(Loader.java:1851)
[INFO] [talledLocalContainer]   at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:48)
[INFO] [talledLocalContainer]   at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:42)
[INFO] [talledLocalContainer]   at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:2992)
[INFO] [talledLocalContainer]   at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:395)
[INFO] [talledLocalContainer]   at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:375)
[INFO] [talledLocalContainer]   at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:139)
[INFO] [talledLocalContainer]   at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195)
[INFO] [talledLocalContainer]   at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103)
[INFO] [talledLocalContainer]   at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
[INFO] [talledLocalContainer]   at org.hibernate.impl.SessionImpl.get(SessionImpl.java:815)
[INFO] [talledLocalContainer]   at org.hibernate.impl.SessionImpl.get(SessionImpl.java:808)
[INFO] [talledLocalContainer]   at org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:174)
[INFO] [talledLocalContainer]   at java.lang.Thread.run(Thread.java:662)
[INFO] [talledLocalContainer] 

...

2012-08-08 12:00:31
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode):

"Thread-18" daemon prio=10 tid=0x000000004217d000 nid=0x22ff runnable [0x00007f594d4f6000]
   java.lang.Thread.State: RUNNABLE
    at java.io.FileInputStream.readBytes(Native Method)
    at java.io.FileInputStream.read(FileInputStream.java:198)
    at org.apache.tools.ant.taskdefs.StreamPumper.run(StreamPumper.java:92)
    at java.lang.Thread.run(Thread.java:662)

"Thread-17" daemon prio=10 tid=0x00000000430fd800 nid=0x22fe runnable [0x00007f594d899000]
   java.lang.Thread.State: RUNNABLE
    at java.io.FileInputStream.readBytes(Native Method)
    at java.io.FileInputStream.read(FileInputStream.java:220)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00000007da0b1be0> (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:90)
    at org.apache.tools.ant.taskdefs.StreamPumper.run(StreamPumper.java:92)
    at java.lang.Thread.run(Thread.java:662)

"process reaper" daemon prio=10 tid=0x00000000436ba800 nid=0x22fc runnable [0x00007f594d697000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.UNIXProcess.waitForProcessExit(Native Method)
    at java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    at java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)

"Thread-16" prio=10 tid=0x0000000042297000 nid=0x22fb in Object.wait() [0x00007f594d798000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007da016138> (a java.lang.UNIXProcess)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.UNIXProcess.waitFor(UNIXProcess.java:165)
    - locked <0x00000007da016138> (a java.lang.UNIXProcess)
    at org.apache.tools.ant.taskdefs.Execute.waitFor(Execute.java:551)
    at org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:482)
    at org.apache.tools.ant.taskdefs.Java.fork(Java.java:784)
    at org.apache.tools.ant.taskdefs.Java.executeJava(Java.java:211)
    at org.apache.tools.ant.taskdefs.Java.executeJava(Java.java:132)
    at org.apache.tools.ant.taskdefs.Java.execute(Java.java:105)
    at org.codehaus.cargo.container.internal.AntContainerExecutorThread.run(AntContainerExecutorThread.java:103)

"pool-1-thread-5" prio=10 tid=0x00007f5948665000 nid=0x22d1 in Object.wait() [0x00007f594db6e000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007847f3058> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
    at java.lang.Object.wait(Object.java:485)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:316)
    - locked <0x00000007847f3058> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:994)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1054)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
    at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-4" prio=10 tid=0x0000000042987800 nid=0x22cc in Object.wait() [0x00007f594dc6f000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007847f3058> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
    at java.lang.Object.wait(Object.java:485)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:316)
    - locked <0x00000007847f3058> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:994)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1054)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
    at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-3" prio=10 tid=0x00007f5948a6e000 nid=0x22cb in Object.wait() [0x00007f594dd70000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007847f3058> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
    at java.lang.Object.wait(Object.java:485)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:316)
    - locked <0x00000007847f3058> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:994)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1054)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
    at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-2" prio=10 tid=0x00007f5948918800 nid=0x22ca in Object.wait() [0x00007f594de71000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007847f3058> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
    at java.lang.Object.wait(Object.java:485)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:316)
    - locked <0x00000007847f3058> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:994)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1054)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
    at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-1" prio=10 tid=0x0000000042b4a000 nid=0x22c9 in Object.wait() [0x00007f594df72000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007847f3058> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
    at java.lang.Object.wait(Object.java:485)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:316)
    - locked <0x00000007847f3058> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:994)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1054)
    at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
    at java.lang.Thread.run(Thread.java:662)

"Low Memory Detector" daemon prio=10 tid=0x00007f5948001800 nid=0x22c3 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x0000000041fc4800 nid=0x22c2 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x0000000041fc1000 nid=0x22c1 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x0000000041fbf000 nid=0x22c0 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x0000000041f9d000 nid=0x22bf in Object.wait() [0x00007f594e732000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007848dd390> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x00000007848dd390> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x0000000041f9b000 nid=0x22be in Object.wait() [0x00007f594e833000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007848dd350> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x00000007848dd350> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x0000000041f2e800 nid=0x22b4 waiting on condition [0x00007f59541a7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.codehaus.cargo.container.spi.util.ContainerUtils.waitTillContainerIsStopped(ContainerUtils.java:88)
    at org.codehaus.cargo.maven2.ContainerStartMojo.doExecute(ContainerStartMojo.java:59)
    at org.codehaus.cargo.maven2.AbstractCargoMojo.execute(AbstractCargoMojo.java:279)
    at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490)
    at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694)
    at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandaloneGoal(DefaultLifecycleExecutor.java:569)
    at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:539)
    at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:387)
    at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:348)
    at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
    at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
    at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
    at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
    at org.codehaus.classworlds.Launcher.main(Launcher.java:375)

"VM Thread" prio=10 tid=0x0000000041f94000 nid=0x22bd runnable 

"GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000041f41800 nid=0x22b5 runnable 

"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000041f43800 nid=0x22b6 runnable 

"GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000041f45800 nid=0x22b7 runnable 

"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000041f47000 nid=0x22b8 runnable 

"GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000041f49000 nid=0x22b9 runnable 

"GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000041f4b000 nid=0x22ba runnable 

"GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000041f4c800 nid=0x22bb runnable 

"GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000041f4e800 nid=0x22bc runnable 

"VM Periodic Task Thread" prio=10 tid=0x00007f594800c000 nid=0x22c4 waiting on condition 

JNI global references: 1299

Heap
 PSYoungGen      total 276160K, used 141490K [0x00000007d64b0000, 0x00000007f7e30000, 0x0000000800000000)
  eden space 256512K, 51% used [0x00000007d64b0000,0x00000007de6d88f8,0x00000007e5f30000)
  from space 19648K, 41% used [0x00000007e5f30000,0x00000007e6734020,0x00000007e7260000)
  to   space 18688K, 0% used [0x00000007f6bf0000,0x00000007f6bf0000,0x00000007f7e30000)
 PSOldGen        total 85376K, used 28607K [0x0000000782e00000, 0x0000000788160000, 0x00000007d64b0000)
  object space 85376K, 33% used [0x0000000782e00000,0x00000007849efdf0,0x0000000788160000)
 PSPermGen       total 21248K, used 20127K [0x000000077dc00000, 0x000000077f0c0000, 0x0000000782e00000)
  object space 21248K, 94% used [0x000000077dc00000,0x000000077efa7ca0,0x000000077f0c0000)

Off the top of my head, it looks like there are an abnormal number of waiting HTTP daemon threads.

aroth
  • 54,026
  • 20
  • 135
  • 176
  • What Tomcat version are you running? If this happens again, can you take a thread dump? Do you have more than one connector configured? If so, can you try *both* connectors to see if they respond if this happens again? – Christopher Schultz Aug 08 '12 at 02:34
  • Running Tomcat 6. I think the exact version is 6.0.26. There should be just a single connector running (the server settings are more-or-less stock, except the port is set to 80). I'm using the Cargo Maven2 plugin to actually start Tomcat and deploy my webapp, if that makes any difference. – aroth Aug 08 '12 at 04:08
  • FYI Tomcat enables an AJP connector on 8009 by default as well. – Christopher Schultz Aug 08 '12 at 18:31
  • Okay, the issue reoccurred and I was able to capture a thread dump. – aroth Aug 09 '12 at 05:06
  • Those waiting HTTP threads are waiting for connections. Are you sure you haven't exhausted your JDBC connection pool? C3P0 looks like it's waiting for connections to become available. – Christopher Schultz Aug 09 '12 at 16:53

2 Answers2

4

Try below steps;

Edit conf/server.xml and find the line where the tag HTTP Connector is defined. It will look something like this, possibly with more parameters:

   <Connector port="8080"/>

Add a URIEncoding="UTF-8" property to the connector:

  <Connector port="8080" URIEncoding="UTF-8"/>

Restart Tomcat

Sai Ye Yan Naing Aye
  • 6,622
  • 12
  • 47
  • 65
  • Thanks, I'll give that a try. In my case, since I'm using Cargo and Maven2 to start the server, this setting is configured by adding `UTF-8` to my `pom.xml` file. – aroth Aug 08 '12 at 04:50
  • And good timing too, shortly after restarting the server it got hit with another round of those requests. Unfortunately even with this setting it complains: `Character decoding failed. Parameter 'ì°ÃI¡*]`3lSù'©Ð6nX\±6$8a<èè%tõü³VÆ»Hÿëe2Y«t,Ñ' with value '' has been ignored`. – aroth Aug 08 '12 at 04:57
  • @aroth remove javax.el el-api xxx in your pom.xml – Sai Ye Yan Naing Aye Aug 08 '12 at 05:10
2

You could try using the javascript function escape(parameter), before you send the request. I hope my answer is not very late and it are useful to someone.

Fernando Martinez
  • 197
  • 1
  • 2
  • 12