My app has been purring along for a month with only the occasional deadline exceeded exception. This morning, in a space of five minutes, I had 150 deadline exceeded exceptions. Now it's back to normal. I understand that this can just happen, and isn't necessarily going to be resolved by anything that I do in my code, but every single exception has appstats in the stack trace (at least the 20 I looked at most carefully). On one hand, this implicates appstats - on the other hand, I guess appstats is probably in every single stack trace that ever runs. I'm trying to determine if appstats was the problem.
I'm not an expert with filters - can anyone tell me if a stack trace like the one below implies that the code was waiting in appstats for a long time? It seems like it does mean that, but I thought maybe filters are chained together in such a way that confuses the stack trace. Also, even if it was waiting in appstats - it seems like the problem might be datastore unavailability (no problems evident on the global appengine status page).
I put asterisks at the beginning of the obvious appstats calls for quick scanning.
javax.servlet.ServletContext log: Exception while dispatching incoming RPC call
com.google.gwt.user.server.rpc.UnexpectedException: Service method 'public abstract com.activegrade.shared.dto.ResultDTO com.activegrade.client.serviceinterfaces.CourseOperationService.getCourseList(com.activegrade.shared.dto.AuthInfo)' threw an unexpected exception: com.google.apphosting.api.DeadlineExceededException: This request (1ff876185c635baf) started at 2010/11/30 14:35:47.278 UTC and was still executing at 2010/11/30 14:36:16.261 UTC.
at com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java:378)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:581)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:207)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:243)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:713)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
**at com.google.appengine.tools.appstats.AppstatsFilter.doFilter(AppstatsFilter.java:140)**
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:97)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:238)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135)
at com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:261)
at com.google.apphosting.base.RuntimePb$EvaluationRuntime$6.handleBlockingRequest(RuntimePb.java:8486)
at com.google.apphosting.base.RuntimePb$EvaluationRuntime$6.handleBlockingRequest(RuntimePb.java:8484)
at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest(BlockingApplicationHandler.java:24)
at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:418)
at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:572)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:448)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:688)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:326)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:318)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:446)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: com.google.apphosting.api.DeadlineExceededException: This request (1ff876185c635baf) started at 2010/11/30 14:35:47.278 UTC and was still executing at 2010/11/30 14:36:16.261 UTC.
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
**at com.google.appengine.tools.appstats.RecordingFuture.get(RecordingFuture.java:99)
**at com.google.appengine.tools.appstats.RecordingFuture$ExtendedFuture.get(RecordingFuture.java:44)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:42)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:42)
at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:71)
at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:32)
at com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:69)
at com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:56)
at com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:156)
at com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:65)
at org.datanucleus.store.appengine.DatastoreElementContainerStoreSpecialization.getChildren(DatastoreElementContainerStoreSpecialization.java:105)
at org.datanucleus.store.appengine.DatastoreFKListStore.listIterator(DatastoreFKListStore.java:48)
at org.datanucleus.store.mapped.scostore.AbstractListStore.listIterator(AbstractListStore.java:84)
at org.datanucleus.store.mapped.scostore.AbstractListStore.iterator(AbstractListStore.java:74)
at org.datanucleus.sco.backed.List.loadFromStore(List.java:241)
at org.datanucleus.sco.backed.List.initialise(List.java:190)
at org.datanucleus.sco.SCOUtils.newSCOInstance(SCOUtils.java:183)
at org.datanucleus.store.mapped.mapping.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:426)
at org.datanucleus.store.mapped.mapping.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:444)
at org.datanucleus.store.appengine.DatastorePersistenceHandler.runPostFetchMappingCallbacks(DatastorePersistenceHandler.java:510)
at org.datanucleus.store.appengine.DatastorePersistenceHandler.fetchObject(DatastorePersistenceHandler.java:489)
at org.datanucleus.store.appengine.query.DatastoreQuery.entityToPojo(DatastoreQuery.java:565)
at org.datanucleus.store.appengine.DatastoreRelationFieldManager.lookupOneToOneChild(DatastoreRelationFieldManager.java:320)
at org.datanucleus.store.appengine.DatastoreRelationFieldManager.fetchRelationField(DatastoreRelationFieldManager.java:274)
at org.datanucleus.store.appengine.DatastoreFieldManager.fetchObjectField(DatastoreFieldManager.java:289)
at org.datanucleus.state.AbstractStateManager.replacingObjectField(AbstractStateManager.java:1197)
at com.activegrade.server.dao.Course.jdoReplaceField(Course.java)
at com.activegrade.server.dao.Course.jdoReplaceFields(Course.java)
at org.datanucleus.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2772)
at org.datanucleus.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2791)
at org.datanucleus.store.appengine.DatastorePersistenceHandler.fetchObject(DatastorePersistenceHandler.java:480)
at org.datanucleus.store.appengine.query.DatastoreQuery.entityToPojo(DatastoreQuery.java:565)
at org.datanucleus.store.appengine.query.DatastoreQuery.entityToPojo(DatastoreQuery.java:520)
at org.datanucleus.store.appengine.query.DatastoreQuery.access$300(DatastoreQuery.java:110)
at org.datanucleus.store.appengine.query.DatastoreQuery$6.apply(DatastoreQuery.java:638)
at org.datanucleus.store.appengine.query.DatastoreQuery$6.apply(DatastoreQuery.java:630)
at org.datanucleus.store.appengine.query.LazyResult.resolveNext(LazyResul
