We've run out of options already. We've tried out clearing the cache (alot of times already), manually terminate workflows that are in progress, with errors, failed to start. We've also tried to restore our site collection. Refreshed jobs through stsadm and run execadmsvcjobs. We found this hotfix - https://www.microsoft.com/en-us/download/details.aspx?id=21066 but we weren't able to install that in MS Win Server 2003 R2. Added Heapsettings.
After we've done all of those, these are the log entries that are recurring:
XX heaps created, above warning threshold of 32. Check for excessive SPWeb or SPSite usage.
The previous instance of the timer job 'Config Refresh', id '{08A3BB10-8FA2-478C-9CCB-F8415F6D7485}' for service '{298FBFD4-A717-466C-A270-0AF3B6CC2D6C}' is still running, so the current instance will be skipped. Consider increasing the interval between jobs.
The previous instance of the timer job 'Immediate Alerts', id '{BA39B9DF-AB1C-466F-9FCD-4E21AE5806DC}' for service '{EC394CD5-4AFA-4DB9-80CC-1F6792251B30}' is still running, so the current instance will be skipped. Consider increasing the interval between jobs.
The previous instance of the timer job 'Workflow', id '{A16329F3-D83D-4E6F-8A05-4A33876AB5F4}' for service '{EC394CD5-4AFA-4DB9-80CC-1F6792251B30}' is still running, so the current instance will be skipped. Consider increasing the interval between jobs.
Potentially excessive number of SPRequest objects (9) currently unreleased on thread 6. Ensure that this object or its parent (such as an SPWeb or SPSite) is being properly disposed. This object is holding on to a separate native heap. Allocation Id for this object: {6E418B9A-AF0C-4FA1-9871-1028CE638F6F} Stack trace of current allocation: at Microsoft.SharePoint.SPRequestManager.Add(SPRequest request, Boolean shareable) at Microsoft.SharePoint.SPGlobal.CreateSPRequestAndSetIdentity(Boolean bNotGlobalAdminCode, String strUrl, Boolean bNotAddToContext, Byte[] UserToken, String userName, Boolean bIgnoreTokenTimeout, Boolean bAsAnonymous) at Microsoft.SharePoint.SPSite.GetSPRequest() at Microsoft.SharePoint.SPSite.get_Request() at Microsoft.SharePoint.SPSite.OpenWeb(Guid gWebId, Int32 mondoHint) at Microsoft.SharePoint.SPSite.OpenWeb(Guid gWebId) at Microsoft.SharePoint.Workflow.SPWinOeWorkflow.get_InitiatorWeb() at Microsoft.SharePoint.Workflow.SPWinOEWSSService.GetWebForListItemService() at Microsoft.SharePoint.Workflow.SPWinOEWSSService.UpdateListItem(Guid id, Guid listId, Int32 itemId, Hashtable itemProperties) at Microsoft.SharePoint.WorkflowActions.ActivityHelper.DoCorrectUpdateMethod(WorkflowContext theContext, Int32 item, Guid listId, Hashtable properties, IListItemService hostInterface) at Microsoft.SharePoint.WorkflowActions.UpdateItemActivity.Execute(ActivityExecutionContext provider) at System.Workflow.ComponentModel.ActivityExecutor´1.Execute(T activity, ActivityExecutionContext executionContext) at System.Workflow.ComponentModel.ActivityExecutor´1.Execute(Activity activity, ActivityExecutionContext executionContext) at System.Workflow.ComponentModel.ActivityExecutorOperation.Run(IWorkflowCoreRuntime workflowCoreRuntime) at System.Workflow.Runtime.Scheduler.Run() at System.Workflow.Runtime.WorkflowExecutor.RunScheduler() at System.Workflow.Runtime.WorkflowExecutor.RunSome(Object ignored) at System.Workflow.Runtime.Hosting.DefaultWorkflowSchedulerService.WorkItem.Invoke(WorkflowSchedulerService service) at System.Workflow.Runtime.Hosting.DefaultWorkflowSchedulerService.QueueWorkerProcess(Object state) at System.Threading.ExecutionContext.runTryCode(Object userData) at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack) at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)
ERROR: request not found in the TrackedRequests. We might be creating and closing webs on different threads. ThreadId = 9, Free call stack = at Microsoft.SharePoint.SPRequestManager.Release(SPRequest request) at Microsoft.SharePoint.SPSite.Close() at Microsoft.SharePoint.SPSite.Dispose() at Microsoft.SharePoint.Workflow.SPWorkflowAutostartEventReceiver.AutoStartWorkflow(SPItemEventProperties properties, Boolean bCreate, Boolean bChange, AssocType atyp) at Microsoft.SharePoint.Workflow.SPWorkflowAutostartEventReceiver.AutoStartWorkflow(SPItemEventProperties properties, Boolean bCreate, Boolean bChange) at Microsoft.SharePoint.Workflow.SPWorkflowAutostartEventReceiver.ItemUpdated(SPItemEventProperties properties) at Microsoft.SharePoint.SPEventManager.RunItemEventReceiver(SPItemEventReceiver receiver, SPItemEventProperties properties, SPEventContext context, String receiverData) at Microsoft.SharePoint.SPEventManager.RunItemEventReceiverHelper(Object receiver, Object properties, SPEventContext context, String receiverData) at Microsoft.SharePoint.SPEventManager.<>c__DisplayClass8`1.b__0() at Microsoft.SharePoint.SPSecurity.CodeToRunElevatedWrapper(Object state) at Microsoft.SharePoint.SPSecurity.RunAsUser(SPUserToken userToken, Boolean bResetContext, WaitCallback code, Object param) at Microsoft.SharePoint.SPSecurity.RunAsUser(SPUserToken userToken, CodeToRunElevated code) at Microsoft.SharePoint.SPEventManager.InvokeEventReceivers[ReceiverType](SPUserToken userToken, RunEventReceiver runEventReceiver, Object receivers, Object properties, Boolean checkCancel) at Microsoft.SharePoint.SPEventManager.InvokeEventReceivers[ReceiverType](Byte[] userTokenBytes, RunEventReceiver runEventReceiver, Object receivers, Object properties, Boolean checkCancel) at Microsoft.SharePoint.SPEventManager.HandleEventCallback[ReceiverType,PropertiesType](Object callbackData) at Microsoft.SharePoint.Utilities.SPThreadPool.WaitCallbackWrapper(Object state) at System.Threading.ExecutionContext.runTryCode(Object userData) at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack) at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state) , Allocation call stack (if present) null Now we're going to try to remove all workflows except for one to narrow down our troubleshooting.
Please help guys. Thank you!