I have an AWS OpenSearch domain with ultrawarm/cold storage disabled.
My application/error log is spammed (new entries every 2-5 minutes) with the following error
[2022-09-05T09:29:13,909][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [eaab83ab8ab8fd53a42b42e694708350] uncaught exception in thread [DefaultDispatcher-worker-1]
OpenSearchStatusException[Rejecting request because cold storage is not enabled on the domain. Enabling cold storage for the first time can take several hours. Please try again later.]
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:195)
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:120)
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:193)
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99)
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:193)
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:266)
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154)
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:193)
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:170)
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:98)
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108)
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:95)
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433)
at org.opensearch.indexmanagement.kraken.ColdIndexMetadataService$getMetadata$response$1.invoke(ColdIndexMetadataService.kt:31)
at org.opensearch.indexmanagement.kraken.ColdIndexMetadataService$getMetadata$response$1.invoke(ColdIndexMetadataService.kt:17)
at org.opensearch.indexmanagement.kraken.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:30)
at org.opensearch.indexmanagement.kraken.ColdIndexMetadataService.getMetadata(ColdIndexMetadataService.kt:31)
at org.opensearch.indexmanagement.indexstatemanagement.IndexMetadataProvider.getISMIndexMetadataByType(IndexMetadataProvider.kt:46)
at org.opensearch.indexmanagement.indexstatemanagement.IndexMetadataProvider$getMultiTypeISMIndexMetadata$2$invokeSuspend$$inlined$forEach$lambda$1.invokeSuspend(IndexMetadataProvider.kt:66)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
I've tried stopping all applications that interacts with OpenSearch, tried to delete my ISM policy, but these errors keep filling the logs
any ideas?
