We experienced an AWS managed opensearch data loss event, and found a completely empty system.
Using the AWS console, I reset the master user and created some of the previously available users, and restored each of the indexes matching the target index pattern from the most recent snapshot in which those indexes were available. I also restored the “.kibana” index from the same snapshot.
Data from the restored indexes are available through the API (tested using “curl”).
Logging into the Opensearch dashboard URL, with the master user, and navigating to “Stack Management / Index Patterns”, I am prompted to “Create an Index Pattern” After creating an index pattern the "discover" link continues to forward to “Stack Management / Index Patterns” and I continue to be prompted to create an index pattern.
“Stack management / Saved Objects” Notifies an “Unable to find Saved Objects” error.
If I query the API $ESURL/.kibana/_doc/_search?q=type:index-pattern. The index pattern is there with fields as expected. If I attempt to create the index pattern again, the .kibana index will then contain another document for the index pattern (according to the _search API)
What should I try next, or what other details could be helpful in diagnosing or correcting the issue ?
Is it possible to do a hard reset of the AWS opensearch Kibana system without losing the indices that we are interested in?
Would upgrading opensearch (from 1.0) be a good choice?
