I have one domain configured on AWS OpenSearch, 3 days ago it starts failing returning HTTP Status code 503 for some requests. There is no clear pattern. A request X sometimes return 503 and sometimes return 200.
The error logs of AWS are constantly logging:
[2021-10-21T20:34:58,046][WARN ][o.e.d.z.ZenDiscovery ] [LV0sigP] not enough master nodes discovered during pinging (found [[Candidate{node={LV0sigP}{XXXXX}{XXXXXXXX}{__IP__}{__IP__}{distributed_snapshot_deletion_enabled=true, __AMAZON_INTERNAL__, __AMAZON_INTERNAL__}, clusterStateVersion=4322152}]], but needed [2]), pinging again
I didn't changed anything on my AWS OpenSearch config, neither on the client, so it seems like an internal problem on AWS. Does anyone any advice?