elasticsearch state changed to red

Question

I am trying to reset elastic user password. while resting password with bin/elasticsearch-reset-password -u elastic command, my cluster status changed from yellow state to red state. all master nodes are up and running and joined in cluster.

**[2023-05-06T19:20:43,954][INFO ][o.e.x.s.a.RealmsAuthenticator] [master2] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]**
**[2023-05-06T19:21:05,109][INFO ][o.e.x.s.a.f.FileUserPasswdStore] [master2] users file [/etc/elasticsearch/users] changed. updating users...**
**[2023-05-06T19:21:05,112][INFO ][o.e.x.s.a.f.FileUserRolesStore] [master2] users roles file [/etc/elasticsearch/users_roles] changed. updating users roles...**
**[2023-05-06T19:21:09,058][INFO ][o.e.x.s.s.SecurityIndexManager] [master2] security index does not exist, creating [.security-7] with alias [.security]**
**[2023-05-06T19:21:09,128][INFO ][o.e.c.m.MetadataCreateIndexService] [master2] [.security-7] creating index, cause [api], templates [], shards [1]/[0]**
**[2023-05-06T19:21:09,143][INFO ][o.e.c.r.a.AllocationService] [master2] current.health="RED" message="Cluster health status changed from [YELLOW] to [RED] (reason: [index [.security-7] created])." previous.health="YELLOW" reason="index [.security-7] created"**
**[2023-05-06T19:21:45,130][INFO ][o.e.x.s.a.f.FileUserPasswdStore] [master2] users file [/etc/elasticsearch/users] changed. updating users...**
**[2023-05-06T19:21:45,131][INFO ][o.e.x.s.a.f.FileUserRolesStore] [master2] users roles file [/etc/elasticsearch/users_roles] changed. updating users roles...
ers roles...
[2023-05-06T19:22:39,342][WARN ][r.suppressed             ] [master2] path: /_security/user/elastic/_password, params: {pretty=, username=elastic}
org.elasticsearch.action.UnavailableShardsException: [.security-7][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update
        at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction.retry(TransportInstanceSingleOperationAction.java:226) ~[elasticsearch-8.7.1.jar:?]
        at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction.doStart(TransportInstanceSingleOperationAction.java:181) ~[elasticsearch-8.7.1.jar:?]
        at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$2.onTimeout(TransportInstanceSingleOperationAction.java:249) ~[elasticsearch-8.7.1.jar:?]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:326) ~[elasticsearch-8.7.1.jar:?]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:264) ~[elasticsearch-8.7.1.jar:?]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:643) ~[elasticsearch-8.7.1.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:891) ~[elasticsearch-8.7.1.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1623) ~[?:?]**

score 0 · Answer 1 · answered May 07 '23 at 10:27

0

These logs entries show several issues with authentication, changes in user configuration files, timeout error during an update process. I would advice to delete the Elasticsearch and redownload, but before running it elasticsearch.bat, make the security policy inside the elatsicsearch.yml file false, then start the Elasticsearch, it should work fine.

# Disable security features
xpack.security.enabled: false

xpack.security.enrollment.enabled: false

# Disable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

# Disable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: false
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

answered May 07 '23 at 10:27

Farkhod Abdukodirov

877
7
24

hello, @farkhod , I have 3 master node in my elasticsearch cluster . should I disable security in all node before initiate cluster ? also can I enable ssl in elasticsearch.yml to secure my cluster after setting passwords ? – Anonym420 May 07 '23 at 11:11
@Anonym420 Hi there, Remember! disabling the security configuration only advised in case you are in your home network, or safe network. In my suggestion, I wanted to say to check by disabling and run the ES followed by `bin/elasticsearch-setup-passwords interactive` command to set new passwords, then you can enable the security functions as well. Try by disabling only few of the security configurations at first. If doesn't work, then you have to disable all security functions. – Farkhod Abdukodirov May 07 '23 at 23:05

elasticsearch state changed to red

1 Answers1