OS: Debian 10 Current Elastic version: 7.10.2 Attempting to upgrade to Elastic version: 7.16.2
I'm attempting to upgrade our Elastic 7.10.2 nodes to v7.16.2 to deploy the recent log4j security fixes. I've done upgrades before using the apt debian installer package. This time, I need to be able to upgrade the packages manually while offline.
The elastic v7.10.2 server I'm attempting to upgrade was originally deployed using the debian installer packages, so the config files are in /etc/elasticsearch and the elasticsearch.service is configured as usual.
Steps taken
- Downloaded the v7.16.2 tar file on linux filesystem.
- Shut down elastic
- Renamed existing /usr/share/elasticsearch to elasticsearch.old
- Untarred v7.16.2 into /usr/share/elasticsearch
- Confirmed the user 'elasticsearch' owns the new install and config directories/files
- Copied original systemd-entrypoint and elasticsearch-keystore file into new /usr/share/elasticsearch/
- Copied latest jvm.options and log4j2.properties into /etc/elasticsearch (to support the newer included JDK)
When I start the elastic service, the service goes into Starting.
systemd[1]: Starting Elasticsearch…
The main elastic log in /var/log/elasticsearch/elk7x-cluster.log shows heath as GREEN and I'm able to sign in to the main elastic URL at this point.
[2022-01-13T23:52:46,986] Cluster health status changed from [RED] to [GREEN]
I'm able to sign in to the main elastic URL at this point. Main elastic page confirms v7.16.2
{
"name" : "elk7x-node01",
"cluster_name" : "elk7x-cluster",
"cluster_uuid" : "ykh1zg-SRheUu4b50FtXoQ",
"version" : {
"number" : "7.16.2",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "2b937c44140b6559905130a8650c64dbd0879cfb",
"build_date" : "2021-12-18T19:42:46.604893745Z",
"build_snapshot" : false,
"lucene_version" : "8.10.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
However, after 75 seconds passes, the elastic log shows "stopping…"
[2022-01-13T23:53:32,753] stopping …
The elasticsearch.service never finishes starting, even though the Cluster is green and available. It reports a 'timeout' once it reaches the elasticsearch.service setting TimeoutStartSec=75
● elasticsearch.service - Elasticsearch
Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: timeout) since Thu 2022-01-13 23:04:50 UTC; 10min ago
Docs: https://www.elastic.co
Process: 16968 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=143)
Main PID: 16968 (code=exited, status=143)
Jan 13 23:03:34 elk7x-node01 systemd[1]: Starting Elasticsearch...
Jan 13 23:04:49 elk7x-node01 systemd[1]: elasticsearch.service: start operation timed out. Terminating.
Jan 13 23:04:50 elk7x-node01 systemd[1]: elasticsearch.service: Failed with result 'timeout'.
If I increase the timeout to 5 minutes, elastic will be available until the 5 minute mark. I also increased the VM RAM and forced jvm.options to 3GB, just in case.
My research shows Status=143 indicates a SIGTERM command. It appears something is issuing a graceful stop command for some reason. I've increased log4j2 for TRACE log output to see what might be causing it, but not seeing anything.
Also, I've looked at all the breaking changes between 7.10.2 and 7.16.2 and compared many files between the 2 versions, but nothing stands out as a problem.
I've spent many hours on this issue and must upgrade our servers. Any advice is appreciated.
Thanks in advance.
