1

Once I enabled encrypt, all my consul cluster failed. this is sudo systemctl consul status:

memberlist: failed to receive: No installed keys could decrypt

Nov 01 08:49:30 server-1 consul[593217]: 2021-11-01T08:49:30.031+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.7:37442
Nov 01 08:49:58 server-1 consul[593217]: 2021-11-01T08:49:58.992+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.4:41492
Nov 01 08:49:59 server-1 consul[593217]: 2021-11-01T08:49:59.882+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.12:35558
Nov 01 08:50:00 server-1 consul[593217]: 2021-11-01T08:50:00.042+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.7:37460
Nov 01 08:50:29 server-1 consul[593217]: 2021-11-01T08:50:29.004+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.4:41510
Nov 01 08:50:29 server-1 consul[593217]: 2021-11-01T08:50:29.895+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.12:35576
Nov 01 08:50:30 server-1 consul[593217]: 2021-11-01T08:50:30.056+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.7:37478
Nov 01 08:50:59 server-1 consul[593217]: 2021-11-01T08:50:59.018+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.4:41528
Nov 01 08:50:59 server-1 consul[593217]: 2021-11-01T08:50:59.909+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.12:35594
Nov 01 08:51:00 server-1 consul[593217]: 2021-11-01T08:51:00.067+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.7:37496

The ACL and TLS are commented out, I even comment encrypt in all of my clients, but still received the above output. How am I able to resolve this? Thanks.

mehdiMj
  • 101
  • 8

1 Answers1

1

You must have exactly the same encrypt settings on both types of node: server and client. Or, you must have encryption disabled everywhere. You can consult the official tutorial of step-by-step enabling encryption on the existing cluster.

UPD: Some useful commands to use one key in all nodes (both client and server).

Get keys info from all nodes

consul keyring -list

Generate new key

consul keygen

Broadcast new key to all nodes

consul keyring -install=<<KEY>>

Switch to a new key

consul keyring -use=<<KEY>>
Pavel Sapezhko
  • 661
  • 1
  • 6
  • 24
  • I actually already did that, but when I `consul keyring -list-primary` I get different results. From server 2 -> `==> Gathering installed primary encryption keys... WAN: 9w9Q79tTnLcz7y8bAus64E54vdR0j7STRzPDWDxMs34= [1/1] dc1 (LAN): nrbQkQigyUL0ME6lnJKZhZuDycwx56FjCeUOJgHGiFg= [1/1] `. But from server 3 I get `==> Gathering installed primary encryption keys... WAN: 9w9Q79tTnLcz7y8bAus64E54vdR0j7STRzPDWDxMs34= [2/2] dc1 (LAN): 9w9Q79tTnLcz7y8bAus64E54vdR0j7STRzPDWDxMs34= [2/2] `. I even tried manually removing `/consul/serf/local.keyring` file, but didn't work either. – mehdiMj Nov 01 '21 at 11:55
  • I also did `consul keyring -remove` my previous token `9w9Q79tTnLcz7y8bAus64E54vdR0j7STRzPDWDxMs34=` but did not allowed `==> Removing gossip encryption key... error: Unexpected response code: 500 (6 errors occurred: * WAN error: 2/2 nodes reported failure * server-1.dc1: Removing the primary key is not allowed * server-3.dc1: Removing the primary key is not allowed * dc1 (LAN) error: 2/2 nodes reported failure * server-1: Removing the primary key is not allowed * server-3: Removing the primary key is not allowed ` and docs didn't help. – mehdiMj Nov 01 '21 at 12:07
  • You should use an equal encryption key for every DC. Try to set an equal key everywhere by using `consul keyring -use` command. – Pavel Sapezhko Nov 01 '21 at 12:09
  • Thnaks, I changed the key on `/etc/consul.d/consul.hcl` and restarted the `consul.service` But now when I try `consul keyring -use=9w9Q79tTnLcz7y8bAus64E54vdR0j7STRzPDWDxMs34=` I get the following error `==> Changing primary gossip encryption key... error: Unexpected response code: 500 (2 errors occurred: * dc1 (LAN) error: 1/1 nodes reported failure * server-2: Requested key is not in the keyring ` – mehdiMj Nov 01 '21 at 12:24
  • Execute `consul keyring -install=9w9Q79tTnLcz7y8bAus64E54vdR0j7STRzPDWDxMs34=` to broadcast key to all nodes. Then try `-use` – Pavel Sapezhko Nov 01 '21 at 12:27
  • I think, I should read more documentations :) I accept your answer as `solved`. please edit and complete it. Thank you for your time and knowledge. – mehdiMj Nov 01 '21 at 12:31