In Evil, The default binding of ESC in insert mode is (evil-normal-state nil).
I want to rebind ESC in insert mode like this:
(define-key evil-insert-state-map (kbd "ESC")
(lambda () (interactive) (message "hello")))
However, when I try to do this I get unexpected behavior. First, the binding does not change. And for a reason I don't understand it also breaks M-x in insert mode.
Why does this happen?
This is likely related to the fact that ESC is an ASCII character which is used in all kinds of "escape sequences" and which Emacs normally considers as an alternative to the Meta modifier.
IIRC Evil handles this issue by mapping the ESC key not to the ESC character (code 27), as done by default in Emacs, but to the escape event. So you might like to try:
(define-key evil-insert-state-map [escape]
(lambda () (interactive) (message "hello")))
