How to stop pynput listener

Question

I'm working on a vim clone made in python, I use pynput to keep track of the current mode (insert, normal), so I'd like so when I press i it switched to insert mode, so i decided to use pynput, to keep track of the inputs, for example, I'd like it so that when i press l, the writing cursor goes right, and h make it go left (the index variable), but when I switch back to normal mode and close the program, this error shows up: <Xlib.error.ConnectionClosedError: Display connection closed by client>, Here's the code:

# Entering writing mode...
def write_mode():
    with Listener(on_press=on_press) as listener :                    
        listener.join()
        listener.stop()

index = 0
is_writing_mode = False


def on_press(key):
    global index
    global is_writing_mode
    if str(key) == "'l'":
        index += 1
        print(index)
    if str(key) == "'h'":
        index -= 1
        print(index)
        
    if str(key) == "'q'":
        is_writing_mode = False
        print("Normal mode")
        line_return()

The thig is, i don't know why it does that, cuz i can exit insert mode, but the aftermath of closing the program shows this error and i don't know what it means