I've been using Vim for quite a while and it supports using the mouse through an terminal. Nothing too radical, but you can click on the screen to navigate to where the mouse click happened, and double click to select a word for replacement and use the scroll.
I'm using a Mac and have tried turning on xterm-mouse-mode but it does nothing apart from complain that mouse-1 is undefined.
I don't think this should be that hard, it looks like the escape sequences can be detected. I just hope that someone must have done this already.