Enabling option-key shortcuts in MATLAB for Mac

Question

Since R2009b, MATLAB has had marvelously customizable keyboard shortcuts through its Keyboard Shortcuts Preferences. This works very well for customizing shortcuts using command and control on a Mac.

Unfortunately, those keybindings seem to be unable to override MATLAB's built-in character map. For example, if I define option-f as cursor-next-word (a la emacs), it accepts the binding. Hitting the key combination does properly move the cursor to the next word, but it additionally prints the ƒ character! I believe this is from the character map (perhaps as opposed to the input map?). Neither EditorMacro nor KeyBindings are able to override this behavior.

I stumbled across this answer from a tangentially related question which gives me hope. In short, he defined a Java class that can handle keyboard events and replace them with other keystroke input. The solution, however, only works as prescribed on Windows. The following modifications were required to run on a Mac:

I needed to change the keycodes to remap to have 'pressed' in the string, like so:

map = {
    '$' '^'
    '#' char(181)  % might be useful for text formatting
};

to:

map = {
    'alt pressed F' '^'
    'alt pressed B' char(181)  % might be useful for text formatting
};

Unfortunately, after running the code, pressing option-f yields cursor-next-word and the ƒ character, just like before. However, if I disable the cursor-next-word binding from the preferences, then I get both ƒ and ^! Indeed, even if I use a simple action like pressed F, the KeyReplacementAction doesn't replace the action but rather augments it. It seems like this behavior is unique to MATLAB on OS X.

It seems as though I'm simply not overriding the correct keymap. I've tried digging through the Java runtime, but I'm not familiar enough with the event dispatch model to know where to look next. Perhaps something within Java's OS-level keymap?

---

Edit: I've since done some more digging around. It appears as though the Mac version of MATLAB does not properly respect the 'consumed' property of a keyEvent. I can attach the KeyReplacementAction to either the inputMap or the keymap, and in both cases I augment the keybinding instead of replacing it. I used reflection to 'unprotect' the consume() method for AWTEvents, but the effect was the same as before.

Following the stack trace around, it appears as though the keyEvent is falling through to an instance of javax.swing.KeyboardManager. It looks like I should be able to unbind keystrokes within the KeyboardManager, but I cannot figure out how to access the instance from the MATLAB handles I have. Perhaps someone more familiar with Swing's event model and the Java debugger could get farther.

---

Edit 2: flolo's answer spurred me to look into X11's keymaps. Several notes:

• Matlab does not seem to respect ~/.Xmodmap or any currently-loaded modmaps.
• Matlab makes use of the $XKEYSYMDB environment variable if it exists at startup. Otherwise, it loads it from $MATLAB/X11/app-defaults/XKeysymDB.
• The whole $MATLAB/X11/app-defaults/ directory looks very interesting; perhaps some hackery there could make this work?
• Where are the X11 keymaps on a Mac? How does MATLAB switch to international keyboard layouts?

---

Edit 3: Hrm, I think X11 is a red herring. lsof -c MATLAB shows that it is accessing /System/Library/Keyboard Layouts/AppleKeyboardLayouts.bundle. Working on that now…

---

Edit 4: MATLAB does indeed use the system keyboard layout. I created one without any bindings as R.M. suggested. This appeared to work — MATLAB does behave properly. Unfortunately, it also breaks my custom Cocoa keybindings in all other programs. Close, but no cigar. (Close enough, in fact, that R.M. won the +500 bounty due to a brief thought that it had worked… until I attempted to compose my congratulatory comment and discovered that I couldn't navigate the text field as usual.)

Answer 1

I finally had a chance to further pursue this over the holidays. And I found a kludge of a solution. Note that this is dynamically changing the Java classes used by the Matlab GUI in runtime; it is completely unsupported and is likely to be very fragile. It works on my version of Matlab (r2011a) on OS X Lion.

Here's what I've learned about how Swing/Matlab process keypress events:

1. A keystroke is pressed.
2. The active text component's inputMap is searched to see if there's a binding for that keystroke.
   • If there is an action bound to that keystroke, dispatch that action's actionPerformed method
   • If there is a string associated with that keystroke, find the action from the text component's actionMap, and then dispatch that action's actionPerformed method
3. No matter what, as a final step, dispatch the action found within the text component's Keymap.getDefaultAction(). Here's where the problem lies.

This solution overrides the Keymap's default action with a wrapper that simply checks to see if any modifier keys were pressed. If they were, the action is silently ignored.

---

Step 1: Create a custom TextAction in Java to ignore modifier keys

import javax.swing.text.TextAction;
import javax.swing.Action;
import java.awt.event.ActionEvent;

public class IgnoreModifiedKeystrokesAction extends TextAction
{
    private static final int ignoredModifiersMask = 
        ActionEvent.CTRL_MASK | ActionEvent.ALT_MASK;
    private Action original;

    public IgnoreModifiedKeystrokesAction(Action act)
    {
        super((String)act.getValue("Name"));
        original = act;
    }

    public void actionPerformed(ActionEvent e)
    {
        if ((e.getModifiers() & ignoredModifiersMask) == 0) {
            /* Only dispatch the original action if no modifiers were used */
            original.actionPerformed(e);
        }
    }

    public Action getOriginalAction()
    {
        return original;
    }
}

Compile to a .jar:

javac IgnoreModifiedKeystrokesAction.java && jar cvf IgnoreModifiedKeystrokesAction.jar IgnoreModifiedKeystrokesAction.class

---

Step 2: Override MATLAB's default Keymap handler in both the command window and editor (from within MATLAB)

The hardest part here is getting the java handles to the command window and editor. It is dependent upon the layout and classnames of the individual editor panes. This may change between versions of Matlab.

javaaddpath('/path/to/IgnoreModifiedKeystrokesAction.jar')
cmdwin = getCommandWindow();
editor = getEditor();

for t = [cmdwin,editor]
    defaultAction = t.getKeymap().getDefaultAction();
    if ~strcmp(defaultAction.class(),'IgnoreModifiedKeystrokesAction')
        newAction = IgnoreModifiedKeystrokesAction(defaultAction);
        t.getKeymap().setDefaultAction(newAction);
    end
end

%% Subfunctions to retrieve handles to the java text pane elements
function cw = getCommandWindow()
    try
        cw = handle(com.mathworks.mde.desk.MLDesktop.getInstance.getClient('Command Window').getComponent(0).getComponent(0).getComponent(0),'CallbackProperties');
        assert(strcmp(cw.class(),'javahandle_withcallbacks.com.mathworks.mde.cmdwin.XCmdWndView'));
    catch %#ok<CTCH>
        cw_client = com.mathworks.mde.desk.MLDesktop.getInstance.getClient('Command Window');
        cw = searchChildComponentsForClass(cw_client,'com.mathworks.mde.cmdwin.XCmdWndView');
    end
    if isempty(cw)
        error('Unable to find the Command Window');
    end
end

function ed = getEditor()
    try
        ed = handle(com.mathworks.mde.desk.MLDesktop.getInstance.getGroupContainer('Editor').getComponent(1).getComponent(0).getComponent(0).getComponent(0).getComponent(1).getComponent(0).getComponent(0).getComponent(0).getComponent(0).getComponent(1).getComponent(0).getComponent(0),'CallbackProperties');
        assert(strcmp(ed.class(),'javahandle_withcallbacks.com.mathworks.mde.editor.EditorSyntaxTextPane'));
    catch %#ok<CTCH>
        ed_group = com.mathworks.mde.desk.MLDesktop.getInstance.getGroupContainer('Editor');
        ed = searchChildComponentsForClass(ed_group,'com.mathworks.mde.editor.EditorSyntaxTextPane');
        % TODO: When in split pane mode, there are two editor panes. Do I need
        % to change actionMaps/inputMaps/Keymaps on both panes?
    end
    if isempty(ed)
        error('Unable to find the Editor Window');
    end
end

function obj = searchChildComponentsForClass(parent,classname)
    % Search Java parent object for a child component with the specified classname
    obj = [];
    if ~ismethod(parent,'getComponentCount') || ~ismethod(parent,'getComponent')
        return
    end
    for i=0:parent.getComponentCount()-1
        child = parent.getComponent(i);
        if strcmp(child.class(),classname)
            obj = child;
        else
            obj = searchChildComponentsForClass(child,classname);
        end
        if ~isempty(obj)
            obj = handle(obj,'CallbackProperties');
            break
        end
    end
end

Now it's possible to define keybindings in the standard preferences window that use the option key!

---

Step 3 (optional): Remove the custom action

cmdwin = getCommandWindow();
editor = getEditor();

for t = [cmdwin,editor]
    defaultAction = t.getKeymap().getDefaultAction();
    if strcmp(defaultAction.class(),'IgnoreModifiedKeystrokesAction')
        oldAction = defaultAction.getOriginalAction();
        t.getKeymap().setDefaultAction(oldAction);
    end
end
javarmpath('/path/to/IgnoreModifiedKeystrokesAction.jar')

Answer 2

I don't have a complete solution, but I have some possible approaches you can try, if you haven't looked at them yet.

MATLAB's keyboard shorcuts are saved in an XML file in /Users/$user/.matlab/$version/$name_keybindings.xml, where $user is your user name, $version the version of MATLAB and $name is whatever you've saved the keybindings as. It looks something like this

<?xml version="1.0" encoding="utf-8"?>
<CustomKeySet derivedfrom="Mac" modifieddefault="false">
   <Context id="Global">
      <Action id="eval-file">
         <Stroke alt="on" code="VK_ENTER" meta="on" sysctrl="on"/>
      </Action>
      <stuff here /stuff>
   </Context>
</CustomKeySet>

I tried changing the derivedfrom value to EmptyBaseSet to see what happens. As expected, none of the shortcuts work, but Opt-character still reproduced a unicode character. This seems indicates that the Opt-f or any option based shortcut behaviour is due to Mac, and out of MATLAB's hands. It's the Windows equivalent of pressing Alt+numpad keys for unicode characters. If MATLAB knew about the option shortcuts, it would indicate a possible conflict in MATLAB>Preferences>Keyboard>Shortcuts, but it doesn't. I don't know enough XML to tell you whether or not you can disable Opt-f=ƒ by editing this file.

My guess is that there is a very high probability that Apple has ensured that applications don't get to tinker very much with this feature, as several non-English (German/Slavic/Latin) languages use these keyboard shortcuts very often.

An alternate option is to try Ukelele, which is a keyboard layout editor for macs. There have been questions on S.O. and related sites, where they've used Ukelele to remap dead keys,another example for dead keys, configure the left & right Cmd differently, swapping € and $, etc. You can try redefining your keyboard layout to disable Opt-f (unless if you need that particular character outside of MATLAB), which should solve the problem.

Lastly, I don't mean to be saying "you should be doing something else" when the question is "how do I do this?", but in general, with macs, I've found mapping Ctrl to the windows/emacs Alt shortcuts to be easier than Opt. Mostly, for the very same reasons, and also because Opt is so damn close to the Cmd key on my laptop that my fat fingers end up pressing Cmd when I don't mean to (it never happens the other way around).

Answer 3

As request an ugly workaround: I am not working on MacOS, so I am entering the realms of "maybe a good tip": Under X exist tools which can capture X-events and execute commands on demand, a quick google gave that e.g. xkbevd also exist under MacOS (and I assume the java tool you mentioned does something similar). There you could maybe catch the ctrl-f and replace it with ctrl-f backspace so you negate the additional character. (Very ugly, and it breaks those codes outside matlab).

Another option in case you dont want to change the behaviour on the commandline, but just in the editor - use external editor: you can define in the preferences another, different from the default. There you can chose emacs, vi, or whatever suits you (and where the remapping works)

BTW: Under Linux this problem doesnt exist with matlab, so it looks like its really MacOS specific.