8

(Android, NDK, C++, OpenGL ES)

I need a way to reliably receive the text input from a (soft)keyboard. The solution can be through Java using a NativeActivity subclass, or anything which works. At the end I need whatever text is being typed, so I can render it myself with OpenGL

Some background: Up until now I was triggering the soft keyboard by calling showSoftInput or hideSoftInputFromWindow thought JNI. This never failed so far. However, the problem is the native activity will not send all characters. Especially some unicode characters outside of ASCII range, or some motion soft keyboard won't work (AKeyEvent_getKeyCode)

It used to be possible to get some of those other unicode characters why checking for KeyEvent.ACTION_MULTIPLE and reading a string of characters. But even this won't work reliably anymore.

So far I failed to find an alternative method. I experimented with programmatically adding a EditText, but never got it to work. Even trying to add a simple Button resulted in the OpenGL view to no longer being rendered.

On iOS I worked around it by having a hiding edit box, which I simply activated to make the keyboard show up. I would then read out the edit box and use the string to render myself in OpenGL.

Pierre
  • 193
  • 1
  • 11
  • Thank you for asking, but I can't follow you. What variable are you refering to and what is the relevance to it? – Pierre Jan 15 '14 at 09:12

4 Answers4

7

I have the same issues, and I have solved it using a 'Character' event that I process separately from the InputEvent.

The problem is this: AKeyEvent_getKeyCode doesn't return the KeyCode for some softkey events, notably the expanded 'unicode/latin' characters when you hold down a key. This prevents the methods @Shammi and @eozgonul from working because the KeyEvent reconstructed on the Java side doesn't have enough information to get a unicode character.

Another issue is that the InputQueue is drained on the C++/Native side before the dispatchKeyEvent event(s) are fired. This means that the KEYDOWN/KEYUP events all fired before the Java code can process the events. (They are not interleaved).

My solution is to capture the unicode characters on the Java side by overriding dispatchKeyEvent and sending the characters to a Queue<Integer> queueLastInputCharacter = new ConcurrentLinkedQueue<Integer>();

// [JAVA]
@Override
public boolean dispatchKeyEvent (KeyEvent event)
{
    int metaState = event.getMetaState(); 
    int unichar = event.getUnicodeChar(metaState);

    // We are queuing the Unicode version of the characters for
    // sending to the app during processEvents() call.

    // We Queue the KeyDown and ActionMultiple Event UnicodeCharacters
    if(event.getAction()==KeyEvent.ACTION_DOWN){
        if(unichar != 0){
            queueLastInputCharacter.offer(Integer.valueOf(unichar));
        }
        else{
            unichar = event.getUnicodeChar(); 

            if(unichar != 0){
                queueLastInputCharacter.offer(Integer.valueOf(unichar));
            }
            else if (event.getDisplayLabel() != 0){
                String aText = new String();
                aText = "";
                aText += event.getDisplayLabel();
                queueLastInputCharacter.offer(Integer.valueOf(Character.codePointAt(aText, 0)));
            }
            else
                queueLastInputCharacter.offer(Integer.valueOf(0));
        }
    }
    else if(event.getAction()==KeyEvent.ACTION_MULTIPLE){
        unichar = (Character.codePointAt(event.getCharacters(), 0));
        queueLastInputCharacter.offer(Integer.valueOf(unichar));
    }


    return super.dispatchKeyEvent(event);
}

The concurrent queue is going to let the threads play nice together.

I have a Java side method that returns the last input character:

// [JAVA]
public int getLastUnicodeChar(){
    if(!queueLastInputCharacter.isEmpty())
        return queueLastInputCharacter.poll().intValue();
    return 0;
}

At the end of my looper code, I tacked on an extra check to see if the queue retained any unicode characters:

// [C++]
int ident;
int events;
struct android_poll_source* source;

// If not rendering, we will block 250ms waiting for events.
// If animating, we loop until all events are read, then continue
// to draw the next frame of animation.
while ((ident = ALooper_pollAll(((nv_app_status_focused(_lpApp)) ? 1 : 250),
                                NULL,
                                &events,
                                (void**)&source)) >= 0)
{
    // Process this event.
    if (source != NULL)
        source->process(_lpApp, source);

    // Check if we are exiting.  If so, dump out
    if (!nv_app_status_running(_lpApp))
        return;
}

static int modtime = 10; // let's not run on every call
if(--modtime == 0) {
    long uniChar = androidUnicodeCharFromKeyEvent();
    while (uniChar != 0) {
        KEvent kCharEvent; // Game engine event
        kCharEvent.ptkKey = K_VK_ERROR;
        kCharEvent.unicodeChar = uniChar;
        kCharEvent.character = uniChar;

        /* Send unicode char */
        kCharEvent.type = K_EVENT_UNICHAR;
        _lpPortableHandler(&kCharEvent);

        if (kCharEvent.character < 127) {
            /* Send ascii char for source compatibility as well */
            kCharEvent.type = K_EVENT_CHAR;
            _lpPortableHandler(&kCharEvent);
        }

        uniChar = androidUnicodeCharFromKeyEvent();
    }
    modtime = 10;
}

The androidUnicodeCharFromKeyEvent function is very similar to @Shammi 's GetStringFromAInputEvent method, only use CallIntMethod to return the jint.

Notes This does require modifying your engine to process character events separate from Key events. Android still has key codes like AKEYCODE_BACK or AKEYCODE_ENTER that are not character events and still need to be handled (and can be handled on the main input looper).

Editboxes, consoles, etc... Things that are expecting user input can be modified to receive a separate character event that builds the string. If you are working on multiple platforms, then you will need to generate these new character events in addition to the normal key input events.

James Poag
  • 2,320
  • 1
  • 13
  • 20
6

I hope this works for you, worked for me so far.

int GetUnicodeChar(struct android_app* app, int eventType, int keyCode, int metaState)
{
JavaVM* javaVM = app->activity->vm;
JNIEnv* jniEnv = app->activity->env;

JavaVMAttachArgs attachArgs;
attachArgs.version = JNI_VERSION_1_6;
attachArgs.name = "NativeThread";
attachArgs.group = NULL;

jint result = javaVM->AttachCurrentThread(&jniEnv, &attachArgs);
if(result == JNI_ERR)
{
    return 0;
}

jclass class_key_event = jniEnv->FindClass("android/view/KeyEvent");
int unicodeKey;

if(metaState == 0)
{
    jmethodID method_get_unicode_char = jniEnv->GetMethodID(class_key_event, "getUnicodeChar", "()I");
    jmethodID eventConstructor = jniEnv->GetMethodID(class_key_event, "<init>", "(II)V");
    jobject eventObj = jniEnv->NewObject(class_key_event, eventConstructor, eventType, keyCode);

    unicodeKey = jniEnv->CallIntMethod(eventObj, method_get_unicode_char);
}

else
{
    jmethodID method_get_unicode_char = jniEnv->GetMethodID(class_key_event, "getUnicodeChar", "(I)I");
    jmethodID eventConstructor = jniEnv->GetMethodID(class_key_event, "<init>", "(II)V");
    jobject eventObj = jniEnv->NewObject(class_key_event, eventConstructor, eventType, keyCode);

    unicodeKey = jniEnv->CallIntMethod(eventObj, method_get_unicode_char, metaState);
}

javaVM->DetachCurrentThread();

LOGI("Unicode key is: %d", unicodeKey);
return unicodeKey;
}

Just call it from your input handler, my structure is approximately as follows:

switch (AInputEvent_getType(event))
    {
        case AINPUT_EVENT_TYPE_KEY:
          switch (AKeyEvent_getAction(event))
          {
            case AKEY_EVENT_ACTION_DOWN:
              int key = AKeyEvent_getKeyCode(event);
              int metaState = AKeyEvent_getMetaState(event);
              int uniValue;
              if(metaState != 0)
                  uniValue = GetUnicodeChar(app, AKEY_EVENT_ACTION_DOWN, key, metaState);
              else
                  uniValue = GetUnicodeChar(app, AKEY_EVENT_ACTION_DOWN, key, 0);

Since you stated that you already open the soft keyboard, I don't go into that part but the code is kind of straight forward. I basically use the Java function of class KeyEvent which has GetUnicodeChar function.

eozgonul
  • 810
  • 6
  • 12
  • thank you for sharing your code. It does not solve the problem. Your code does work, in fact it is very similar to what I already do. The problem is, it won't work for all keys. Example. Touch a soft key on the keyboard and hold it, some keys will open up a selection with other keys. Here, some of the keys simply won't work. The input handler is fired with AINPUT_EVENT_TYPE_KEY, but unicode result will be "0". This all was good up until Android 4.1 and seems to have radically changed since 4.3 (something along the lines) – Pierre Jan 15 '14 at 09:08
  • This is awesome. It works, as a caveat, I tried simplifying the code to not do anything weird surrounding metaState and it seems to work the same with less code. – Charles Lohr Oct 01 '19 at 23:45
2

Eozgonul's solution worked for me. I adopted it and modified it to split the work between Java and the native side. Basically I extend NativeActivity to derive my own class which allows me to move as much as possible to Java. I also ended up passing all the data from the input event. I wanted to make sure I captured as much as possible in the created KeyEvent object.

package com.MyCompany.MyApp;

import android.os.Bundle;
import android.view.inputmethod.InputMethodManager;
import android.content.Context;
import android.view.KeyEvent;

public class MyNativeActivity extends android.app.NativeActivity
{

    // Need this for screen rotation to send configuration changed callbacks to native
    @Override
    public void onConfigurationChanged( android.content.res.Configuration newConfig )
    {
        super.onConfigurationChanged( newConfig );
    }

    public void showKeyboard()
    {
        InputMethodManager imm = ( InputMethodManager )getSystemService( Context.INPUT_METHOD_SERVICE );
        imm.showSoftInput( this.getWindow().getDecorView(), InputMethodManager.SHOW_FORCED );
    }


    public void hideKeyboard()
    {
        InputMethodManager imm = ( InputMethodManager )getSystemService( Context.INPUT_METHOD_SERVICE );
        imm.hideSoftInputFromWindow( this.getWindow().getDecorView().getWindowToken(), 0 );
    }

    public String stringFromKeyCode( long downTime, long eventTime, 
            int eventAction, int keyCode, int repeatCount, int metaState, 
            int deviceId, int scanCode, int flags, int source )
    {
        String strReturn;

        KeyEvent keyEvent = new KeyEvent( downTime, eventTime, eventAction, keyCode, repeatCount, metaState, deviceId, scanCode, flags, source );

        if ( metaState == 0 )
        {
            int unicodeChar = keyEvent.getUnicodeChar();
            if ( eventAction == KeyEvent.ACTION_MULTIPLE && unicodeChar == keyEvent.KEYCODE_UNKNOWN )
            {
                strReturn = keyEvent.getCharacters();
            }
            else
            {
                strReturn = Character.toString( ( char )unicodeChar );
            }
        }
        else
        {
            strReturn = Character.toString( ( char )( keyEvent.getUnicodeChar( metaState ) ) );
        }

        return strReturn;
    }
 }

On the native side...

std::string GetStringFromAInputEvent( android_app* pApp, AInputEvent* pInputEvent )
{
    std::string strReturn;

    JavaVM* pJavaVM = pApp->activity->vm;
    JNIEnv* pJNIEnv = pApp->activity->env;

    JavaVMAttachArgs javaVMAttachArgs;
    javaVMAttachArgs.version = JNI_VERSION_1_6;
    javaVMAttachArgs.name = "NativeThread";
    javaVMAttachArgs.group = NULL;

    jint jResult;
    jResult = pJavaVM->AttachCurrentThread( &pJNIEnv, &javaVMAttachArgs );
    if ( jResult != JNI_ERR )
    {
        // Retrieves NativeActivity.
        jobject nativeActivity = pNativeActivity->clazz;
        jclass ClassNativeActivity = pJNIEnv->GetObjectClass( nativeActivity );

        jmethodID MethodStringFromKeyCode = pJNIEnv->GetMethodID( ClassNativeActivity, "stringFromKeyCode", "(JJIIIIIIII)Ljava/lang/String;" );
        jlong jDownTime = AKeyEvent_getDownTime( pInputEvent );
        jlong jEventTime = AKeyEvent_getEventTime( pInputEvent );
        jint jEventAction = AKeyEvent_getAction( pInputEvent );
        jint jKeyCode = AKeyEvent_getKeyCode( pInputEvent );
        jint jRepeatCount = AKeyEvent_getRepeatCount( pInputEvent );
        jint jMetaState = AKeyEvent_getMetaState( pInputEvent );
        jint jDeviceID = AInputEvent_getDeviceId( pInputEvent );
        jint jScanCode = AKeyEvent_getScanCode( pInputEvent );
        jint jFlags = AKeyEvent_getFlags( pInputEvent );
        jint jSource = AInputEvent_getSource( pInputEvent );

        jstring jKeyCodeString = ( jstring )pJNIEnv->CallObjectMethod( nativeActivity, MethodStringFromKeyCode, 
            jDownTime, jEventTime, jEventAction, 
            jKeyCode, jRepeatCount, jMetaState,
            jDeviceID, jScanCode, jFlags, jSource );

        const char* keyCodeString = pJNIEnv->GetStringUTFChars( keyCodeString, nullptr );
        strReturn = std::string( keyCodeString );
        pJNIEnv->ReleaseStringUTFChars( jKeyCodeString, keyCodeString );

        // Finished with the JVM.
        pJavaVM->DetachCurrentThread();
    }

    return strReturn;
}

The 2 reasons I went with this approach..

  • Reduces code syntax complexity by moving code to java and only having you to call one jni wrapper method on the native side.

  • Java is the preferred Android language and this allows me to quickly iterate on java based solutions. Moreover most existing solutions are in java.

Shammi
  • 492
  • 5
  • 13
  • Your reply animates me to revisit my code. I noticed you are using "AKeyEvent_getEventAction", yet I do not find such a function. Either I am on an old NDK or you actually meant "AKeyEvent_getAction" – Pierre Mar 02 '17 at 16:25
  • Thanks for pointing that out. You are right. There is no AKeyInput_getEventAction. It's probably a typo when I refactored the code to fit all together in this post. – Shammi Mar 04 '17 at 01:52
0

Basically this will solve the issue.
NativeActivity override onKeyDown()

But you'll have to implement some other way than the NDK key input to get the onKeyMultiple's event.getCharacters() string into your code.

Community
  • 1
  • 1
J Decker
  • 537
  • 5
  • 9