EyeGesture and EyeGestureManager clarity needed

Question

Google review team requires glassewares to:

Dim the screen if there isn't an expectation that a user is looking at it.

This is consistent with the "in the here and now" experience of Glass. Glassware should always dim the screen if there isn't an expectation that a user is looking at it. Ideally it behaves like a timeline and dims after 15s. A user can 'rewake' the screen by looking up.

Update to be made: If a user is not looking at the results set in the card scroller, dim the screen.

This hints at using the EyeGesture, which doesn't seem to be mentioned anywhere on the Glass Develop Page.

After some searching I found this EyeGesture library (github) that from this stackoverflow post (Google Glass Eye Gesture Crashing (EyeGestureLib)) doesn't seem to work anymore (and hasn't been updated in 4 months+).

The accepted answer (from the stackoverflow post) proposed using this revised EyeGesture library (github)

It was also mentioned (in the stackoverflow post - as a comment ) that:

Basically, you're trying to expose classes that exist in the Glass environment itself, but not through the official APIs. By declaring these stub classes (none of the methods are implemented) and by putting them into the com.google.android.glass.eye package, we're allowing our code to compile with these unimplemented classes. At runtime, the system has implementations of those classes and the application will instead use the system's implementations.

Here are my following questions:

1. Will there be (and when) an offcial API for EyeGesture's any time soon?
2. I tried Implementing the revised EyeGesture library into my activity by following the guide proposed without any luck. What could I be doing wrong?
3. Is there something I'm missing for it to be detected? I know that with the GestureDetector I'm required to Override the onGenericMotionEvent(MotionEvent event), is there something similar for the EyeGesture?

Here is what I'm currently doing:

I have a package named com.google.android.glass and in this package I have the following:

• EyeGesture enum that implements Parcelable
• EyeGestureManager class

I have in the main package:

• GestureIds class (This one is different the github in that it's a public class and not private)

In my activity I have:

private void createEyeGestureDetector(ResultActivity resultActivity) {
    final GestureIds gestureIds = new GestureIds();
    //The github guide didn't mention any class names for 
    //mEyeGestureManager and mEyeGestureListener .. so I added some..
    EyeGestureManager mEyeGestureManager = EyeGestureManager.from(resultActivity);
    EyeGestureManager.Listener mEyeGestureListener = new EyeGestureManager.Listener() {
        @Override
        public void onDetected(EyeGesture gesture) {
            Log.i("EyeGestureListener", "Gesture: " + gesture.getId());
            int id = gesture.getId();
            if(id == gestureIds.WINK_ID || id == gestureIds.DOUBLE_WINK_ID) {
                Log.d("EyeGesture", "Wink");
            } else if (id == gestureIds.BLINK_ID || id == gestureIds.DOUBLE_BLINK_ID){
                Log.d("EyeGesture", "Blink");
            } else if (id == gestureIds.LOOK_AT_SCREEN_ID || id == gestureIds.LOOK_AWAY_FROM_SCREEN_ID) {
                Log.d("EyeGesture", "Screen");
            }

        }
    };
}

In my onCreate I have:

//..
super.onCreate(bundle);
createEyeGestureDetector(this);
//..

Update Logcat:

When I do:

for (EyeGesture eg : EyeGesture.values()) {
    boolean supported = mEyeGestureManager.isSupported(eg);
    Log.w("yupyup", eg.name() + ":" + supported);
}

I get:

12-10 18:40:51.252    2405-2405/com.google.android.glass.websurg.websurg W/yupyup﹕ WINK:true
12-10 18:40:51.252    2405-2405/com.google.android.glass.websurg.websurg W/yupyup﹕ DOUBLE_WINK:false
12-10 18:40:51.252    2405-2405/com.google.android.glass.websurg.websurg W/yupyup﹕ BLINK:false
12-10 18:40:51.252    2405-2405/com.google.android.glass.websurg.websurg W/yupyup﹕ DOUBLE_BLINK:true
12-10 18:40:51.260    2405-2405/com.google.android.glass.websurg.websurg W/yupyup﹕ DON:true
12-10 18:40:51.268    2405-2405/com.google.android.glass.websurg.websurg W/yupyup﹕ DOFF:true
12-10 18:40:51.268    2405-2405/com.google.android.glass.websurg.websurg W/yupyup﹕ LOOK_AT_SCREEN:true
12-10 18:40:51.268    2405-2405/com.google.android.glass.websurg.websurg W/yupyup﹕ LOOK_AWAY_FROM_SCREEN:false

I also added (from the first github link):

@Override
protected void onStart(){
    super.onStart();
    createEyeGestureDetector(this);
    for (EyeGesture eg : EyeGesture.values()) {
        boolean supported = mEyeGestureManager.isSupported(eg);
        Log.w("yupyup", eg.name() + ":" + supported);
    }
    mEyeGestureManager.register(EyeGesture.LOOK_AT_SCREEN, mEyeGestureListener);
    mEyeGestureManager.register(EyeGesture.LOOK_AWAY_FROM_SCREEN, mEyeGestureListener);
    mEyeGestureManager.register(EyeGesture.WINK, mEyeGestureListener);

}

and

@Override
protected void onStop(){
    mEyeGestureManager.unregister(EyeGesture.LOOK_AT_SCREEN, mEyeGestureListener);
    mEyeGestureManager.unregister(EyeGesture.LOOK_AWAY_FROM_SCREEN, mEyeGestureListener);
    mEyeGestureManager.unregister(EyeGesture.WINK, mEyeGestureListener);
    super.onStop();
}

This gives me:

12-10 18:46:11.314    2553-2553/com.google.android.glass.websurg.websurg I/EyeGestureManager﹕ Removing listener: com.google.android.glass.websurg.websurg.ResultActivity$1@41b8b908 for eye gesture: LOOK_AT_SCREEN
12-10 18:46:11.314    2553-2553/com.google.android.glass.websurg.websurg I/EyeGestureManager﹕ Removing listener: com.google.android.glass.websurg.websurg.ResultActivity$1@41b8b908 for eye gesture: LOOK_AWAY_FROM_SCREEN
12-10 18:46:11.314    2553-2553/com.google.android.glass.websurg.websurg I/EyeGestureManager﹕ Removing listener: com.google.android.glass.websurg.websurg.ResultActivity$1@41b8b908 for eye gesture: WINK

However they do not get detected.. even the WINK since it seems to be supported.

Answer 1

Google team already answered some of these but I will go ahead and provide more details about their answer and also provide an alternate way of doing these stuff you requested.

Dim the screen if there isn't an expectation that a user is looking at it.

This is consistent with the "in the here and now" experience of Glass. Glassware should always dim the screen if there isn't an expectation that a user is looking at it. Ideally it behaves like a timeline and dims after 15s. A user can 're-wake' the screen by looking up.

Update to be made: If a user is not looking at the results set in the card scroller, dim the screen.

Glass handles that itself but the problem is that if the user doesn't touch the Glass pad for about 10 seconds or more, Glass will go to sleep and your App will stop running. Great way of fixing this is to make Glass screen always on and check when the user looks at the screen or when they remove the Glass.

If the user looks at the screen, increase the brightness of the screen, if they look away, decrease the brightness of the screen.

If they remove the Glass from their face, decrease the brightness to zero, turn off the screen and stop running all the big CPU intensive code you have.

If they put back the Glass on their face, increase the brightness of the screen,turn on the Screen and then enable all your CPU intensive code.

You could just have a boolean variable to determine when to start or stop running. This method is recommended if you don't want your app to stop running after no touch event for seconds. It also saves battery when running your app.

Code Examples for the things I said above are below:

To Get Screen Brightness:

//Get Screen Brightness
    public float getScreenBrightness() {
       WindowManager.LayoutParams wMLayout = getWindow().getAttributes();
       return  wMLayout.screenBrightness;
    }

To Set Screen Brightness(0 to 1):

 //Set Screen Brightness
    public boolean setScreenBrightness(float sBrightness){
        if(sBrightness>=0){
            WindowManager.LayoutParams wMLayout = getWindow().getAttributes();
            wMLayout.screenBrightness = sBrightness; //Modify Brightness
            getWindow().setAttributes(wMLayout); //Apply changes
            return true;
        }else
        {
            return false;
        }
    }

To Keep the Screen On or Off:

    //Turn Screen On/Off
public void keepScreenOn(boolean screenOn){
    if(screenOn) {
        getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
    }else{
        getWindow().clearFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
    }
}

Don't forgeet to add permision in the Manifest:

<uses-permission android:name="android.permission.WAKE_LOCK" />

If you are just doing developing and don't want to worry about permision right now, you can use:

<uses-permission android:name="com.google.android.glass.permission.DEVELOPMENT" />

and avoid having to look up what permission to use. I suggest you use that for now as it will save you time during development and you don't have to worry about permission when coding.

[EYE GESTURE]

No Google official API available for detecting those. Anything available now is a little hack to access hidden Glass API for that. Glass team is working on it and they said the API will only be release when it is reliable. Right now, it is NOT perfect according to them.

NOTE

The answer I am about to post below SHOULD work but may NOT work on the next Glass update. When they do update, something magically changes and one function will STOP working. Glass API and Glass itself is on Beta Mode and therefore expect things to keep changing until official EYE Gesture API gets released.

There are two ways to detect Eye Gesture. One way is to use IntentFilter and wait for "gesture" message.Another way is to use Stub Library to Access the hidden Glass API. I will talk about both here as there are prons and cons for each method.

Method 1 (Stub Lib):

This is the way you are currently trying to do it.

Pros:

Can detect more gestures

Cons:

Wink CANNOT be stooped from taking pictures.

Yo are using different library than the one I used that is still working. I will try to fix your problem if that doesn't work, You should then do it the way I did mine with the library I used too.

You got Step 1 wrong.

Step 1: Create the stubs:

Create a package called com.google.android.glass. In this package create two classes: EyeGesture and EyeGestureManager

It should be

com.google.android.glass.eye

NOT

com.google.android.glass

com.google.android.glass may have worked in the past but there were too many updates.

So, EyeGesture and EyeGestureManager must be placed in your package called com.google.android.glass.eye

If Eye gesture is still not detected, forget about that library and use the one I am currently using. Close your project and create a new one.

Steps:

1) Download the library from here. (Last update was 4 months ago). The one you are currently using was probably last updated 8 months ago or even a year ago.

https://github.com/prt2121/EyeGestureLib

The zip file will have a long name like "EyeGestureLib-fwenindioniwenubwudew".

Rename the Zip file to "EyeGestureLib".

Extract the folder inside with a long name like "EyeGestureLib-f8a9fef3bde4396f947106e78cd0be7c7ecdd5a6"

Rename that folder to "EyeGestureLib"

The "EyeGestureLib" folder should have two folders inside it called "EyeGestureStub" and "EyeGestureDemoApp" plus other useless files.

2) Open Eclipse and create a new project. create a simple MainActivty class activity.

3) Inside your MainActivity class:

private EyeGestureManager mEyeGestureManager;
private EyeGestureListener mEyeGestureListener;

private EyeGesture target1 = EyeGesture.WINK;
private EyeGesture target2 = EyeGesture.DOUBLE_BLINK;
private EyeGesture target3 = EyeGesture.LOOK_AT_SCREEN;

Inside onCreate:

mEyeGestureManager = EyeGestureManager.from(this);
mEyeGestureListener = new EyeGestureListener();

Inside onStart:

mEyeGestureManager.register(target1, mEyeGestureListener);
mEyeGestureManager.register(target2, mEyeGestureListener);
mEyeGestureManager.register(target3, mEyeGestureListener);

Inside onStop:

mEyeGestureManager.unregister(target1, mEyeGestureListener);
mEyeGestureManager.unregister(target2, mEyeGestureListener);
mEyeGestureManager.unregister(target3, mEyeGestureListener);

Inside MainActivity (Not inside any function but just anywhere inside you MainActivity class):

private class EyeGestureListener implements Listener {

    @Override
    public void onEnableStateChange(EyeGesture eyeGesture, boolean paramBoolean) {

    }

    @Override
    public void onDetected(final EyeGesture eyeGesture) {
       //Show what we just detected
       Log.i(eyeGesture.toString() , " is detected");

              //Check which eye event occured
        if (eyeGesture.name() == target1.name()) {
            // Wink
            Log.i("EyeGesture: ", " you just winked");
        } else if (eyeGesture.name() == target2.name()) {
            // Double blink
            Log.i("EyeGesture: ", " you just double winked");
        } else if (eyeGesture.name() == target3.name()) {
            // Look at Screen
            Log.i("EyeGesture: ", " you Looked at Screen");
        }

    }
}

4) You will get error. Import the EyeGestureStub that is inside the EyeGestureLib folder to fix it.

To fix the error:

a) Go to File -> Import -> Android -> Existing Android Code into Workspace

Click Next, Browse and Browse the EyeGestureStub folder inside EyeGestureLib folder.

Make sure to exclude the "EyeGestureDemoApp" if it is there. You ONLY need EyeGestureLib folder which contains EyeGesture and EyeGestureManager.

b) Right click on "EyeGestureStub" -> Properties -> Android -> On the right side,under Project Build Target make sure that "Glass Development Kit Preview" check-box is checked.

Under Library, make sure that the "Is Library" check-box is checked.

Click Apply and Ok to exit the window.

c) Open Android SDK Manger. check for the version of Android SDK Build-tools installed. I have 21.1.1.

d) Open the project.properties of EyeGestureStub and change sdk.buildtools=18.1.1 to sdk.buildtools=21.1.1 Finish.

Done. It should work if you followed the instruction.

Run it and choose MainActivity as the the Launch Activity.

<-------------------------------------------------------------------------------------------------------------------------------->

[STILL NOT WORKING? IMPORT EVERYTHING && work from there]

If you can't get it to Work, delete the current project and import the whole project downloaded then work from there up. This is the easiest way. You may need to fix some errors before you can compile.**

To import the project,

1) Go to File -> Other -> Android -> Android Project from Existing Code.

Next -> Browse

then choose the EyeGestureLib folder which contains both the EyeGestureStub and EyeGestureDemoApp.

Make sure under Project to Import that both EyeGestureStub and EyeGestureDemoApp are check-box are checked then click Finish.

2) Right click on "EyeGestureStub" -> Properties -> Android -> On the right side,under Project Build Target make sure that "Glass Development Kit Preview" check-box is checked.

Under Library, make sure that the "Is Library" check-box is checked.

Click Apply and Ok to exit the window.

3) Right click on "MainActivity" -> Properties -> Android -> On the right side,under Project Build Target make sure that "Glass Development Kit Preview" check-box is checked.

4) You will get invisible error that will not be showing.

To see it Go to Windows -> Show View -> Problems There, you will see all the problems.

Next step to fix it, we have to match the Android SDK Build-tools version with the ones listed in the project.properties of both EyeGestureStub and MainActivity

a) Open Android SDK Manger. check for the version of Android SDK Build-tools installed. I have 21.1.1.

b) Open the project.properties of EyeGestureStub and change sdk.buildtools=18.1.1 to sdk.buildtools=21.1.1

c) Open the project.properties of MainActivity and change sdk.buildtools=18.1.1 to sdk.buildtools=21.1.1

Note: Changing the first project.properties may automatically change the second one.

Done. It should work if you followed the instruction.

Run it and choose MainActivity as the the Launch Activity.

<-------------------------------------------------------------------------------------------------------------------------------->

Method 2 (IntentFilter)

Pros:

Wink CAN be stopped from taking pictures.

Cons:

Detects WINK ONLY

The first method can receive four events (WINK,DOUBLE_WINK,DOUBLE_BLINK,LOOK_AT_SCREEN,) but this method can ONLY receive one event (WINK).

This method is useful if you just want to detect ONLY WINK without Glass taking a picture.

To listen to Intent, you have to extend BroadcastReceiver.

public class EyeGesture extends BroadcastReceiver {
    @Override
    public void onReceive(Context context, Intent intent) {
        if (intent.getStringExtra("gesture").equals("WINK")) {

             //Disable Camera Snapshot
            abortBroadcast();

            Log.e("WINKED ","");
        }else {
            Log.e("SOMETHING", "is detected " + intent.getStringExtra("gesture"));
        }
    }
}

You must register the intent in the Manifest as below:

 <receiver android:name="com.inno.inno.glassplugin.EyeGesture">
                <intent-filter>
                    <action android:name="com.google.android.glass.action.EYE_GESTURE" />
                </intent-filter>
 </receiver>

The name specified in the Manifest must match the name of the class listening to the intent which is EyeGesture.

Simple as that. No library required but only WINK can be detected. It also stops Glass from taking picture when wink is detected. You can comment abortBroadcast(); if you want Glass to take picture when event is detected.

This is for any one looking to detect Eye Gesture from Glass at this moment. These are the only current solutions around until Google releases their official Eye Gesture API.

You should file for a new Glass API feature here. File it as Glass Eye Gesture API Request. If the Glass team receives too much of this feature request, they will make it their top priority and release it. I already filed for one.

Answer 2

Got a return from the Google Glass Review team. Their response to :

Dim the screen if there isn't an expectation that a user is looking at it.

This is consistent with the "in the here and now" experience of Glass. Glassware should always dim the screen if there isn't an expectation that a user is looking at it. Ideally it behaves like a timeline and dims after 15s. A user can 'rewake' the screen by looking up.

Update to be made: If a user is not looking at the results set in the card scroller, dim the screen.

was this:

The platform handles this, are you overriding this in some way, are you holding a wake-lock when the result is shown?

So it seems as of now it is not intended to actually work straight with the EyeGesture, since it apparantely does it automatically (need confirmation on this part). In any case there is no point in trying to handle the LOOK_AT_SCREEN since the LOOK_AWAY_FROM_SCREEN isn't handled.

12-10 18:40:51.268    2405-2405/com.google.android.glass.websurg.websurg W/yupyup﹕ LOOK_AWAY_FROM_SCREEN:false

For those interested in handling the Wink EyeGesture here is what apparantely works according to some information I've gathered (needs to be confirmed).

The idea is to use is to use an EyeGesture and EyeGestureManager stub. Pretty much they exist within the environment but not in the API. This means to access them you need to create Subs that durring runtime will work (atleast that's how I understood it).

Apparantely there is a known bug when handling the WINK EyeGesture. It will take a picture. This may be caused from within the Google Glass settings where Take a picture when a wink is detected is activated. (needs to be confirmed).

So how to actually handle it?

Step 1: Create the stubs:

Create a package called com.google.android.glass. In this package create two classes: EyeGesture and EyeGestureManager

EyeGesture:

package com.google.android.glass.eye;

    import android.os.Parcel;
    import android.os.Parcelable;

    /**
     *  https://gist.github.com/victorkp/9094a6aea9db236a97f3E
     * 
     */
    public enum EyeGesture implements Parcelable {
        BLINK, DOFF, DON, DOUBLE_BLINK, DOUBLE_WINK, LOOK_AT_SCREEN, LOOK_AWAY_FROM_SCREEN, WINK;

        public int getId(){
            return -1;
        }

        @Override
        public int describeContents() {
            return 0;
        }

        @Override
        public void writeToParcel(Parcel dest, int flags) {

        }
    }

EyeGestureManager:

package com.google.android.glass.eye;

import android.content.Context;

/**
 * 
 * If there are any updates required check: https://gist.github.com/victorkp/9094a6aea9db236a97f3
 * 
 */
public class EyeGestureManager {
    public static final int INFINITE_TIMEOUT = -1;
    public static final String SERVICE_NAME = "eye_gesture";

    public interface Listener {
        public void onDetected(EyeGesture gesture);
    }

    public static EyeGestureManager from(Context paramContext) {
        return null;
    }

    public void activateGazeLogging(boolean paramBoolean) {
    }

    public boolean applyAndSaveCalibration(EyeGesture paramEyeGesture) {
        return false;
    }

    public boolean clearCalibration(EyeGesture paramEyeGesture) {
        return false;
    }

    public void enableGazeService(boolean paramBoolean) {
    }

    public boolean endCalibrationInterval(EyeGesture paramEyeGesture) {
        return false;
    }

    public boolean isCalibrationComplete(EyeGesture paramEyeGesture) {
        return false;
    }

    public boolean isGazeLogging() {
        return false;
    }

    public boolean isRegistered() {
        return false;
    }

    public boolean isSupported(EyeGesture paramEyeGesture) {
        return false;
    }

    public boolean loadCalibration(EyeGesture paramEyeGesture) {
        return false;
    }

    public boolean register(EyeGesture gesture, EyeGestureManager.Listener listener){
        return false;
    }

    public boolean startCalibrationInterval(EyeGesture paramEyeGesture) {
        return false;
    }

    public boolean unregister(EyeGesture gesture, EyeGestureManager.Listener listener) {
        return false;
    }
}

Great, you've got both stubs. First thing you should notice is that they aren't 100% like the one from the github link. Some of the functions have been deprecated, I've kept the ones that work. (I haven't tried every single one).

What is next? Well you need (not really but it's easier if you do) to create the GestureId class (yes you can put this anywhere you want to).

GestureId:

package com.google.android.glass.websurg.websurg;

import com.google.android.glass.eye.EyeGesture;

/**
 * 
 *
 * For updates check out: https://gist.github.com/victorkp/9094a6aea9db236a97f3
 * 
 *
 * 
 */
public class GestureIds {
    public int BLINK_ID;
    public int WINK_ID;
    public int DOUBLE_BLINK_ID;
    public int DOUBLE_WINK_ID;
    public int LOOK_AT_SCREEN_ID;
    public int LOOK_AWAY_FROM_SCREEN_ID;

    public GestureIds() {
        BLINK_ID = EyeGesture.BLINK.getId();
        WINK_ID = EyeGesture.WINK.getId();
        DOUBLE_BLINK_ID = EyeGesture.DOUBLE_BLINK.getId();
        DOUBLE_WINK_ID = EyeGesture.DOUBLE_WINK.getId();
        LOOK_AT_SCREEN_ID = EyeGesture.LOOK_AT_SCREEN.getId();
        LOOK_AWAY_FROM_SCREEN_ID = EyeGesture.LOOK_AWAY_FROM_SCREEN.getId();
    }
}

Great now you have all the classes to actually get started. Keep in mind the stubs are the ones that will work correctly during runtime (atleast I think so since I don't get any errors?)

Say you have a MainActivity and you want to add the EyeGesture:

private void createEyeGestureDetector(Context context) {
    mGestureIds = new GestureIds();
    mEyeGestureManager = EyeGestureManager.from(context);
    mEyeGestureListener = new EyeGestureManager.Listener() {
        @Override
        public void onDetected(EyeGesture gesture) {
            Log.w("EyeGestureListener", "Gesture: " + gesture.getId());
            int id = gesture.getId();
            if (id == mGestureIds.WINK_ID || id == mGestureIds.DOUBLE_WINK_ID) {
                Log.d("EyeGesture", "Wink");
            } else if (id == mGestureIds.BLINK_ID || id == mGestureIds.DOUBLE_BLINK_ID) {
                Log.d("EyeGesture", "Blink");
            } else if (id == mGestureIds.LOOK_AT_SCREEN_ID || id == mGestureIds.LOOK_AWAY_FROM_SCREEN_ID) {
                Log.d("EyeGesture", "Screen");
            }
            runOnUiThread(new Runnable() {
                @Override
                public void run() {
                    Log.w("detected", "omg detected");
                }
            });

        }
    };    
}

Notice I have also a run(). I've added both since one github linked had it and the other didn't. Tried both and doesn't seem to get detected.

You call this (the one right above) function in your onCreate(); , then in your onResume();

mEyeGestureManager.register(EyeGesture.LOOK_AT_SCREEN, mEyeGestureListener);
mEyeGestureManager.register(EyeGesture.LOOK_AWAY_FROM_SCREEN, mEyeGestureListener);
mEyeGestureManager.register(EyeGesture.WINK, mEyeGestureListener);

The on your onPause():

mEyeGestureManager.unregister(EyeGesture.LOOK_AT_SCREEN, mEyeGestureListener);
        mEyeGestureManager.unregister(EyeGesture.LOOK_AWAY_FROM_SCREEN, mEyeGestureListener);
        mEyeGestureManager.unregister(EyeGesture.WINK, mEyeGestureListener);

Now all this seems to work (no errors when the functions are called, and logs are shown saying they were called (notice there are no logs in the stubs)).

However, my google glass doesn't seem to be detecting the EyeGesture. I'm pretty sure it's all good, or that I'm missing something minor. I won't accept it as an answer since, it only answers part of my question. Feel free to try this out yourself and let me know how it works.