1

I am trying to run inference with tensorflow android on a darkflow yolo model. I could successfully ran on the default tiny-yolo-voc model, but when I change my model (and it's parameters accordingly) to the yolo one, I get the following exception: 

java.lang.RuntimeException: Unable to resume activity {com.bendaf.tfdroidtest/com.bendaf.tfdroidtest.MainActivity}: java.lang.IllegalArgumentException: No OpKernel was registered to support Op 'ExtractImagePatches' with these attrs.  Registered devices: [CPU], Registered kernels:
  <no registered kernels>

   [[Node: ExtractImagePatches = ExtractImagePatches[T=DT_FLOAT, ksizes=[1, 2, 2, 1], padding="VALID", rates=[1, 1, 1, 1], strides=[1, 2, 2, 1]](47-leaky)]]
    at android.app.ActivityThread.performResumeActivity(ActivityThread.java:3841)
    at android.app.ActivityThread.handleResumeActivity(ActivityThread.java:3882)
    at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3048)
    at android.app.ActivityThread.-wrap14(ActivityThread.java)
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1639)
    at android.os.Handler.dispatchMessage(Handler.java:102)
    at android.os.Looper.loop(Looper.java:154)
    at android.app.ActivityThread.main(ActivityThread.java:6780)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1496)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1386)
 Caused by: java.lang.IllegalArgumentException: No OpKernel was registered to support Op 'ExtractImagePatches' with these attrs.  Registered devices: [CPU], Registered kernels:
  <no registered kernels>

   [[Node: ExtractImagePatches = ExtractImagePatches[T=DT_FLOAT, ksizes=[1, 2, 2, 1], padding="VALID", rates=[1, 1, 1, 1], strides=[1, 2, 2, 1]](47-leaky)]]
    at org.tensorflow.Session.run(Native Method)
    at org.tensorflow.Session.access$100(Session.java:48)
    at org.tensorflow.Session$Runner.runHelper(Session.java:295)
    at org.tensorflow.Session$Runner.run(Session.java:245)
    at org.tensorflow.contrib.android.TensorFlowInferenceInterface.run(TensorFlowInferenceInterface.java:143)
    at org.tensorflow.contrib.android.TensorFlowInferenceInterface.run(TensorFlowInferenceInterface.java:112)
    at com.bendaf.tfdroidtest.inference.InferenceRunner.runInference(InferenceRunner.java:44)
    at com.bendaf.tfdroidtest.MainActivity.runInferenceWithTiming(MainActivity.java:65)
    at com.bendaf.tfdroidtest.MainActivity.onResume(MainActivity.java:59)
    at android.app.Instrumentation.callActivityOnResume(Instrumentation.java:1277)
    at android.app.Activity.performResume(Activity.java:7094)
    at android.app.ActivityThread.performResumeActivity(ActivityThread.java:3818)
      ... 10 more

I assume regarding to this issue comment, that the ExtracImagePatches operator was not recognized by tensorflow. But how can I make tensorflow recognize it on my android device?

What I did so far:

  • generated the yolo.pb file with darkflow from the yolo.cfg according to this. (With the same command I generated the tiny-yolo-voc.pb):

$ ./flow --model cfg/yolo.cfg --load bin/yolo.weights --savepb --verbalise

  • optimized the yolo.pb file according to this possible solution and placed it to my android project's assets folder:

$ bazel-bin/tensorflow/tools/graph_transforms/transform_graph --in_graph=../darkflow/darkflow/built_graph/yolo.pb --out_graph=../android/test/TfDroidTest/app/src/main/assets/optimized_yolo.pb --inputs='input' --outputs='output' --transforms='strip_unused_nodes(type=float, shape="1,2,2,1") fold_constants(ignore_errors=true) fold_batch_norms fold_old_batch_norms'

  • Modified my tensorflow/contrib/makefile/tf_op_files.txt to contain the line tensorflow/core/kernels/extract_image_patches_op.cc according to this.

  • Rebuilt android tensorflow with these commands:

$ cd tensorflow/contrib/makefile/ && ./build_all_android.sh && cd -

$ ./configure

$ bazel build -c opt //tensorflow/contrib/android:libtensorflow_inference.so --crosstool_top=//external:android/crosstool --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --cpu=arm64-v8a

$ bazel build //tensorflow/contrib/android:android_tensorflow_inference_java

and added the .so and .jar files to my android project.

  • Rebuilt and deploy my android project with native support. Here is the relevant part of my InferenceRunner.java:

    mInferenceInterface.feed(INPUT_NODE, floatValues, 1, mInputSize, mInputSize, 3);
mInferenceInterface.run(new String[]{OUTPUT_NODE});

final float[] resu =
        new float[mGridSize * mGridSize * (mNumOfLabels + 5) * 5];
mInferenceInterface.fetch(OUTPUT_NODE, resu);

the exception is thrown in the second line. As I mentioned above, this works with the tiny-yolo-voc model, but not with the yolo model.

I am on Ubuntu 16.04, android API level 24. If you need any more info, feel free to comment. Thank you for your time!

bendaf
  • 2,981
  • 5
  • 27
  • 62
  • You need to compile in the ExtractImagePatches operation into a custom build of the TensorFlow library. There's some documentation here: http://www.oreilly.com/data/free/building-mobile-applications-with-tensorflow.csp – Pete Warden Aug 23 '17 at 15:46

2 Answers2

1

Add the implementation to the build

If you’re using Bazel, you’ll want to add the files you’ve found to the android_extended_ops_group1 or android_extended_ops_group2 targets. You may also need to include any .cc files they depend on in there. If the build complains about missing header files, add the .h’s that are needed into the android_extended_ops target.

If you’re using a makefile, go to tensorflow/contrib/makefile/ tf_op_files.txt and add the right implementation files there.

ccoiusu
  • 11
  • 1
0

If you are using Darkflow, you can try to change/swap method names (forward and _forward) in the .\darkflow\net\ops convolution.py reorg class.

See reference: using graph-yolo.pb error android example

Hunor Czirjek
  • 1
  • 2