Measuring performance using Android instrumentation test

Question

My goal was to write automatic performance test for Android CPU intensive code using an instrumental test (AndroidJUnitRunner).

I was very surprised to find that the test results are not reliable, to simulate CPU intensive code, I want to test, I wrote the following loop

for(int i=0;i<1000000;i++){
    Math.pow(2,i);
}

The code was tested as an instrumental test and within an Android app

The result I got was as follows:

Instrumental test showed ~230ms to complete the loop whereas the same code on the same device (G5) took ~600ms

I will appreciate any clue why the execution of same code on AndroidJUnitRunner takes three times less time than on the real device while both of them finally are executed on the same device

Probably the AndroidJUnitRunner has more priority than the code in your app — MatPag, Jul 06 '17 at 19:45
which begs the question, is there any possibility to set the specific code to be more prioritized — Denis Voloshin, Jul 07 '17 at 06:37
the code in the app is executed on the main thread I thought it has the highest priority anyway — Denis Voloshin, Jul 07 '17 at 06:57
I tried to use Thread.currentThread().setPriority(Thread.MAX_PRIORITY) and run it from background thread, I got the same result :( — Denis Voloshin, Jul 07 '17 at 07:25
Mmmm i think we are missing something... Maybe the test run with a small footprint then the entire app... — MatPag, Jul 07 '17 at 07:48
Supposedly, when you are running a standard app there are plenty of other things going on, whereas when you run an instrumented test - only that much chunk of code is being executed. — azizbekian, Jul 10 '17 at 05:55
it was my first suspicion as well, I tried to stop all running stuff on my device but got any effect, I think that instrumented test is executed on the JVM although it requires a device runtime to get mocked Android context, I don't see other explanation, the problem that I can't find any documented confirmation to my theory. — Denis Voloshin, Jul 10 '17 at 06:43

score 5 · Accepted Answer · answered Jul 15 '17 at 17:13

Simply speed comes from CPU, while you are executing some code, if CPU is not doing any heavy work and all CPU cores are up, it will execute your code pretty fast. In android, 'UI rendering' is most intensive work that AndroidJUnitRunner is not doing, that's why it is fast.

If you want to understand how android perform in different scenarios, take a look at this: https://www.youtube.com/playlist?list=PLWz5rJ2EKKc9CBxr3BVjPTPoDPLdPIFCE

score 1 · Answer 2 · answered Jul 16 '17 at 08:34

I think that the reason for the difference in results is the difference in the work that the the device needs to do in parallel to the test. (The overhead).

I suggest that you try to use a test framework that has a larger overhead than AndroidJUnitRunner. You can give Robotium a try.

Measuring performance using Android instrumentation test

2 Answers2