Simple Benchmark Testing with Vogar

Question

I want to write simple benchmark software for android. I researched and found Caliper + Vogar. But there is no example about them. Is there anyone use it ? I need simple code to call Caliper Benchmarks from android Activity.

Thanks for help.

score 0 · Answer 1 · answered Nov 22 '11 at 02:40

There's a complete example of a Caliper Benchmark for Android, though not for an Activity, in Android Application Testing Guide, which also explains:

Caliper benchmark extends normally com.google.caliper.SimpleBenchmark which implements the Benchmark interface. Benchmarks are structured in a similar fashion as JUnit 3 tests and maintain the same structure with the difference that here benchmarks start with the prefix time as opposed to test.

The benchmarks in question looks like this:

public final void timeCelsiusToFahrenheit(int reps) {
         for (int i=0; i < reps; i++) {
           for (double t: temps) {
             TemperatureConverter.celsiusToFahrenheit(t);
           }
         }
}

One of the most interesting features is that caliper can upload the benchmark results and you can compare them graphically

caliper benchmarks

Thanks for your response, but I already have some benchmark methods as your example. Basically I want to show test results in an activiy. — ziya, Nov 22 '11 at 22:40

Kevin Ding · Answer 2 · 2020-02-01T13:15:38.117

recently , I do the same thing as you, and I succeed

1.download & configure environment for build aosp

2.build aosp

3.build /external/vogar, you will get vogar under out/host/linux-x86/bin/vogar

4.write your test code using caliper

5.run you test on the host with vogar

vogar --mode app_process --benchmark

vogar will automatically compile your code to dex file and push them to device,

call the app_process(which will call dalvikvm command with android JNI registerd) to run vogar test runner to run your testcase

vogar need your specifing classpath to compile or to build your testcase

the following is my log when I ran the vogar:

dvdface@ubuntu:~/Source/aosp/out/host/linux-x86/bin$ ./vogar --mode

app_process --benchmark

~/Source/aosp/libcore/benchmarks/src/benchmarks/AdditionBenchmark.java

Actions: 1 executing

[/home/dvdface/Source/aosp/prebuilts/sdk/tools/jack, -D,

jack.java.source.version=1.8, -D, jack.android.min-api-level=o-b1,

-cp, /home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/core-libart_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/core-oj_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/conscrypt_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/okhttp_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/legacy-test_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/bouncycastle_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/ext_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/framework_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/telephony-common_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/mms-common_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/framework_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/android.policy_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/services_intermediates/classes.jack:/home/dvdface/Source/aosp/out/target/common/obj/JAVA_LIBRARIES/apache-xml_intermediates/classes.jack:/home/dvdface/Source/aosp/out/host/linux-x86/framework/vogar.jar,

--output-jack, /tmp/vogar/24a8baa0-30f1-4861-99c1-6660e97bf1d8/benchmarks.AdditionBenchmark/benchmarks.AdditionBenchmark.jack,

--import-resource, /tmp/vogar/24a8baa0-30f1-4861-99c1-6660e97bf1d8/benchmarks.AdditionBenchmarkexecuting

[adb, shell, mkdir,

/data/local/tmp/vogar/run/benchmarks.AdditionBenchmark]

executing [/home/dvdface/Source/aosp/prebuilts/sdk/tools/jack,

--output-dex-zip, /tmp/vogar/24a8baa0-30f1-4861-99c1-6660e97bf1d8/benchmarks.AdditionBenchmark/benchmarks.AdditionBenchmark.dex.jar,

-D, jack.java.source.version=1.8, -D, jack.android.min-api-level=o-b1, --multi-dex, native, --import, /tmp/vogar/24a8baa0-30f1-4861-99c1-6660e97bf1d8/benchmarks.AdditionBenchmark/benchmarks.AdditionBenchmark.jackAction

benchmarks.AdditionBenchmark executing [adb, shell, cd

/data/local/tmp/vogar/run/benchmarks.AdditionBenchmark &&

ANDROID_DATA=/data/local/tmp/vogar app_process

-Djava.class.path=/data/local/tmp/vogar/run/benchmarks.AdditionBenchmark.dex.jar

-Djava.io.tmpdir=/data/local/tmp/vogar/run/benchmarks.AdditionBenchmark

-Duser.home=/data/local/tmp/vogar/run/user.home /data/local/tmp/vogar/run/benchmarks.AdditionBenchmark

vogar.target.TestRunn Experiment selection:
  Benchmark Methods:   [timeAddConstantToLocalDouble, timeAddConstantToLocalFloat, timeAddConstantToLocalInt,
timeAddConstantToLocalLong, timeAddTwoLocalDoubles,

timeAddTwoLocalFloats, timeAddTwoLocalInts, timeAddTwoLocalLongs]
  Instruments:   [runtime]

  User parameters:   {}

  Virtual machines:  [app_process]

  Selection type:    Full cartesian product

This selection yields 8 experiments.

Trial Report (1 of 8):

  Experiment {instrument=runtime, benchmarkMethod=timeAddConstantToLocalDouble, vm=app_process,
parameters={}}
  Results:

    runtime(ns): min=3.48, 1st qu.=3.48, median=3.48, mean=3.48, 3rd qu.=3.48, max=3.48

Trial Report (2 of 8):

  Experiment {instrument=runtime, benchmarkMethod=timeAddConstantToLocalFloat, vm=app_process,
parameters={}}
  Results:

    runtime(ns): min=5.28, 1st qu.=5.28, median=5.28, mean=5.28, 3rd qu.=5.28, max=5.28

Trial Report (3 of 8):

  Experiment {instrument=runtime, benchmarkMethod=timeAddConstantToLocalInt, vm=app_process,
parameters={}}
  Results:

    runtime(ns): min=2.11, 1st qu.=2.11, median=2.11, mean=2.11, 3rd qu.=2.11, max=2.11

Trial Report (4 of 8):

  Experiment {instrument=runtime, benchmarkMethod=timeAddConstantToLocalLong, vm=app_process,
parameters={}}
  Results:

    runtime(ns): min=2.10, 1st qu.=2.10, median=2.10, mean=2.10, 3rd qu.=2.10, max=2.10

Trial Report (5 of 8):

  Experiment {instrument=runtime, benchmarkMethod=timeAddTwoLocalDoubles, vm=app_process, parameters={}}

  Results:

    runtime(ns): min=5.31, 1st qu.=5.31, median=5.31, mean=5.31, 3rd qu.=5.31, max=5.31

Trial Report (6 of 8):

  Experiment {instrument=runtime, benchmarkMethod=timeAddTwoLocalFloats, vm=app_process, parameters={}}

  Results:

    runtime(ns): min=5.23, 1st qu.=5.23, median=5.23, mean=5.23, 3rd qu.=5.23, max=5.23

Trial Report (7 of 8):

  Experiment {instrument=runtime, benchmarkMethod=timeAddTwoLocalInts, vm=app_process, parameters={}}

  Results:

    runtime(ns): min=1.63, 1st qu.=1.63, median=1.63, mean=1.63, 3rd qu.=1.63, max=1.63

Trial Report (8 of 8):

  Experiment {instrument=runtime, benchmarkMethod=timeAddTwoLocalLongs, vm=app_process, parameters={}}

  Results:

    runtime(ns): min=2.14, 1st qu.=2.14, median=2.14, mean=2.14, 3rd qu.=2.14, max=2.14

Collected 8 measurements from:

  1 instrument(s)

  1 virtual machine(s)

  8 benchmark(s)

Execution complete: 19.90 s.

Instrument com.google.caliper.runner.AllocationInstrument not supported on Dalvik, ignoring

Instrument com.google.caliper.runner.AllocationInstrument not supported on Dalvik, ignoring

Some trials failed to upload. Consider uploading them manually.   benchmarks.AdditionBenchmark OK (SUCCESS) received exit value 0 from
destroyed command adb shell cd

/data/local/tmp/vogar/run/benchmarks.AdditionBenchmark &&

ANDROID_DATA=/data/local/tmp/vogar app_process

-Djava.class.path=/data/local/tmp/vogar/run/benchmarks.AdditionBenchmark.dex.jar

-Djava.io.tmpdir=/data/local/tmp/vogar/run/benchmarks.AdditionBenchmark

-Duser.home=/data/local/tmp/vogar/run/user.home /data/local/tmp/vogar/run/benchmarks.AdditioMoving

/data/local/tmp/vogar/run/benchmarks.AdditionBenchmark/benchmarks.AdditionBenchmark.2018-04-01T15:43:17Z.json

to ./vogar-results Outcomes: 1. All successful. Took 10m35s.

what do you mean? my answer is valid , you can see the logs I offer, there is test result in the log. — Kevin Ding, Jan 22 '20 at 15:09
ok, I get it, almost means not. I succeed build and run the vogar testcase but failed to upload result to website to show result in graph. — Kevin Ding, Feb 01 '20 at 13:19

Simple Benchmark Testing with Vogar

2 Answers2