0

For a long time, I was able to solve my problems by reading many different posts here on StackOverflow, but now my problem gets quite interesting. I am stuck and maybe someone can give me some hints:

I would like to test Android applications using ActivityInstrumentationTestCase2. My test scenario is like follows: I have several test data sets (Hex Strings, encoding valid and invalid NDEF messages) in a DB on my Android device. I generate a java file having one test function for each row entry in DB containing a call to the app I would like to test.

Here is a example of one of those functions:

public void testNdefPush0() {

    Intent intent = new Intent(NfcAdapter.ACTION_TAG_DISCOVERED);
    NdefMessage[] msg;
    try {
        msg = new NdefMessage[] { new NdefMessage(new byte[] { (byte) 0xD1,
                (byte) 0x02, (byte) 0x1F, (byte) 0x53, (byte) 0x70,
                (byte) 0x91, (byte) 0x01, (byte) 0x0D, (byte) 0x00,
                (byte) 0x6E, (byte) 0x66, (byte) 0x63, (byte) 0x2D,
                (byte) 0x66, (byte) 0x6F, (byte) 0x72, (byte) 0x75,
                (byte) 0x6D, (byte) 0x2E, (byte) 0x6F, (byte) 0x72,
                (byte) 0x67, (byte) 0x51, (byte) 0x01, (byte) 0x0A,
                (byte) 0x55, (byte) 0x67, (byte) 0x6F, (byte) 0x6F,
                (byte) 0x67, (byte) 0x6C, (byte) 0x65, (byte) 0x2E,
                (byte) 0x63, (byte) 0x6F, (byte) 0x6D }) };
    } catch (Exception e) {
        throw new RuntimeException("Failed to create NdefMessage", e);
    }
    intent.putExtra(NfcAdapter.EXTRA_NDEF_MESSAGES, msg);

    setActivityIntent(intent);

    solo = new Solo(getInstrumentation(), getActivity());

}

As you can see, I make use of Robotium framework and I followed their FAQs and tutorials to set all this up (especially: https://code.google.com/p/robotium/wiki/RobotiumForAPKFiles)

I am using a specialised Instrumentation Runner, which is able to write the results in a xml file and seems suitable for my needs; here is my Manifest file:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="de.nfc"
    android:versionCode="1"
    android:versionName="1.0" >

    <uses-sdk android:minSdkVersion="10" />

    <uses-permission android:name="android.permission.NFC" />
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

    <instrumentation
        android:name="pl.polidea.instrumentation.PolideaInstrumentationTestRunner"
        android:targetPackage="se.anyro.nfc_reader" />

    <application
        android:icon="@drawable/ic_launcher"
        android:label="@string/app_name" >
        <uses-library android:name="android.test.runner" />
    </application>

</manifest>

What I have tried/achieved so far:

  • The apps I would like to test are third party apps, so I re-signed them with my debug key. (works)
  • Since I would like to test different apps, I implemented the generation of Java file containing the test cases on the phone; so by button click a *.java file holding all my tests (constructor, setUp(), testNdefPush0() ... testNdefPush100(), tearDown() ) and a suitable Manifest.xml is generated on the phone. Then these files are pulled to a PC, compiled (ant clean, ant debug, ant install -r /path/to/target.apk) in a test project and installed on the phone

  • I can run tests using adb shell:

    adb -d shell am instrument -w -r -e class de.nfc.tests.NfcTestCase -e junitOutputDirectory /mnt/sdcard/ de.nfc/pl.polidea.instrumentation.PolideaInstrumentationTestRunner

  • in case the test data are valid, instrumentation works and generates a xml file having the results on sdcard.

  • in case the test data is invalid (see example) Instrumentation crashes and does not continue with the following tests:

    INSTRUMENTATION_RESULT: longMsg=java.lang.NullPointerException: Unable to start activity ComponentInfo{se.anyro.nfc_reader/se.anyro.nfc_reader.TagViewer}: java.lang.NullPointerException

Question

How can I achieve, that tests, which failed are logged as failed, and JUnit continues with the following tests.

I found here a similar question Android JUnit test suite hangs after Activity throws NullPointerException in one test case, but no replies so far :(

Has anyone experienced in testing (third) party apps by passing test data via an Intent to them and can share/provide me some advice/experience?

I do hope I made the problem clear, cause it is quite complex setup and I hope I mentioned all relevant facts for someone not familiar with the topic (you know forest and trees ;) ) If not, don't mind to ask, I'll try to explain :D

Community
  • 1
  • 1
norwied
  • 1
  • 2

1 Answers1

0

This is our way handle the problem: we use a python parse to parse the result of "dexdump test.apk" to parse out all test class/methods, and we run them one by one.

Take a look at the dexdump result and it is quite obvious. The parser take less than 200 lines of code and should be finished in one day. The requirements are that you don't have complex test case class hierarchy otherwise you need to construct the hierarchy from dexdump to figure which tests need to run.

superb
  • 963
  • 1
  • 10
  • 21