7

I've created automatic Android and iOS UI-Tests for my Xamarin application with the Xamarin UITest framework. When running the tests locally, they work fine, but when running them on the Bitrise CI, the iOS tests run fine, but the Android UI Tests keep failing with the following exception:

StartFirstActivity_WaitForActivity_ExpectButtonToHaveText
SetUp : System.Exception : Timed out waiting for result of ClearAppData2
Stack trace:
  at Xamarin.UITest.Shared.Android.Commands.CommandAdbClearAppData.Execute (IProcessRunner processRunner, IAndroidSdkTools androidSdkTools) <0x38b3e90 + 0x0064b> in <filename unknown>:0 
  at Xamarin.UITest.Shared.Execution.Executor.Execute[TDep1,TDep2] (ICommand2 command) <0x32b6478 + 0x00092> in <filename unknown>:0 
  at Xamarin.UITest.Shared.Android.LocalAndroidAppLifeCycle.EnsureInstalled (Xamarin.UITest.Shared.Android.ApkFile appApkFile, Xamarin.UITest.Shared.Android.ApkFile testServerApkFile) <0x37418c8 + 0x0017a> in <filename unknown>:0 
  at Xamarin.UITest.Android.AndroidApp..ctor (IAndroidAppConfiguration appConfiguration) <0x31a15e8 + 0x0047a> in <filename unknown>:0 
  at Xamarin.UITest.Configuration.AndroidAppConfigurator.StartApp (AppDataMode appDataMode) <0x30b4298 + 0x00063> in <filename unknown>:0 
  at SightPlayer.Core.Test.AppInitializer.StartApp (Platform platform) <0x30b2448 + 0x000ef> in <filename unknown>:0 
  at SightPlayer.Core.Test.Tests.BeforeEachTest () <0x30b23f8 + 0x00013> in <filename unknown>:0 
  at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) <0x30b2208 + 0x00093> in <filename unknown>:0

Android test are running with Xamarin.UITest version 1.3.5. This is important, as there seems to have been a bug until version 1.3.3. I've also tried ignoring failing tests, but then the test-runner fails with other Android tests. Interestingly enough that sometimes individual tests pass.

Has anyone encountered this behaviour before? Do you have any advice on how to fix this?

When decompiling CommandAdbClearAppData (which throws the exception), I see

// ISSUE: reference to a compiler-generated field
string str1 = string.Format("/data/data/{0}/files/calabash_failure.out", (object)executeCAnonStorey0.svr);
// ISSUE: reference to a compiler-generated field
string str2 = string.Format("/data/data/{0}/files/calabash_finished.out", (object)executeCAnonStorey0.svr);
while (DateTime.UtcNow < utcNow + TimeSpan.FromSeconds(10.0))
{
    if (func(string.Format("ls {0}", (object)str1)).Output.Trim().Equals(str1))
        throw new Exception("Clear app data failed with " + func(string.Format("cat {0}", (object)str1)).Output);
    if (func(string.Format("ls {0}", (object)str2)).Output.Trim().Equals(str2) && func(string.Format("cat {0}", (object)str2)).Output.Trim().Equals("SUCCESSFUL"))
        return;
}
throw new Exception("Timed out waiting for result of ClearAppData2");

which indicates that the generated file could not be found within ten seconds. Could it be, that the emulator is simply too slow and the emulator takes longer than ten seconds to create those files?

Alexander Pacha
  • 9,187
  • 3
  • 68
  • 108
  • When you run these tests locally, are you running them on an emulator as well? Emulators are definitely a bit slower than a plugged in device. I would recommend trying to change the `WaitTimes` configuration to a longer wait time: https://developer.xamarin.com/api/member/Xamarin.UITest.Configuration.AndroidAppConfigurator.WaitTimes/p/Xamarin.UITest.Utils.IWaitTimes/ – Jon Douglas Apr 03 '16 at 22:54
  • I am always running the tests on an emulator, but with emulators that run with x86 images (so they are actually almost faster than real devices). And I already do have WaitTimes configured to six minutes. – Alexander Pacha Apr 04 '16 at 09:36

3 Answers3

1

I think you need to look at some Configuration

If your app is not waiting to launch the test you could implement WaitTimes.

Create a class like this:

public class WaitTimes : IWaitTimes
{
    public TimeSpan GestureWaitTimeout
    {
        get { return TimeSpan.FromMinutes(1); }
    }

    public TimeSpan WaitForTimeout
    {
        get { return TimeSpan.FromMinutes(1); }
    }
}

Then implement it like this:

_app = ConfigureApp 
    .Android
    .EnableLocalScreenshots()
    .ApkFile(apkFile)
    .DeviceSerial("###")
    .ApiKey("###")
    .Debug()
    .WaitTimes(new WaitTimes()) 
    .StartApp();

Otherwise you could just wait for a specific element to load, like this:

_app.WaitForElement(c => c.Marked("Login"), "Time out expired", TimeSpan.FromSeconds(15));
var result = _app.Query(c => c.Marked("Login"));
Assert.AreEqual(1,result.Length);
_app.Screenshot("Test passed with success");

Here is some more information about WaitTime and WaitForElement that could be helpful.

Community
  • 1
  • 1
Jagadeesh Govindaraj
  • 6,977
  • 6
  • 32
  • 52
  • I already had wait-times implemented to wait for six minutes and configured to use them. The problem seems to be the timeout in `CommandAdbClearAppData`, doesn't it? – Alexander Pacha Apr 04 '16 at 09:37
1

Could it be, that the emulator is simply too slow and the emulator takes longer than ten seconds to create those files?

Yes, that could be the problem.

Do you have any advice on how to fix this?

The latest dev builds of the Xamarin.UITest nuget package have increased that timeout interval from 10 seconds to 60 seconds.

Try Xamarin.UITest 1.3.6.1476-dev or newer.

Glenn Wilson
  • 241
  • 1
  • 8
  • Updated to the newest version (1.3.7-1478-dev) and everything inbetween - no luck with my Android Tests so far. Right now they time-out after 30 minutes (CI kills it). Locally on a Mac and Windows they run fine. – Alexander Pacha Apr 14 '16 at 21:43
1

I solved this issue uploading a unsigned version of the APK to my testing device (same thing for Xamarin Android Player). I had to uncheck the "Sign the .APK file" box from the Android Package Signing option in the project properties. The WaitTimes way didn't work for me.

luisguerrerolab
  • 27
  • 9