1

I'm trying to use the testng.xml... and looks like Device Farm is ignoring the whole file.

My example is simple. I have a factory class that instantiate test classes, and this is how my xml file looks like

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
    <test name="test">
        <classes>
            <class name="Factory"/>
        </classes>
    </test>
</suite>

I even tried to exclude methods manually

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
    <test name="test">
        <classes>
            <class name="Factory"/>
            <class name="TestSingleUrl">
                    <methods>
                        <exclude name="testCorrectSiteKey"/>
                        <exclude name="testIsTagImplemented"/>
                        <exclude name="testHttpsScheme"/>
                        <exclude name="testCorrectTagName"/>
                        <exclude name="testCorrectBootstrapPath"/>
                    </methods>
            </class>
        </classes>
    </test>
</suite>

But I'm getting a "Can't invoke method: either make it static or create a non-args constructor" . Which means that the Device Farm is trying to run the methods from TestSingleUrl Class..But the one who call these test should be the factory

Does anyone know how can we make the Device Farm to accept our xml file?

This is my Factory Class:

public class Factory{

    private String targetSiteKey;

    //Data Provider that consumes the first line, which contains the target sitekey of the tag
    @DataProvider(name = "urlProvider")
    public Iterator<Object[]> createData() throws IOException {
        List<String> lines = IOUtils.readLines(new InputStreamReader(this.getClass().getResourceAsStream("Urls.csv")));
        List<Object[]> objectArrays = lines.stream().map(x -> new Object[]{x}).collect(Collectors.toList());

        Iterator<Object[]> itr = objectArrays.iterator();
        targetSiteKey = itr.next()[0].toString();

        return itr;
    }

    @Factory(dataProvider="urlProvider")
    public Object[] creatingTests(String url){
        System.out.println(System.currentTimeMillis());
        String siteKey=null;
        String path = null;
        String tagName = null;
        WebElement browsi;
        try{
            RemoteWebDriver driver = DriverWrapper.getDriver();
            driver.get(url);
            browsi = driver.findElement(By.id("browsi-tag"));

            tagName = browsi.getTagName();
            siteKey = browsi.getAttribute("data-sitekey");
            path = browsi.getAttribute("src");

        }catch(Exception ignored){}
        finally{
            return new Object[] {new TestSingleUrl(targetSiteKey,siteKey,path,tagName,url)};
        }
    }
}

And this is my TestSingleUrl Class:

public class TestSingleUrl {
    private String targetSiteKey,siteKey,path,tagName,url;

    public TestSingleUrl(String targetSiteKey,String siteKey, String path, String tagName,String url){
        this.targetSiteKey = targetSiteKey;
        this.siteKey=siteKey;
        this.path=path;
        this.tagName=tagName;
        this.url=url;
    }

    @Test
    public void testCorrectSiteKey(){
        System.out.println(url);
        Assert.assertEquals(siteKey, targetSiteKey);
    }

    @Test
    public void testIsTagImplemented(){
        System.out.println(url);
        Assert.assertFalse(siteKey.isEmpty());
        Assert.assertFalse(tagName.isEmpty());
        Assert.assertFalse(path.isEmpty());
    }

    @Test
    public void testHttpsScheme(){
        System.out.println(url);
        Assert.assertTrue(path.contains("https"));
    }

    @Test
    public void testCorrectTagName(){
        System.out.println(url);
        Assert.assertEquals(tagName,"script");
    }

    @Test
    public void testCorrectBootstrapPath(){
        System.out.println(url);
        Assert.assertTrue(path.contains("middycdn-a.akamaihd.net/bootstrap/bootstrap.js"));
    }
}

I think that the error messages I'm getting is because the device farm is trying to run every method with a @Test annotation that it finds. If it was reading my xml file it wouldn't happen.

RocketRaccoon

Bruno Lerner
  • 101
  • 1
  • 8
  • Could you share Factory and TestSingleUrl? Looks like you have incorrectly implemented Factory pattern. – RocketRaccoon Aug 02 '17 at 13:07
  • Hey @RocketRaccoon , I just put the classes. I don't think is an error in my classes.. cause i can run it in a local device.. the problem is when i send it to the device farm. – Bruno Lerner Aug 02 '17 at 13:49
  • And how are you executing it there? – RocketRaccoon Aug 02 '17 at 17:26
  • Check a little bit about AWS Device Farm. So, have you tried to put your testng.xml at the root of *-test.jar dependency? Also, your code looks fine, it's an AWS's configuration issue, I think. – RocketRaccoon Aug 03 '17 at 06:33
  • I'm creating the zip file with the testng.xml file at the root of *-test.jar, I'm using Maven with the Surefire plugin and some others. I saw that they don't support every Testng feature but they said that the basics xml configuration (like which classes to run) they support. But I've been trying for days... – Bruno Lerner Aug 03 '17 at 09:02
  • Going to play a liitle bit with AWS Device Farm and TestNG on a weekends and get back with result after. – RocketRaccoon Aug 03 '17 at 09:59

1 Answers1

0

I'm not sure exactly but I believe you should be able to place the testng.xml in a resources directory and reference it from the pom.xml .

Looks like this post shows how to reference the testng.xml from the pom.xml: How to call testng.xml file from pom.xml in Maven

When you build the project then it will be included in the jar.

The awslabs project builds it by using this mvn command

mvn clean package -DskipTests=true

https://github.com/awslabs/aws-device-farm-appium-tests-for-sample-app

[Update] I downloaded the sample project from awslabs and imported it into eclipse. I then added this plugin to the pom.xml file:

  <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-surefire-plugin</artifactId>
           <version>2.12.4</version>
           <configuration>
              <suiteXmlFiles>
                 <suiteXmlFile>src/test/resources/testng.xml</suiteXmlFile>
              </suiteXmlFiles>
           </configuration>
        </plugin>     

I then created the testng.xml in ./src/test/resources/

I copied and pasted the xml provided here:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
    <test name="test">
        <classes>
            <class name="Factory"/>
        </classes>
    </test>
</suite>

I then built the project using

mvn clean package -DskipTests=true

This produced a zip-with-dependencies.zip file in the target directory.

I uploaded this test package to device farm by creating a new project and selecting Appium Java Testng as the type of tests that I want to run.

I encountered this parse error:

Failed to generate the test suites[TestNG] [ERROR] Cannot find class in classpath: Factory

Which is expected since I don't have that class in my classpath, however, this proves that Device Farm did read the testng.xml.

[update]

I am able to reproduce the problem when I use the exclude tag in the testng.xml. I.E when I use this xml:

 <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
    <test name="test">
        <classes>
            <class name="factory.TestSingleUrl">
                <methods>
                    <exclude name="testCorrectSiteKey" />
                    <exclude name="testIsTagImplemented" />
                    <exclude name="testHttpsScheme" />
                    <exclude name="testCorrectTagName" />
                    <exclude name="testCorrectBootstrapPath" />
                </methods>
            </class>
        </classes>
    </test>
</suite>

Going to investigate further on the method tags.

[update]

Looks like this is a known issue in Device Farm: AWS Device farm seems to be ignoring TestNG annotations

Guess we will just need to wait for them to update their environment and the parser.

You might be able to run your tests on a private device. You'll need to contact aws to get one at this email:

aws-devicefarm-support@amazon.com

[Update]

Looks like they've updated the webDriverAgent. See if it works now.

Hope that helps.

Best Regards

James

jmp
  • 2,175
  • 2
  • 17
  • 16
  • I made it work on a local device. The problem is when I send it to AWS Device Farm, they don't respect the testng.xml file. – Bruno Lerner Aug 06 '17 at 08:01
  • Does device farm give you any parsing results or error messages you can share? – jmp Aug 06 '17 at 12:17
  • The error message they are giving me is "Can't invoke method: either make it static or create a non-args constructor" . That is because they are trying to run the Test methods from the TestSingleUrl class, but their constructors have arguments because is the factory that instantiate them. So that's why I implied that they are ignoring the xml file – Bruno Lerner Aug 06 '17 at 12:58