Is it possible to run a test over a complete data set passing the complete data set in @dataprovider and report the result of the whole?

Question

I have a testNg class where I have to run the same test over a complete set of data. I could provide each data line in the @dataprovider, however, the result for each data line would be displayed in the report file. In my case, I have to test millions of data. This would cause a very cluttered report. Below is an example.

TestData: My Test data is a file containing multiple JSON lines where each JSON object should have a name and occupation.

{"name":"Jim", "occupation":"racing"} 
{"name":,}

My Test NG class will look like:

public class TestJSon {
    @DataProvider("lineProvider")
    public Object[][] lineProvider() {
        // return JSON lines
    }

    @Test (dataProvider = "lineProvider")
    public void testJsonLines(String name, String occupation) {
        // Test name and occupation
    }
}

Now the output report will contain:

testJsonLines passed for line with name Jim testJsonLines failed for line {"name":,} testJsonLines failed for line {} testJsonLines failed for line {"name":"Jack","occupation":"}

I would like a report where it is simply printed:

testJSON inputs failed for inputs in lines 2,3 and 4

To summarize, I am trying to make my report group the results of a run over all data ra rather than print each. Is there a way to do this ?

score 0 · Answer 1 · answered Jul 08 '16 at 05:10

I had a similar issue. We test millions of data - dataprovider approach either leads to outofmemory error or the report is not something which is easy to look at.

I ended up having my own implementation of IReporter which writes the result to an excel sheet which allows me to have filters or do any customization i want. Ofcourse this has to be in conjunction with other listeners (ITestListener/IInvokedMethodListener) to capture the results the way you want it to.

In my case, I am using my own parallelism as well to get over the outofmemory error with so much data - I usually used to hit it at 90000 testdata.

Is it possible to run a test over a complete data set passing the complete data set in @dataprovider and report the result of the whole?

1 Answers1