Correct test Case execution gives a 'Failed' status

Question

I have a strange problem with testCase execution.

I have a test case with the following structure:

Setup script:

  set session_id = 0

  set session_index = 0

  store custom properties in an external file

and the test case body

Test Case:

  data source

    SOAP - authentication on components (gives session_id for each component)

    groovy - recover useful data (store session_id in custom prop session_id_<session_index>)

  source loop

  SOAP - start asynchronous service on component 1

  SOAP - check status on component 1

  goto 'check status' while status = Pending

  SOAP - check status on component 2

  goto 'disconnect terminals' is session_id = 0

  SOAP disconnect

  groovy disconnect terminal : loop for each session_index: set session_id = session_id_<session_index> and use it in the SOAP disconnect step (I use run test step)

TearDown Script : restore the custom properties

All of these steps execute with success, I don't have any error logged but the overall test has a FAILED status.

I realized that the problem comes from the goto steps: I'm looping back onto the SOAP request to check status WHILE the status returned is set to 'pending'. So each time the request returns 'pending' it's failing and I send the checking request again. When the status is finally 'success', the testStep becomes green (OK) and the testCase continues.

In the overall result, the testStep which is played several times is stored once for each 'iteration' and it's result is OK only for the last occurrence, so the overall test status is FAILED.

score 1 · Accepted Answer · answered Oct 17 '17 at 13:42

There is a workaround for this problem :

I checked the test results and checked that some of my test steps where considered as failed in testRunner.results. To do so, I set the following bit of code in my teardown script tab

for (testStep in testRunner.getResults()){
    log.info "status " + testStep.getTestStep().getName() + " : " + testStep.getStatus()
}

These steps were the ones on which I was looping, waiting for the status different from 'pending'. At the end of the test, as the status was finally 'success' the step was set as OK in the testCase (green-flagged), even if several occurrences of the testStep were 'failed' (which is confusing)

So I found I can update those status doing the following for the concerned steps:

for (results in testRunner.results){
 // implement a selection condition
    results.status = "OK"
}

I just have to make sure I apply this only on the concerned testSteps.

But at the end the overall status is still FAILED, still in my teardown script:

log.info "TEST RUNNER STATUS after update" + testRunner.getStatus()

If I'm sure of my steps and result, I can overwrite it as follows:

testRunner.status = "FINISHED"
log.info "TEST RUNNER STATUS after update" + testRunner.getStatus

And my overall test is ok (green bar)

I know it's a bit tricky, but as long as I set the proper conditions to update those parameters, I'm sure not to hide real failures.

If someone has a cleaner way to do this, I'd be grateful

Correct test Case execution gives a 'Failed' status

1 Answers1