0

Right now when my team made a deploy in out QA environment, I run one suite tests of robotframework. This, have several test that failed at first cause the environment is not "warm up". So, in the same pipeline I have the "--rerun" option if some test has failed. Usually, in the second run they work just fine. Then, I merge the outputs with rebot:

rebot --merge output.xml output2.xml

And even the log.html is showing correctly the information (on the test and on the suite level).

Status is PASS

Now comes the fun part. Even when in the output.xml now I have two runs of the test (all with some failled and the retries with all pass), when I upload this to XRay, it create a test execution with the results of the first run only.

So, my question is: why? In the output.xml is clearly al the results as the last run. If not, I would understand that this create one test execution and then put all the results inside (first and second run) which is not the case.

It seems to me that XRay is not importing the data correctly.

jesusfj710
  • 15
  • 3
  • I think you are mistaken here, the output.xml still has only the original values here, it is only the log.html that is merged – Jiri Janous Aug 24 '21 at 11:30
  • It would be interesting if you could share the merged output.xml file – Sérgio Aug 24 '21 at 13:14
  • @JiriJanous is correct. I thought that `output.xml` would be overwrite by the merge, but is not. Only the `log.html` is regenerated. This, with @Sérgio suggesting me to upload each run with is own `output.xml` would be the correct way to go. Thanks you both! – jesusfj710 Aug 24 '21 at 15:00
  • you're welcome :) appreciate if you could mark the answer as correct @jesusfj710. Thanks – Sérgio Aug 24 '21 at 15:26

1 Answers1

1

First, I have never used rebot in that way. However, I think what you obtained in Xray is the consequence of the "merged" output.xml you have and how Xray works. Whenever you upload test automation results to Xray, usually a Test Execution is going to be created, containing Test Runs (one per each Test).

Test issues will be autoprovisioned unless they already exist; if they exist, only Test Runs will be created for those Tests. A Test Execution cannot contain more than one Test Run for each Test. In other words, a test execution is like a task for running (or that contains the results of) a list of tests; this list cannot have duplicates.

I would advise to upload all the reports, individually. Xray will consider the latest results whenever showing coverage status.

Sérgio
  • 1,777
  • 2
  • 10
  • 12