Wrong tests results in Jira Xray from wdio cucumber json reporter

Question

We have a Story that is linked to the Cucumber Test and we execute this Test in Test Execution. The feature file from Test Execution looks like this:

@ABC-164
@ABC-163
Feature: Homepage test
    @ABC-162
    Scenario: Homepage test
        Feature: Go to homepage
            Scenario: Homepage
                Given I open the url "https://example.com/"
                Then I expect that the url is "https://google.com"

The first one is always passed and the second one failed. We are generating a JSON report from wdio cucumber json reporter, which you can see below:

[
  {
    "keyword": "Feature",
    "type": "feature",
    "description": "",
    "line": 3,
    "name": "Homepage test",
    "uri": "Can not be determined",
    "tags": [
      {
        "type": "Tag",
        "location": { "line": 1, "column": 1 },
        "name": "@ABC-164"
      },
      {
        "type": "Tag",
        "location": { "line": 2, "column": 1 },
        "name": "@ABC-163"
      }
    ],
    "elements": [
      {
        "keyword": "Scenario",
        "type": "scenario",
        "description": "\t\tFeature: Go to homepage",
        "name": "Homepage test",
        "tags": [
          { "name": "@ABC-164", "location": { "line": 1, "column": 1 } },
          { "name": "@ABC-163", "location": { "line": 2, "column": 1 } },
          { "name": "@ABC-162", "location": { "line": 6, "column": 2 } }
        ],
        "id": "homepage-test;homepage-test",
        "steps": [
          {
            "arguments": [],
            "keyword": "Before",
            "name": "Hook",
            "result": { "status": "passed", "duration": 2000000 },
            "line": "",
            "match": { "location": "can not be determined with webdriver.io" }
          },
          {
            "arguments": [],
            "keyword": "Before",
            "name": "Hook",
            "result": { "status": "passed", "duration": 0 },
            "line": "",
            "match": { "location": "can not be determined with webdriver.io" }
          }
        ]
      },
      {
        "keyword": "Scenario",
        "type": "scenario",
        "description": "",
        "name": "Homepage",
        "tags": [
          { "name": "@ABC-164", "location": { "line": 1, "column": 1 } },
          { "name": "@ABC-163", "location": { "line": 2, "column": 1 } }
        ],
        "id": "homepage-test;homepage",
        "steps": [
          {
            "arguments": [],
            "keyword": "Before",
            "name": "Hook",
            "result": { "status": "passed", "duration": 0 },
            "line": "",
            "match": { "location": "can not be determined with webdriver.io" }
          },
          {
            "arguments": [],
            "keyword": "Given",
            "name": "I open the url \"https://example.com/\"",
            "result": { "status": "passed", "duration": 587000000 },
            "line": 12,
            "match": { "location": "can not be determined with webdriver.io" }
          },
          {
            "arguments": [],
            "keyword": "Then",
            "name": "I expect that the url is \"https://google.com\"",
            "result": {
              "status": "failed",
              "duration": 11000000,
              "error_message": "Error: expect(received).toEqual(expected)"
            },
            "line": 13,
            "match": { "location": "can not be determined with webdriver.io" }
          },
          {
            "arguments": [],
            "keyword": "After",
            "name": "Hook",
            "result": { "status": "passed", "duration": 0 },
            "line": "",
            "match": { "location": "can not be determined with webdriver.io" }
          }
        ]
      }
    ],
    "id": "homepage-test",
    "metadata": {
      "browser": { "name": "chrome", "version": "88.0.4324.150" },
      "device": "Device name not known",
      "platform": { "name": "osx", "version": "Version not known" }
    }
  }
]

Xray only reads the status from Before Hooks from the first object and totally ignore the rest of the statuses.

The problem occurs when we use Jenkins Plugin and also with direct sending via curl like that: curl -k -H "Content-Type: application/json" -X POST -u login:pass --data @result.json https://jira/rest/...

Any help would be appreciated.

Answer 1

You cannot have a Feature underneath a Scenario. It’s a syntax error although Cucumber may not report it properly.

Answer 2

As Aslak mentioned in the comment, the .feature file syntax is not correct. A proper .feature file would be something like the example ahead. Please also have in mind that the tags you have (@ABC-xxx) may need to have some prefixes, such as REQ or TEST for example, depending on your settings.

Btw, it's unclear if you're exporting the Test Execution from Xray in order to generate the .feature file, but I don't think that's the case. Just to emphasize that in order to report results against existing Tests (Cucumber Scenario/Scenario Outlines) they must already exist beforehand in Xray.

@ABC-164
@ABC-163
Feature: Homepage test
    @ABC-162
    Scenario: Homepage
      Given I open the url "https://example.com/"
      Then I expect that the url is "https://google.com"*

Answer 3

For those who will have the same problem. We managed to find the solution. Thank you very much for the answers!

It looks like we had an old structure of the feature file in Jira (back then when we didn't have stories and preconditions). When we start to use it properly: linked Stories to Tests and add Pre-condition then Xray created doubled parts of scenarios. So we deleted the redundant parts from Cucumber Tests, create proper Pre-condition, and now the feature file is created properly like that:

@ABC-169
@ABC-131
Feature: Main feature title

    Background:
        #@ABC-166
        Given I open the url "https://example.com/"
        Then I expect that the url is "https://google.com"

    @ABC-132 @ABC-168 @ABC-141
    Scenario: Main scenario
        Given Lorem ipsum
        Then Et sit amet