Jira Xray empty Test Details with Behave or Cubumber reports

Question

I'm trying to follow the Testing using Behave in Python tutorial. I can get the import to work, but the "The execution details displays the result of the Cucumber Scenario." does not work.

Here's what I'm doing:

I'm creating a new Test Execution (say, PROJ-123).
I'm creating a new Automated[Cucumber] test (say, PROJ-234)
I'm creating a new Automated[Cucumber] test (say, PROJ-345)

I'm using the following feature file with Behave

@PROJ-123
Feature: Verify something

Scenario Outline: Verify something with <data>
  Given I use the data <data>
   Then the result is <result>

@PROJ-234
Examples:
| data | result |
|  1   |    1   |

@PROJ-345
Examples:
| data | result |
|  2   |    4   |

I'm running behave with:

behave -k --format=cucumber_json:PrettyCucumberJSONFormatter -o cucumber.json --junit --format=json -o reports/data.json x.feature

I'm importing the report with:

curl -H "Content-Type: application/json" -X POST -u user:password --data @reports/data.json "https://jira.example.com/rest/raven/1.0/import/execution/behave"

The server reply is:

{"testExecIssue":{"id":"574356","key":"PROJ-123","self":"https://jira.example.com/rest/api/2/issue/574356"},"testIssues":{"success":[{"id":"574408","key":"PROJ-234","self":"https://jira.example.com/rest/api/2/issue/574408"},{"id":"574409","key":"PROJ-345","self":"https://jira.example.com/rest/api/2/issue/574409"}]}}

But when I look at the Test Details for either PROG-234 or PROJ-345, it's empty:

I've also tried to import the Cucumber JSON test report:

curl -H "Content-Type: application/json" -X POST -u user:pass --data @cucumber.json https://jira.example.com/rest/raven/1.0/import/execution/cucumber

{"testExecIssue":{"id":"574356","key":"PROJ-123","self":"https://jira.example.com/rest/api/2/issue/574356"},"testIssues":{"success":[{"id":"574408","key":"PROJ-234","self":"https://jira.example.com/rest/api/2/issue/574408"},{"id":"574409","key":"PROJ-345","self":"https://jira.example.com/rest/api/2/issue/574409"}]}}

The result is exactly the same: empty Test Details for either PROG-234 or PROJ-345.

I'm using Jira Data Center v8.13.1 with Xray.

Edit 1: Sergio's comment below states that if I have a feature like the one below it should work:

@PROJ-123
Feature: Verify something

  @PROJ-234
  # Jira Test ID
  Scenario Outline: Verify something with <data>
    Given I use the data <data>
     Then the result is <result>

Examples:
  | data | result |
  |   1  |    1   |
  |   2  |    4   |

This second feature file generates the following Cucumber JSON report:

[
{
  "description": "",
  "elements": [
    {
      "description": "",
      "id": "verify-something;verify-something-with-1----@1.1-",
      "keyword": "Scenario Outline",
      "line": 13,
      "location": "x.feature:13",
      "name": "Verify something with 1 -- @1.1 ",
      "steps": [
        {
          "keyword": "Given",
          "line": 7,
          "match": {
            "location": "steps/x.py:3"
          },
          "name": "I use the data 1",
          "result": {
            "duration": 1996756,
            "status": "passed"
          },
          "step_type": "given"
        },
        {
          "keyword": "Then",
          "line": 8,
          "match": {
            "location": "steps/x.py:7"
          },
          "name": "the result is 1",
          "result": {
            "duration": 993013,
            "status": "passed"
          },
          "step_type": "then"
        }
      ],
      "tags": [
        {
          "line": 1,
          "name": "PROJ-234"
        }
      ],
      "type": "scenario"
    },
    {
      "description": "",
      "id": "verify-something;verify-something-with-2----@1.2-",
      "keyword": "Scenario Outline",
      "line": 14,
      "location": "x.feature:14",
      "name": "Verify something with 2 -- @1.2 ",
      "steps": [
        {
          "keyword": "Given",
          "line": 7,
          "match": {
            "location": "steps/x.py:3"
          },
          "name": "I use the data 2",
          "result": {
            "duration": 1998901,
            "status": "passed"
          },
          "step_type": "given"
        },
        {
          "keyword": "Then",
          "line": 8,
          "match": {
            "location": "steps/x.py:7"
          },
          "name": "the result is 4",
          "result": {
            "duration": 0,
            "status": "passed"
          },
          "step_type": "then"
        }
      ],
      "tags": [
        {
          "line": 1,
          "name": "PROJ-234"
        }
      ],
      "type": "scenario"
    }
  ],
  "id": "verify-something",
  "keyword": "Feature",
  "line": 2,
  "name": "Verify something",
  "status": "passed",
  "tags": [
    {
      "line": 1,
      "name": "PROJ-123"
    }
  ],
  "uri": "x.feature"
}
]

It doesn't. The Test Details are still empty (with Behave or Cucumber reports).

score 0 · Answer 1 · answered Mar 19 '21 at 10:12

0

Multiple "Examples" sections in a given Scenario Outline are not supported, at this time. Also, the tag should be added on the Scenario Outline entry and not on the Examples section. Another important thing is that the Tests you have, the Cucumber Type should be "Scenario Outline" and not "Scenario". This should fix your initial problem.

answered Mar 19 '21 at 10:12

Sérgio

1,777
2
10
12

I've changed my feature file, but I still don't get the Test Details to be filled in. Another point is that the only thing that doesn't work is PROJ-234 and PROJ-345 not getting the Test details filled. They're correctly associated with PROJ-123. – Leonardo Mar 19 '21 at 14:48
Did you change the Scenario Type to "Scenario Outline" in the corresponding Test issues in Jira? – Sérgio Mar 19 '21 at 15:28
Yes, the feature files are the ones I collated above, they're both Scenario Outlines. – Leonardo Mar 19 '21 at 17:53
Can you please update your question with the cucumber.json file generated when running the Scenario Outline (it must have just one Examples section)? – Sérgio Mar 22 '21 at 09:58

Leonardo · Answer 2 · 2021-03-22T15:30:16.913

0

This isn't supposed to work, I should have read the tutorial more carefully:

The test (specification) is initialy created in Jira as a Cucumber Test and afterwards, it is exported using the UI or the REST API.

The Test Details won't be populated by a test report.

edited Mar 22 '21 at 15:30

answered Mar 19 '21 at 17:54

Leonardo

1,533
17
28

score 0 · Answer 3 · answered Mar 30 '23 at 13:07

It is not supported. However, it can be easily solved by using Jira api to update the issue. The only requisite is we need to create a Jira access token.

After creating the test execution in Jira:

curl -H "Content-Type: application/json" -X POST -H "Authorization: Bearer $token"  --data @"results/behave.json" "$BASE_URL/api/v2/import/execution/behave"
{"id":"ID","key":"XXX-NN","self":"https://BASE_URL/rest/api/2/issue/493054"}%

We get the key of the test execution in Jira. We can now use Jira api to update the test execution title and description. We first create a json file with the information we want to send:

info.json

{"fields":{"summary":"title changed","description":"description changed"}}

And then we update the test execution using the following PUT call:

JIRA_USER=your_user@user.com
JIRA_TOKEN=YOUR_JIRA_TOKEN
curl -D- -u $JIRA_USER:$JIRA_TOKEN -X PUT -H "Content-Type: application/json" https:/BASE_URL/rest/api/2/issue/XXX-NN -d @update.json

Jira Xray empty Test Details with Behave or Cubumber reports

3 Answers3