Newman report generation works locally but not from CI

Question

I have a GitLab CI job running a series of Postman requests using a custom environment. I'm using Newman to run them alongside the newman-reporter-htmlextra npm plugin to generate a test report.

The job looks like the following:

postman-tests:
  stage: postman-tests
  image: 
    name: wojciechzurek/newman-ci
  before_script:
    - cd ci/tests/postman
    - npm install -g newman-reporter-htmlextra
  script:
    - newman run Non-regression_tests.postman_collection.json -e Tests.postman_environment.json \
      --reporters htmlextra --reporter-htmlextra-export newman-results.html
    - ls -la # Check report generation
  artifacts:
    when: always
    paths:
      - newman-results.html
  allow_failure: true

When I run newman on my mac (newman 4.5.0), the requests and associated tests run properly and the report is generated. However, the job fails and the report is not generated:

 $ newman run Non-regression_tests.postman_collection.json -e Tests.postman_environment.json --reporters htmlextra --reporter-htmlextra-export newman-results.html --color

Uploading artifacts...

WARNING: newman-results.html: no matching files    
ERROR: No files to upload                          
ERROR: Job failed: exit code 1

It seems that the issue may be caused by the testing series in itself rather than the report generation, as the job fails even when I don't generate the report.

I tried different runners: Docker with official newman images, SSH and shell over machines where I had installed newman (version 4.5.6) and the htmlextra reporter beforehand. All fail.
It's interesting to note that the tests series and report generation both succeed when run locally on the machines behind the SSH and shell runners, but they do fail when launched from GitLab CI.

What did I forget/do wrong that prevents the test report generation from GitLab CI?

Does it work with just the CLI reporter? – Danny Dainton Nov 19 '19 at 16:36 — Danny Dainton, Nov 19 '19 at 16:36

score 4 · Accepted Answer · answered Nov 19 '19 at 16:42

My .yml for testing, look like this - It's very basic but I've just run it again and it was running fine:

stages:
    - test

newman_tests:
    stage: test
    image: 
        name: postman/newman_alpine33
        entrypoint: [""]
    script:
        - newman --version
        - npm install -g newman-reporter-htmlextra
        - newman run collection.json -e environment.json --reporters cli,htmlextra --reporter-htmlextra-export testReport.html
    artifacts:
        when: always
        paths:
            - testReport.html

One thing that I do have is entrypoint: [""] in the image block.

Newman report generation works locally but not from CI

1 Answers1