Gitlab ci- report jmeter test failures

Question

We have Gitlab job for our Jmeter+Maven integration project. Which will execute .jmx file and generate jmeter dashboard report and send a mail. Following is my yml file

stages:
- test
test_ba_bpmm_qaa-jb:
stage: test
tags:
- qa-api-automation

script:
- echo "-------------Running maven command to run jmeter tests---------"
- pwd 
- mvn clean install
- echo "-------------------Execution completed---------"
- cd target/jmeter
- pwd
- zip -r bpm_api_testresult.zip reports
- echo "This is the message body" | swaks --to abc@xyz.com --from 
"contact@xyz.com" --server 192.178.176.45 --auth LOGIN --auth-user 
"qauser1@sip.test" --auth-password "abc123" --attach 
"bpm_api_testresult.zip"
allow_failure: false
only:
- master

And following is the output:

[INFO] -------------------------------------------------------
[INFO]  P E R F O R M A N C E    T E S T S
[INFO] -------------------------------------------------------
[INFO]  
[INFO]  
[INFO] Executing test: Master_BPM_APIs.jmx
[INFO] Starting process with:[java, -Xms512M, -Xmx512M, -jar, ApacheJMeter- 
4.0.jar, -d, /home/gitlab-runner/builds/af1bddbf/0/sip-jbpm6-5/bpm-api- 
automation/target/jmeter, -e, -j, /home/gitlab-runner/builds/af1bddbf/0/sip- 
jbpm6-5/bpm-api-automation/target/jmeter/logs/Master_BPM_APIs.jmx.log, -l, 
/home/gitlab-runner/builds/af1bddbf/0/sip-jbpm6-5/bpm-api- 
automation/target/jmeter/results/Master_BPM_APIs.csv, -n, -o, /home/gitlab- 
runner/builds/af1bddbf/0/sip-jbpm6-5/bpm-api- 
automation/target/jmeter/reports/Master_BPM_APIs_20181023_164847, -t, 
/home/gitlab-runner/builds/af1bddbf/0/sip-jbpm6-5/bpm-api- 
automation/target/jmeter/testFiles/Master_BPM_APIs.jmx]
[INFO] Creating summariser <summary>
[INFO] Created the tree successfully using /home/gitlab- 
runner/builds/af1bddbf/0/sip-jbpm6-5/bpm-api- 
automation/target/jmeter/testFiles/Master_BPM_APIs.jmx
[INFO] Starting the test @ Tue Oct 23 16:48:50 IST 2018 (1540293530191)
[INFO] Waiting for possible Shutdown/StopTestNow/Heapdump message on port 
4445
[INFO] 1
[INFO] summary =      8 in 00:00:02 =    4.6/s Avg:    82 Min:     2 Max:   
207 Err:     1 (12.50%)
[INFO] Tidying up ...    @ Tue Oct 23 16:48:52 IST 2018 (1540293532577)
[INFO] ... end of run
[INFO] Completed Test: /home/gitlab-runner/builds/af1bddbf/0/sip-jbpm6- 
5/bpm-api-automation/target/jmeter/testFiles/Master_BPM_APIs.jmx
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ bpm-api- 
automation ---
[INFO] Installing /home/gitlab-runner/builds/af1bddbf/0/sip-jbpm6-5/bpm-api- 
automation/target/bpm-api-automation-0.0.1-SNAPSHOT.jar to /home/gitlab- 
runner/.m2/repository/com/causeway/bpm/bpm-api-automation/0.0.1- 
SNAPSHOT/bpm-api-automation-0.0.1-SNAPSHOT.jar
[INFO] Installing /home/gitlab-runner/builds/af1bddbf/0/sip-jbpm6-5/bpm-api- 
automation/pom.xml to /home/gitlab- 
runner/.m2/repository/com/causeway/bpm/bpm-api-automation/0.0.1- 
SNAPSHOT/bpm-api-automation-0.0.1-SNAPSHOT.pom
[INFO] --------------------------------------------------------------------- 
 ---
[INFO] BUILD SUCCESS
[INFO] --------------------------------------------------------------------- 
 ---
[INFO] Total time: 11.011 s
[INFO] Finished at: 2018-10-23T16:48:54+05:30
[INFO] --------------------------------------------------------------------- 
 ---
[INFO] Shutdown detected, destroying JMeter process...
[32;1m$ echo "-------------------Execution completed---------"[0;m
-------------------Execution completed---------

But what we need is if any error/ test failure then build should set to fail or return build status 0 or 1 and return complete error in job log console

score 0 · Answer 1 · answered Oct 24 '18 at 17:09

Given you're using Jmeter Maven Plugin already if you add the next block to your pom.xml file:
```
<execution>
     <id>jmeter-check-results</id>
     <goals>
         <goal>results</goal>
     </goals>
</execution>
```
the Maven build will be marked as failed causing the whole Gitlab CI job failure.
You can switch to Taurus tool which provides powerful and flexible Pass/Fail Criteria subsystem allowing you to define how to deal with failures using simple declarative statements like
```
- avg-rt of IndexPage>150ms for 10s, stop as failed
- fail of CheckoutPage>50% for 10s, stop as failed
```
so when (if) the criteria will be met - Taurus will return non-zero exit code
JMeter-only solution: you can add a JSR223 Listener with the code like:
```
if (prev.isSuccessful()) {
    System.exit(-1)
}
```
it will return -1 exit status to the parent process, however in this case .jtl results file can be incomplete or corrupt

Gitlab ci- report jmeter test failures

1 Answers1