why jmeter assertion failure but maven project mark success

Question

I use jmeter-maven-plugin for maven and jmeter In jmeter , I have some assertion failure , but why the maven project also marked success? someone help me. Tanks!

here is my maven project pom.xml

<build>
        <plugins>
            <plugin>
                <groupId>com.lazerycode.jmeter</groupId>
                <artifactId>jmeter-maven-plugin</artifactId>
                <version>1.10.1</version>
                <executions>
                    <execution>
                        <id>jmeter-tests</id>
                        <phase>verify</phase>
                        <goals>
                            <goal>jmeter</goal>
                        </goals>
                        <configuration>
                            <overrideRootLogLevel>debug</overrideRootLogLevel>

                            <testFilesIncluded>
                                <jMeterTestFile>${includedTestFiles}</jMeterTestFile>
                            </testFilesIncluded>
                            <testFilesExcluded>
                                <excludeJMeterTestFile>${excluedTestFiles}</excludeJMeterTestFile>
                            </testFilesExcluded>
                            <jmeterPlugins>
                                <plugin>
                                    <groupId>kg.apc</groupId>
                                    <artifactId>jmeter-plugins-standard</artifactId>
                                </plugin>
                            </jmeterPlugins>

                            <propertiesJMeter>
                                <dateQueryValueS>${dateQueryValueS}</dateQueryValueS>>
                            </propertiesJMeter>
                        </configuration>
                    </execution>
                </executions>
                <dependencies>
                    <dependency>
                        <groupId>kg.apc</groupId>
                        <artifactId>jmeter-plugins-standard</artifactId>
                        <version>1.3.1</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>

here is the build

mvn verify -DincludedTestFiles=*unshipOrder.jmx -DexcluedTestFiles=noneOfExclued -DdateQueryValueS="2016-04-28 09:09:53"

[INFO]
[INFO] -------------------------------------------------------
[INFO]  P E R F O R M A N C E    T E S T S
[INFO] -------------------------------------------------------
[INFO]
[INFO]
[info]
[debug] JMeter is called with the following command line arguments: -n -t E:\study\jmeter\src\test\jmeter\xbn-online-unshipOrder.jmx -l E:\study\jmeter\target\jmeter\results\20160513-xbn-online-unshipOrder.jtl -d E:\study\jmeter\target\jmeter -L DEBUG -j E:\study\jmeter\target\jmeter\logs\xbn-online-unshipOrder.jmx.log
[info] Executing test: xbn-online-unshipOrder.jmx
[debug] Creating summariser <summary>
[debug] Created the tree successfully using E:\study\jmeter\src\test\jmeter\xbn-online-unshipOrder.jmx
[debug] Starting the test @ Fri May 13 10:20:31 CST 2016 (1463106031631)
[debug] Waiting for possible shutdown message on port 4445
[debug] summary +      1 in     1s =    1.9/s Avg:   370 Min:   370 Max:   370 Err:     0 (0.00%) Active: 1 Started: 1 Finished: 0
[debug] summary +      1 in   0.1s =   10.4/s Avg:    64 Min:    64 Max:    64 Err:     1 (100.00%) Active: 0 Started: 1 Finished: 1
[debug] summary =      2 in     1s =    3.2/s Avg:   217 Min:    64 Max:   370 Err:     1 (50.00%)
[debug] Tidying up ...    @ Fri May 13 10:20:32 CST 2016 (1463106032341)
[debug] ... end of run
[info] Completed Test: xbn-online-unshipOrder.jmx
[INFO]
[INFO] Test Results:
[INFO]
[INFO] Tests Run: 1, Failures: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.629 s
[INFO] Finished at: 2016-05-13T10:20:32+08:00
[INFO] Final Memory: 22M/223M
[INFO] ------------------------------------------------------------------------

You probably want to show what the assertion failure is. – Tibrogargan May 13 '16 at 03:29 — Tibrogargan, May 13 '16 at 03:29

Asanke · Answer 1 · 2016-12-15T00:04:52.310

I think this is the way Maven works.

Here: Maven executes the project through its lifecycle. While doing so its executes the plugins defined/configured in the pom.xml(Everything on the effective POM (in eclipse terms)).

So here "jmeter-maven-plugin" is one executor that's being executed during this process. The plugin does its job and provides the results.

Also maven completes its work with a SUCCESS note.Because the build from Mavens perspective is successful.

(This will be similar if you run JUnit/TestNG test suite) Here its up to us to find a way(tool) to interpret the generated results in a presentable manner.

Try below:

Put it on a CI tool like Jenkins.(Explained in one of the links in the answer by @DmitriT)
Use maven 3.0 dash board
jmeter-analysis-maven-plugin
jmeter-graph-maven-plugin

score 1 · Answer 2 · answered Dec 01 '17 at 22:42

If you use version 2.6.0 of plugin or upper , you just have to set this:

<executions>
   <execution>
      <id>performance test</id>
     <goals><goal>jmeter</goal></goals>
   </execution>
   <execution>
       <id>verify</id>
       <goals><goal>results</goal></goals>
   </execution>
</executions>

score 0 · Answer 3 · answered May 13 '16 at 03:45

0

Actually by default it should be failing, it's hard to say what's wrong without seeing your test plan and log files. What assertion is being used and how it is configured?

In the meantime you can try working it around by adding the following line to <configuration> section:

<ignoreResultFailures>false</ignoreResultFailures>

References:

answered May 13 '16 at 03:45

Dmitri T

159,985
5
83
133

I use Bean Shell Assertion: `int totalCount = Integer.parseInt(vars.get("totalCount_g0")); if(totalCount != 0){ Failure=true ; FailureMessage = "The totalCount was not as expected, now the totalCount is : " + totalCount ; }` and totalCount come from response – user6328248 May 13 '16 at 04:52

score 0 · Answer 4 · answered Oct 30 '17 at 18:04

You need to add this configuration to your POM file within the <configuration> tag <scanResultsForFailedRequests>true</scanResultsForFailedRequests> <ignoreResultFailures>false</ignoreResultFailures>

Also try taking out the <configuration> tag from the <executions> tag

and finally the executions block should look like this

<executions>
  <execution>
    <id>configure</id>
    <goals>
      <goal>configure</goal>
     </goals>
  </execution>
  <execution>
    <id>performance test</id>
    <goals>
      <goal>jmeter</goal>
    </goals>
  </execution>
  <execution>
    <id>verify</id>
    <goals>
      <goal>results</goal>
    </goals>
  </execution>
</executions>

why jmeter assertion failure but maven project mark success

4 Answers4