TeamCity build Succeeds although my Jmeter test fails on Response Assertion

Question

I have some API test written in Jmeter and I run them trough our CI tool which is TeamCity. The problem is that TeamCity always shows build as Succeeded although I can see in my Jmeter report that some assertions from the test are failing. What I want is that TeamCity in that case shows build as Failed, accordingly.

TeamCity configuration is using Command Line as a Build step which triggers the following command:

jmeter -n -t jMeterTeamCityExample.jmx -l log\api_tests_log.jmx -e -o api-tests-results

Here is an example .jmx file that I made for testing purposes:

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="3.2" jmeter="3.3 r1808647">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="jMeter + TeamCity example" enabled="true">
      <stringProp name="TestPlan.comments"></stringProp>
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <stringProp name="TestPlan.user_define_classpath"></stringProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">stopthread</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <stringProp name="LoopController.loops">1</stringProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">1</stringProp>
        <stringProp name="ThreadGroup.ramp_time">1</stringProp>
        <longProp name="ThreadGroup.start_time">1530777613000</longProp>
        <longProp name="ThreadGroup.end_time">1530777613000</longProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
      </ThreadGroup>
      <hashTree>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="GET post" enabled="true">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
            <collectionProp name="Arguments.arguments"/>
          </elementProp>
          <stringProp name="HTTPSampler.domain">jsonplaceholder.typicode.com</stringProp>
          <stringProp name="HTTPSampler.port"></stringProp>
          <stringProp name="HTTPSampler.protocol"></stringProp>
          <stringProp name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp name="HTTPSampler.path">posts/1</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          <stringProp name="HTTPSampler.connect_timeout"></stringProp>
          <stringProp name="HTTPSampler.response_timeout"></stringProp>
        </HTTPSamplerProxy>
        <hashTree>
          <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
            <collectionProp name="Asserion.test_strings">
              <stringProp name="51508">400</stringProp>
            </collectionProp>
            <stringProp name="Assertion.test_field">Assertion.response_code</stringProp>
            <boolProp name="Assertion.assume_success">false</boolProp>
            <intProp name="Assertion.test_type">8</intProp>
          </ResponseAssertion>
          <hashTree/>
        </hashTree>
      </hashTree>
    </hashTree>
    <WorkBench guiclass="WorkBenchGui" testclass="WorkBench" testname="WorkBench" enabled="true">
      <boolProp name="WorkBench.save">true</boolProp>
    </WorkBench>
    <hashTree/>
  </hashTree>
</jmeterTestPlan>

In the file I have added invalid response code (400) in the Response Assertion by purpose, to cause the test to fail.

Conclusion and question: When I run this test through the TeamCity with the command provided above, TeamCity shows Success although the test itself fails on Response Assertion. I want that TeamCity shows fail if any response assertion from Jmeter Test plan fails. Anyone having any idea?

NOTE: I know about some Jmeter plugins for TeamCity but I would like to find a solution by using CommandLine build step only.

see https://stackoverflow.com/questions/42790863/teamcity-results-dont-match-jmeters — Ori Marko, Jul 05 '18 at 12:00
I may have answered your question on this thread: https://stackoverflow.com/a/52935009/5755161 — Michael Munson, Oct 22 '18 at 17:53

score 0 · Answer 1 · edited Jul 08 '19 at 16:21

I found a not very elegant solution.

Add build step to dump the jtl file to the team city build log:

I created command line build step with this command line on windows: powershell -nologo "& "Get-Content <location of your .jtl file> -Tail 100"

On linux you would do: cat <location of your .jtl file>

I added a build failure condition to look for this text in the build log "text false"

Here is a section of the build log which shows my failed meter assertion inside the build log:

[18:00:31]Step 5/5: check file for failed assertions (Command Line)
[18:00:31][Step 5/5] Starting: C:\BuildAgent\temp\agentTmp\custom_script6595741193421826142.cmd
[18:00:31][Step 5/5] in directory: C:\apache-jmeter-5.1.1\JmeterREsults\Assertionfailureexample\106
[18:00:31][Step 5/5] 1562598029549  58  HTTP Request API missioncontrol May2019 200 OK  Thread Group 1-1    text    false   Value expected to match regexp '114.13295998454568', but it did not match: '[112.13295998454568,113.30262810285055,82.08123404643247]'  16425   741 1   1   https://xxxxx-api.yyyyy
.com/api/page_mission_control/  58  0   0
[18:00:31][Step 5/5] Process exited with code 0

TeamCity build Succeeds although my Jmeter test fails on Response Assertion

1 Answers1