Handle flaky tests when outputting junit from ruby cucumber (retry breaking the pipeline)

Question

I am outputting cucumber test results with two format flags:

--format junit
--format AllureCucumber::Formatter

The first is for CICD stage gates (100% passing tests in one environment triggers deployment to the next). The second is to output html dashboards.

I am additionally using the flag: --retry 2 because a couple of my tests are flaky (may fail once, but pass the second time around).

The Allure format handles reties gracefully. The problem is that the junit formatted xml handles flaky tests as failures.

Is there a workaround to tell xml to register 0 failures if all tests eventually pass?

I have found reference to this issue here.
Looking at this thread, this thread, and the jUnit docs, I don't think that retries/flaky get accounted for and I may need to patch the ruby-cucumber gem's formatter to just ignore the first failure when outputting jUnit.

Thanks

Answer 1

My quick and dirty patch for now is to patch the jUnit formatter to only increment the failure count like below. This only works for a single retry. When I have more time, I will see if there is a variable that can be checked to know if there are retries remaining for a test case.

def build_testcase(result, scenario_designation, output, test_case)
  duration = Cucumber::Formatter::ResultBuilder.new(result).test_case_duration
  @current_feature_data[:time] += duration
  classname = @current_feature_data[:feature].name
  name = scenario_designation

  @current_feature_data[:builder].testcase(:classname => classname, :name => name, :time => format('%.6f', duration)) do
    if !result.passed? && result.ok?(@config.strict)
      @current_feature_data[:builder].skipped
      @current_feature_data[:skipped] += 1
    elsif !result.passed? && !same_feature_as_previous_test_case?(test_case.feature)
      status = result.to_sym
      exception = get_backtrace_object(result)
      @current_feature_data[:builder].failure(:message => "#{status} #{name}", :type => status) do
        @current_feature_data[:builder].cdata! output
        @current_feature_data[:builder].cdata!(format_exception(exception)) if exception
      end
      @current_feature_data[:failures] += 1 
    end
    @current_feature_data[:builder].tag!('system-out') do
      @current_feature_data[:builder].cdata! strip_control_chars(@interceptedout.buffer_string)
    end
    @current_feature_data[:builder].tag!('system-err') do
      @current_feature_data[:builder].cdata! strip_control_chars(@interceptederr.buffer_string)
    end
  end
  @current_feature_data[:tests] += 1
end