Code coverage not working when used in conjunction with the parallel_spec task

Question

I would like to know why parallel rspec is showing a different coverage percentage and missed resources compared to when I run without parallelisation.

Here is the output:

    Sysctl[net.ipv6.conf.all.accept_redirects]
      Sysctl[net.ipv6.conf.all.disable_ipv6]
      Sysctl[net.ipv6.conf.default.accept_ra]
      Sysctl[net.ipv6.conf.default.accept_redirects]
      Sysctl[net.ipv6.conf.default.disable_ipv6]
      Sysctl[net.ipv6.conf.lo.disable_ipv6]
      Sysctl[vm.min_free_kbytes]
      Sysctl[vm.swappiness]
      Systemd::Unit_file[puppet_runner.service]
      Users[application]
      Users[global]
    F

Failures:

  1) Code coverage. Must be at least 95% of code coverage
     Failure/Error: RSpec::Puppet::Coverage.report!(95)

       expected: >= 95.0
            got:    79.01
     # /usr/local/bundle/gems/rspec-puppet-2.6.11/lib/rspec-puppet/coverage.rb:104:in `block in coverage_test'
     # /usr/local/bundle/gems/rspec-puppet-2.6.11/lib/rspec-puppet/coverage.rb:106:in `coverage_test'
     # /usr/local/bundle/gems/rspec-puppet-2.6.11/lib/rspec-puppet/coverage.rb:95:in `report!'
     # ./spec/spec_helper.rb:22:in `block (2 levels) in <top (required)>'

Finished in 42.12 seconds (files took 2.11 seconds to load)
995 examples, 1 failure

Failed examples:

rspec  # Code coverage. Must be at least 95% of code coverage


2292 examples, 2 failures


....................................................................

    Total resources:   1512
    Touched resources: 1479
    Resource coverage: 97.82%
    Untouched resources:

      Apt::Source[archive.ubuntu.com-lsbdistcodename-backports]
      Apt::Source[archive.ubuntu.com-lsbdistcodename-security]
      Apt::Source[archive.ubuntu.com-lsbdistcodename-updates]
      Apt::Source[archive.ubuntu.com-lsbdistcodename]
      Apt::Source[postgresql]

    Finished in 1 minute 25.3 seconds (files took 1.43 seconds to load)
    2292 examples, 0 failures

Could you update to provide more information about what you mean by "parallel rspec" so that we can reproduce - thanks. — Alex Harvey, Jun 08 '18 at 10:55
I would be more interested in chasing down the fact that your use of parallel rspec is causing two errors. That is probably the first place to start. Additionally, this question could use formatting to make it readable. — Matthew Schuchard, Jun 08 '18 at 16:33

Alex Harvey · Answer 1 · 2018-06-28T08:23:17.683

Because it is not entirely clear from the question, I assume here that you have set up code coverage by adding a line to your spec/spec_helper.rb like:

at_exit { RSpec::Puppet::Coverage.report!(95) }

The coverage report is a feature provided by rspec-puppet.

Also, I have assumed that you have more than one spec file that contain your tests and that these are being run in parallel by calling the parallel_spec task that is provided by puppetlabs_spec_helper.

The problem is this:

For code coverage to work properly, all of the Rspec tasks need to run within the same process (see the code here).

Meanwhile, for parallelisation to occur, there must be multiple spec files, which are run in parallel in separate processes. That limitation arises from the parallel_tests library that is used by the parallel_spec task. See its README.

The code coverage report, therefore, only counts resources that were seen inside each process.

Example:

class test {
  file { '/tmp/foo':
    ensure => file,
  }
  file { '/tmp/bar':
    ensure => file,
  }
}

Spec file 1:

require 'spec_helper'

describe 'test' do
  it 'is expected to contain file /tmp/foo' do
    is_expected.to contain_file('/tmp/foo').with({
      'ensure' => 'file',
    })
  end
end

Spec file 2:

require 'spec_helper'

describe 'test' do
  it 'is expected to contain file /tmp/bar' do
    is_expected.to contain_file('/tmp/bar').with({
      'ensure' => 'file',
    })
  end
end

spec_helper.rb:

require 'puppetlabs_spec_helper/module_spec_helper'
at_exit { RSpec::Puppet::Coverage.report!(95) }

Run in parallel:

Total resources:   2
Touched resources: 1
Resource coverage: 50.00%
Untouched resources:

  File[/tmp/bar]

Finished in 0.10445 seconds (files took 1.03 seconds to load)
1 example, 0 failures



Total resources:   2
Touched resources: 1
Resource coverage: 50.00%
Untouched resources:

  File[/tmp/foo]
Must be at least 95% of code coverage (FAILED - 1)

4 examples, 0 failures

Took 1 seconds

Run without parallelisation:

Finished in 0.12772 seconds (files took 1.01 seconds to load)
2 examples, 0 failures


Total resources:   2
Touched resources: 2
Resource coverage: 100.00%

Code coverage not working when used in conjunction with the parallel_spec task

1 Answers1