18

I am trying to execute rspec from ruby, and get the status or number of failures from a method or something like that. Actually I am running something like this:

system("rspec 'myfilepath'")

but I only can get the string returned by the function. Is there any way to do this directly using objects?

Nakilon
  • 34,866
  • 14
  • 107
  • 142
Leo
  • 421
  • 4
  • 5

4 Answers4

28

I think the best way would be using RSpec's configuration and Formatter. This would not involve parsing the IO stream, also gives much richer result customisation programmatically.

RSpec 2:

require 'rspec'

config = RSpec.configuration

# optionally set the console output to colourful
# equivalent to set --color in .rspec file
config.color = true

# using the output to create a formatter
# documentation formatter is one of the default rspec formatter options
json_formatter = RSpec::Core::Formatters::JsonFormatter.new(config.output)

# set up the reporter with this formatter
reporter =  RSpec::Core::Reporter.new(json_formatter)
config.instance_variable_set(:@reporter, reporter)

# run the test with rspec runner
# 'my_spec.rb' is the location of the spec file
RSpec::Core::Runner.run(['my_spec.rb'])

Now you can use the json_formatter object to get result and summary of a spec test. 

# gets an array of examples executed in this test run
json_formatter.output_hash

An example of output_hash value can be found here:

RSpec 3

require 'rspec'
require 'rspec/core/formatters/json_formatter'

config = RSpec.configuration

formatter = RSpec::Core::Formatters::JsonFormatter.new(config.output_stream)

# create reporter with json formatter
reporter =  RSpec::Core::Reporter.new(config)
config.instance_variable_set(:@reporter, reporter)

# internal hack
# api may not be stable, make sure lock down Rspec version
loader = config.send(:formatter_loader)
notifications = loader.send(:notifications_for, RSpec::Core::Formatters::JsonFormatter)

reporter.register_listener(formatter, *notifications)

RSpec::Core::Runner.run(['spec.rb'])

# here's your json hash
p formatter.output_hash

Other Resources

activars
  • 1,664
  • 13
  • 15
  • I tried this and documentation_formatter.examples is an empty array – hadees Dec 26 '12 at 07:20
  • did you specify any test to run e.g. RSpec::Core::Runner.run(['my_spec.rb']), 'my_spec.rb' is the rspec test file, that needs to include file path. Maybe you didn't include the spec file correctly, then there's no test result. – activars Dec 28 '12 at 13:46
  • No, the test ran fine I saw output on the console but the documentation_formatter didn't seem to capture it. – hadees Dec 29 '12 at 03:00
  • do you have a Github gist or snippet that I can take look? It's difficult to diagnose the issue without example.. – activars Jan 03 '13 at 17:35
  • I double checked the lastest rspec, I think it's better to use the JsonFormatter, if you are still interested, please take a look the updated answer and related example links. – activars Jan 06 '13 at 15:55
  • `built_in_formatter` is a private method, it won't let me call it on `config` – hadees Apr 05 '13 at 20:27
  • @hadees , I just updated the example to use RSpec::Core::Formatters::JsonFormatter directly to create formatters. – activars Apr 09 '13 at 09:08
  • rerunning would cause same spec to be run multiple times. Is there a way to reset the whole stream? – lulalala Feb 15 '16 at 02:54
8

I suggest you to take a look into rspec source code to find out the answer. I think you can start with example_group_runner

Edit: Ok here is the way:

RSpec::Core::Runner::run(options, err, out)

Options - array of directories, err & out - streams. For example

RSpec::Core::Runner.run(['spec', 'another_specs'], $stderr, $stdout)
iafonov
  • 5,152
  • 1
  • 25
  • 21
  • Thanks for the reply, but something else will help, because this only bring to me more questions :( – Leo Aug 08 '11 at 19:30
  • I've added some info to answer. Hope this will help. – iafonov Aug 09 '11 at 08:54
  • Helps a lot! Something similar is: https://github.com/rspec/rspec-core/issues/359 – Leo Aug 11 '11 at 13:19
  • I have other problem ... seems when I execute this inside a for statement, the IO object crash: – Leo Aug 11 '11 at 19:57
  • Try to use a documentation formatter, the result can be queried programatically - no need to parse the IO. see here: http://stackoverflow.com/a/10412793/1041418 – activars Jul 05 '12 at 22:06
1

Your problem is that you're using the Kernel#system method to execute your command, which only returns true or false based on whether or not it can find the command and run it successfully. Instead you want to capture the output of the rspec command. Essentially you want to capture everything that rspec outputs to STDOUT. You can then iterate through the output to find and parse the line which will tell you how many examples were run and how many failures there were.

Something along the following lines:

require 'open3'
stdin, stdout, stderr = Open3.popen3('rspec spec/models/my_crazy_spec.rb')
total_examples = 0
total_failures = 0
stdout.readlines.each do |line|
  if line =~ /(\d*) examples, (\d*) failures/
    total_examples = $1
    total_failures = $2
  end
end
puts total_examples
puts total_failures

This should output the number of total examples and number of failures - adapt as needed.

skorks
  • 4,376
  • 18
  • 31
  • If you really want to use raw command line - I think it is better to wrap it into rake task. RSpec already has tasks to automate it - https://github.com/dchelimsky/rspec/blob/master/lib/spec/rake/spectask.rb – iafonov Aug 09 '11 at 06:22
  • You might want to consider the solution I mentioned below. No IO parsing. http://stackoverflow.com/a/10412793/1041418 – activars Jul 05 '12 at 22:04
0

This one prints to console and at the same time captures the message. The formatter.stop is just a stub function, I don't know what it is for normally, I had to include it to use DocumentationFormatter. Also the formatter output contains console coloring codes.

formatter = RSpec::Core::Formatters::DocumentationFormatter.new(StringIO.new)
def formatter.stop(arg1)
end
RSpec.configuration.reporter.register_listener(formatter, :message, :dump_summary, :dump_profile, :stop, :seed, :close, :start, :example_group_started)

RSpec::Core::Runner.run(['test.rb','-fdocumentation'])

puts formatter.output.string
Krzysiek
  • 185
  • 1
  • 10