Use the same examples in multiple Cucumber scenario outlines

Question

How do I structure tests for the following program:

I'm writing a unit test framework for simulated combinatorial circuits. This framework will support multiple digital logic simulators (JLS, Logisim, TKGate, etc.) Thus, each test should be run once for each supported simulator.

My first idea is to do something like this:

Scenario Outline:  Test of valid circuit
  when I run DLUnit with "testCircuit1.<type> testFile"
  Then I should see "All tests (4) passed." on stdout
  Examples:
      | type |
      | jls  |  # extension for JLS files
      | circ |  # extension for Logisim files
      | v    |  # extension for tkgate files

Scenario Outline:  Test of invalid circuit
  when I run DLUnit with "brokenCircuit1.<type> testFile"
  Then I should see "There were failures" on stdout
  Examples:
      | type |
      | jls  |
      | circ |
      | v    |

 # Many more tests to follow

Although this will technically work, it results in feature code that may be difficult to maintain: Each feature is followed by a list of supported simulators. Adding support for an additional simulator would require adding the same line to each test.

I could also create jls.feature, then use sed to automatically create logisim.feature and tkgate.feature; but, I'd like to avoid that type of complexity if Cucumber offers a simpler built-in solution.

Maybe I don't get it but as far as I understand, you will only have each new simulator twice, to each one of the scenarios above which seems fine to me. — Eugene S, Aug 23 '16 at 01:46
I do not like referring to external files. It's hiding the decision rule. — Dave McNulla, Aug 23 '16 at 04:06
I only listed two examples above. I expect 10 to 20 scenarios when I'm done. — Zack, Aug 23 '16 at 13:43
@DaveMcNulla, which external files are you referring to specifically? The input data files (`brokenCircuit.[jls,circuit,v]` and `testFile`)? — Zack, Aug 23 '16 at 13:44
To complete my comment: @zach that's what I'm talking about, the files getting run. If the contents are not relevant then don't mention them, hide it in the step. If what is in the is relevant, put it in the step definition text. You can always create the file thay gets run. — Dave McNulla, Aug 23 '16 at 14:49
Does it have to be Cucumber? RSpec, for example, allows you to define test cases at runtime. — Dave Schweisguth, Aug 24 '16 at 11:59
@DaveSchweisguth, RSpec could work also. I used RSpec with a previous project and found that the tests weren't quite as readable as I wanted. But, perhaps I should give RSpec another shot. — Zack, Aug 29 '16 at 22:16

score 1 · Accepted Answer · answered Aug 29 '16 at 23:25

Perhaps you could do something like this in RSpec:

describe DLUnit do
  [
    'jls', 'testCircuit1', true,
    'jls', 'brokenCircuit1', false,
    # ...
  ].each do |simulator, circuit, expected_validity|
    it "with the #{simulator} simulator finds the #{circuit} circuit #{expected_validity ? 'valid' : 'invalid' }" do
      actual_output = DLUnit.run "#{circuit.simulator}" # obviously I'm making this part up
      expect(actual_output).to include(expected_validity ? 'passed' : 'failures')
    end
  end
end

The test code itself is a little involved, but you only have to write it once, and the RSpec output should be clear.

In this case, using RSpec is the way to go. The tests don't read quite as nicely as with Cucumber; but, overall, RSpec is much simpler. I did have to write a couple custom matchers to get the specs to read nicely; but, that was a small price to pay for all the other benefits. — Zack, Sep 06 '16 at 15:08

score 0 · Answer 2 · answered Aug 22 '16 at 20:46

Not much of an upgrade to what you already have but how about merging into one scenario outline. Addition of new simulator you will need to make two changes in one examples table. Also you can make it more configurable in terms of changes in a different valid test file for two simulators or different result messages. But same can be done for your existing solution all be it you will have to do change the steps and examples.

Scenario Outline:  Testing circuit
  when I run <kind> DLUnit with "<circuit>.<type> testFile"
  Then I should see <result> on stdout

      Examples: 
      | type | kind    | circuit         | result               |
      | jls  | valid   | testCircuit1    | All tests (4) passed |
      | jls  | invalid | brokenCircuit1  | There were failures  |
      | circ | valid   | testCircuit1    | All tests (4) passed |
      | circ | invalid | brokenCircuit1  | There were failures  |
      | v    | valid   | testCircuit1    | All tests (4) passed |
      | v    | invalid | brokenCircuit1  | There were failures  |

Not a bad idea; but (as you suggested), it might be more of a "lateral move" than an improvement. My main concern is that the scenario name no longer clearly indicates what is being tested. I listed two example tests above; but, I'll be adding many more tests to verify that different types of failures are correctly recognized and reported. — Zack, Aug 23 '16 at 13:47
As you mentioned you could have 10 to 20 scenarios for each simulator, you got to decide whether you will prefer to run all the tests for a simulator together or one kind of a test for all simulators. And then split the scenarios. The approach I mentioned would be too cumbersome as the examples section will explode. — Grasshopper, Aug 23 '16 at 14:18

Use the same examples in multiple Cucumber scenario outlines

2 Answers2