How to select the test suites dynamically in soap ui based on the command line argument value

Question

In our organization we are using SOAP UI 5.2.1 (open source). We have a soapui-project file as below

Project1.xml:

  Test Suite 1 (Smoke, Functional, Regression)
  Test Suite 2 (Functional)
  Test Suite 3 (Smoke, Functional)
  Test Suite 4 (Regression)

The same Project1.xml file will be used for Smoke, Functional and Regression test. Currently all the suites are running for all the types of test which is time consuming.

Is there any way to pass a parameter from command line and based on that corresponding test suite should be run

If my run command is like "testrunner Project1.xml ", Only test suites with Functional tag should run

1. How to tag each tets suite as provided above
2. what should be the run command which accepts the value of the run mode.

Is there any other way we can achieve this?

Answer 1

Create category tags in three places: each test is going to be tagged with a category in a property called categories. Then at the project level, we want to specify properties includeCategories and excludeCategories, which will list categories to include and categories to exclude, respectively. Each of these will be just a comma-separated list. The following script must be copied as a startup script in every single testcase. If you pay the $500 for a license, then you only need this script once as a custom event - I discussed that here.

def testCategories = []
def excludeCategories = []
def includeCategories = []

def tstCats = testRunner.testCase.getPropertyValue("categories")
def exCats = testRunner.testCase.testSuite.project.getPropertyValue("excludeCategories")
def inCats = testRunner.testCase.testSuite.project.getPropertyValue("includeCategories")

if(tstCats != null) tstCats.split(',').each { testCategories << it.trim() }
if(exCats != null) exCats.split(',').each { excludeCategories << it.trim() }
if(inCats != null) inCats.split(',').each { includeCategories << it.trim() }

The first three lines each define an empty List. Next three lines read all the properties in, and the last three lines parse them. The above code expects that the categories property just specifies something like: category1, category2, etc. - no quotes, no brackets.

First let’s deal with exclude categories. The meaning of these is usually that if a test is tagged with any of these, do not run it.

// exclude categories
excludeCategories.each {
    if(testCategories.contains(it))
        testRunner.cancel("${context.testCase.name} TestCase cancelled; excludeCategory = ${it}!")
}

And now the include categories. The meaning of these is that if the test is not tagged, skip it. Only if a test is tagged with any of these, then run it.

// include categories
if(includeCategories.size() != 0) {
    def cancelTest = true
    includeCategories.each {
        if(testCategories.contains(it))
            cancelTest = false
    }

    if(cancelTest)
        testRunner.cancel("${context.testCase.name} TestCase cancelled; includeCategories = ${includeCategories}!")
}

Answer 2

Here is how I would go about it.

Get user input thru command line(for testrunner utility) using
system arguments(-Darg="value"), and we are only enabling the suites as per the input, and disable the rest of the suites.

Having said that, to execute only Regression, use -DEXECUTION_GROUP="Regression".

For each test suite, create a custom property, say EXECUTION_GROUP and define values as you wanted. Say:

Test Suite 1 : property name EXECUTION_GROUP and value as Smoke,Functional,Regression Test Suite 2 : property name EXECUTION_GROUP and value as Functional
Test Suite 3 : property name EXECUTION_GROUP and value as Smoke,Functional
Test Suite 4 : property name EXECUTION_GROUP and value as Regression

Now, double click on the project, you would find Load Script button on right hand side below. And add the below script to it, then save the project.

What does the script do?

This script reads System property say EXECUTION_GROUP as command line options
And toggle to test suites according to the user input automatically.
If list is found enable only those suites, disable the rest of the suites, otherwise.
If system property is not set or not passed in the command-line, then all the suites are enabled.

Project Load Script:

/**
* this script reads System property say EXECUTION_GROUP as command line options
* and toggle to test suites according to the user input
* if list is found enable the suite, disable it otherwise.
* if system property is set, then all the suites are enabled
**/ 
//Below closure toggle the suite based on the list of names
def toggleSuite = { suite, list ->
   def groups = suite.getPropertyValue('EXECUTION_GROUP').split(',').collect{it.trim()}
   def isGroupFound = false
   list.each { group -> 
      if (groups.contains(group)) {
          isGroupFound = true
      }
   }
   if (!isGroupFound) {
      suite.disabled = true
   } else {
      suite.disabled = false
   }    
}

//Reads the system property
def userInput = System.getProperty('EXECUTION_GROUP')
log.info "Command line input: $userInput"
def cmdLineOptions = []

//Checks if the user provided value is null i.e., system property not set
if (null != userInput) {
    cmdLineOptions = userInput.split(',').collect{it.trim()}
    if (null != cmdLineOptions) {
        log.info "User has provided the execution group as input"
        log.info cmdLineOptions
        project.testSuiteList.each { suite -> toggleSuite(suite, cmdLineOptions) }
    } else {
        log.info "Receieved empty list of options, so disabling all the suites"
        project.testSuiteList.each { suite -> suite.disabled = true }
    }
} else {
    log.info "All suites are being enabled as no system property input found"
    project.testSuiteList.each { suite -> suite.disabled = false }
}

How to use it command line?

I believe you know how to call a project command-line (based on the question), i.e., using testrunner.bas/.sh of SOAPUI_HOME/bin.

while calling above testrunner utility, add EXECUTION_GROUP as system argument at the end.
i.e.,For example, to execute Smoke and Functional groups
add -DEXECUTION_GROUP="Smoke,Functional" at the end of the command.

So, command may look like:
testrunner.bat <options> project.xml -DEXECUTION_GROUP="Smoke,Functional"

You would see in the log which groups are being called(refer to log statements in the above script).

Well, how do I test above from within SoapUI?

The above mentioned behavior is applicable to only command line execution as per the request in the author's question. Then you may think, what happens when you run it from with in SoapUI? You are right, that will not behave as same when you run from with SoaUI. The reason being it does not get system argument EXECUTION_GROUP value. Usually you may not need it, but how do you realize if that is working or not, may be you want to test it before use.

But there is a trick if you want to have the same behavior from within SoapUI. That is, just add below statement in the top (that sets the system argument, be aware that this statement should be limited to test, otherwise defeat the actual purpose and uncomment it after the tests) System.setProperty('EXECUTION_GROUP', 'Smoke') and run the script.

And you would see the unwanted test groups disabled.