Result Execution and Presentation when Blackbox Testing

Question

I have developed a Blackbox Test Environment in Python 3.2 for testing a piece of hardware. In this environment I have a TestExecution.py module where I run my tests as follows:

while(True):
    TestWithRestart("Test122")
    TestWithRestart("Test123",keys="invalid_keys.dat")
    TestWithOneComPort("Test200", keys="invalid_keys.dat")
    TestWithTwoComPorts("Test200")
    TestWithTwoComPorts("Test200", ppc_simulation_script="Test200.pcc")
    TestWithNoComPort()
    TestTime("Test500")
    Test600()
    TestWithComPortNoise("Test600")
    TestWithComPortInteruption("Test601")

Each hardware release I test is represented on my PC by its own Test Environment folder. This folder contains logs, keys and a TestExecution.py. Each Test Case has its own results folder and in this folder I have log folders for each execution of the test.

Its also possible that I need to design new tests for a new hardware release. In this case it can take numerous attempts until I get this test to work properly.

With regard to the Pass/Fail status of a test, I do this by manually checking within my log files. The next improvement will be to automate the process of establishing if a test passed or not. I will write separate classes for this. This process will be ongoing.

I'm wondering if I can integrate my environment with Continuous Integration Software with a view to presenting both test execution and/or results in a nice graphical form. It would also be nice to select the tests I wish to execute. What open source software would you recommend?

Thanks,

Barry

malenkiy_scot · Accepted Answer · 2012-02-13T13:54:07.840

1

Jenkins. For example, you can dump your test results in JUnit XML format and Jenkins will automatically produce nice graphs.

Plugins depend on your needs, of course, but here is a list of the essential plugins plus my favorites (some of them are bundled in the basic package):

Ant
A version control integration plugin (like Subversion, depends on what you are using)
Parameterized Trigger Plugin
Build Timeout Plugin
Log Parser Plugin
Regex Email Plugin
Artifact Deployer Plugin
Extended e-mail Plugin

As a Python programmer you will also benefit greatly from Python Jenkins API Wrapper.

In general, however, be careful with plugins: sometimes they are unstable and/or don't function properly. A look at plugin revision history usually can tell you if it is well-maintained.

You may install Jenkins locally on your machine and play with it for a few days before deciding if it fits your needs.

edited Feb 13 '12 at 13:54

answered Feb 13 '12 at 12:01

malenkiy_scot

16,415
6
64
87

What Jenkins plugins should I consider using? – Baz Feb 13 '12 at 13:21
Thanks! Am I right in saying that I should run my tests using the Python plugin Build step? – Baz Feb 13 '12 at 14:33
Not necessarily: you may want to start with scripts run in 'Execute Shell' (or use Ant if you are comfortable with it). See how far it takes you. I've never used Python plugin, so I can't vouch for it. I call Python (including the API Wrapper) from shell/Ant. – malenkiy_scot Feb 13 '12 at 14:54
I see. I have install sh.exe now on my windows machine. Have you any suggestions on how to run subsets of Test Cases? – Baz Feb 13 '12 at 15:42
Use 'Execute Windows Batch Command' build steps. Make sure to use %ENV_VAR% syntax instead of $ENV_VAR or ${ENV_VAR} in batch file steps. Running test subsets (modeled on unittest module): write scripts that generate and run custom test suits that filter test cases by your criteria provided as parameters. Those parameters then can be provided as Jenkins job parameters. This is just a suggestion - look for other ways as well. There is [jenkins] tag on StackOverflow. – malenkiy_scot Feb 13 '12 at 16:17

Result Execution and Presentation when Blackbox Testing

1 Answers1