4

We have a data acquisition system made of many programs and scripts running together, some sequentially, most in parallel, on different machines.

We use Jenkins to build and run unit tests.

We now want to test the system as a whole by using a data generator as input to the system and checking the output of the data acquisition chain.

Simplified view of the steps to test the system :

  1. Restore system (database, packages in the repository, relaunch virtual machines)
  2. Configure the system as needed (write config files to be used, update database)
  3. Launch the few parent processes needed and wait for them to be ready
  4. Start data input
  5. Once done, collect metrics and results, compare with expected results, check for error messages in the logs, ...

Ideally we shall run it with different configurations and input.

What tool would you use to do it ? Or would you try to do it directly in jenkins ?

Barth
  • 15,135
  • 20
  • 70
  • 105

4 Answers4

4

You don't tell us enough about your setup to give you specific answers. It strongly depends on the language/environment you are using. You probably need a combination of one or more of the following:

  • Vagrant/Chef or Puppet/Veewee/VirtualBox to automatically build VMs and configure and run them. That's assuming a Linux platform. Chef + Linux containers is a lot faster, especially if you run it on a ram disk (it is supposed to be a repeatable process, so only the artifacts need to be stored)
  • A package cache with version control for all the system packages you need to install on the VMs
  • Some framework to run acceptance tests, Cucumber, Robot or so. If your unit tests are good, you might be able to get away with smoke tests.
  • A simulation of your system, that you can feed run-time results from the log so you know how and when to scale.

You are very likely to run into the slow tests problem as you have to populate a database, and might have to split test jobs. In general, the feedback loop to get this working is way too slow, and you'll probably need to be very creative to get an acceptable cycle time.

In different environments, there are different tools for this, and having multiple stacks is not much fun. Jenkins runs on a Java stack, Puppet & Chef on a Ruby stack, and there are similar tools for the Python, c & perl stack. You'll have to decide for yourself if you want to have best-in-class with integration problems, or a single stack and coding things yourself. Both can work, and neither is trivial. Team experience is most likely the deciding factor there.

The feedback loop in DevOps is a lot worse than that of a modern CI development environment, so you'll have to do much more planning & research up front. The number of experiments you can run in a day is much lower, at least a factor 20 or so, so making things work the first time is going to be difficult. Make sure you have experienced people doing that.

Stephan Eggermont
  • 15,847
  • 1
  • 38
  • 65
2

The answer is: Robot Framework (see 1 or 2).

  • Supports Data Driven Tests and Behavioural Tests
  • Tests are written using user-defined keywords, in a human-like language. Easy to develop, easy to read, easy to maintain.
  • Lot of libraries to access different functionality (ssh connections, database, .....)
  • Can be extended with custom libraries written in python, java or other languages.
  • For each test a detailed log is generated

I don't know Jenkins, but I am pretty sure you can integrate Robot Framework with it. It's as simple as running a Python script.

OGrandeDiEnne
  • 872
  • 7
  • 14
2

Depending on the complexity of your system you can use a pletora of tools. Generically, you need some sort of job flow with visualization of results at the end. Instead of giving a deifnitive answer, let me point you to some tools/plugins.

Jenkins:

  • The MultiJob plugin let's you specify the workflow into phases, with parallel jobs phase
  • The BuildFlow plugin let's you create your own build flow with parallel and dependent jobs in a simple DSL.
  • The Plot plugin to collect metrics and plot these over time
  • The xUnit plugin to capture JUnit style xml or JSON from the tests, for instance generated by a little script or an xUnit framework such as py.test for Python or something more fancy.
  • RobotFramework plugin related to OFrandeDiEnne's answer.

Vagrant: Vagrant is basically a command-line interface on top of VirtualBox VMs. You can use this to start multiple VMs simultaneously (can even be done from Jenkins). Use a provisioner such as Puppet, Chef, SaltStack to setup your database, configuration, etc.

Build flow tools, such as offered by Run-Time Design Automation (RTDA). Commercial. This will most likely be an overkill.

Martijn Rutten
  • 763
  • 6
  • 14
  • Right, but this is the theory. I am more interested on real life experiences. Specially, **I would like to know about the problems and limitations people found when using those tools** to test complex systems. In example: do they escalate? are the tests easyly maintainable? can I parametrize them (i.e. test for several OSs, architectures, etc)? does parametrization and visualization go well together? etc. Which framework to use for handling the virtual machines, how much integration code will I be writing? – salva May 07 '13 at 10:36
  • I feel this set of tools could work to set up a test of a complicated multi-component system, but it would not be something homogeneous, maintainable, easy to understand, etc. – OGrandeDiEnne May 10 '13 at 21:50
  • Robot Framework is a tool designed to write Acceptance tests, not specifically tight to any system architecture or workflow. I use it to check a system composed by ~20 network components, with hundreds of tests. – OGrandeDiEnne May 10 '13 at 21:52
  • 1
    A similar tool, and probably more famous, is Cucumber [link](http://cukes.info) . I am not familiar with it so I can't compare. Cucumber is based on Ruby and originally used for WebApps. – OGrandeDiEnne May 10 '13 at 21:56
1

If your test servers have OpenSSH or some other SSH server service, you could start the test via SSH Chanel.

For example, in Java language:

  1. You could use TestNG or JUnit to organize the test cases, both are easily integrated with Jenkins.
  2. In the test codes, import JSCH and create your own ssh client to execute your command lines on the remote servers. Or push the command line to plink.exe (part of Putty) or some other 3rd party ssh clients which would also be used to execute your command line. For parallel execution, just use multithreads to push the command.
  3. Check returned code and messages.

In same way, you could also use Robot Framework to do this, which is also easily integrated with Jenkins(There is a 3rd party plugin for this). And it also has ootb ssh library.

Even more, if your application does not have command line entry, you could also use AutoIT library to operate the application via GUI.

SUT
  • 384
  • 1
  • 7
  • 23