1

PROBLEM

I have a fairly large Hydro-Pneumatic system model composed of roughly 20-25 different subsystems. Each of the subsystem is then composed of digital logic, edge delay blocks, and gateway to external output ports (real-world outputs). Additionally some of the small building blocks are legacy C code imported as S-functions in order to maximise cost-benefit ratio. Each of the subsytem model has been designed in SIMULINK with the use of basic blockset i.e. no add-on commercial blocksets (e.g. aerospace, simscape, simMechanics, etc.) used.

The main issue is that I don't have sufficient knowledge of testing models in SIMULINK. I know that there is a bespoke unit testing framework that Mathworks provides for testing and verification. The trouble is that I am not quite sure how this will be relevant to my domain. Additionally, my subsystems are quite complex themselves and unit testing each of them is a nightmare. BUT if black-box testing with inputs and outputs is the only way, so be it and I am happy to accept that.

Although my question lies around testing large-scale and complex systems in SIMULINK, my aim is to get suggestions from experienced SIMULINK users who have done it in the past and will be doing it a lot in the future. I have done unit testing in SIMULINK before, but all those tests where not even near the vicinity of my correct design.

Any help will be appreciated!!

UPDATE SINCE @PHILGODDARD'S COMMENTS

I forgot to mention above that I am familiar with the HIL and PIL. However, those are valid only when you have your target real-time platform available for loop testing. What if someone wants to do a soft real-time testing?

I am waiting to go through the Mathworks large-scale system testing Webinar in a day or two. Hopfully, I can get some better answers/suggestions through that?

UPDATE SINCE @AM304's COMMENTS

To clarify the context a bit further, we are modelling all the interacting systems in software without involving any physical devices i.e. the modelling and simulation of all the systems in software with the outputs being displayed in operator/instructor terminals..... e.g. we have electrical, air conditioning, and hydro-pneumatic systems working together, but we have modelled them in software. therefore, when we simulate it, all the necessary signals are generated from the behaviour of the software models, no actual hardware/physical device is involved in delivering those behavioural outputs.

Andy Campbell
  • 2,177
  • 13
  • 15
ha9u63a7
  • 6,233
  • 16
  • 73
  • 108
  • The first thing I'd do is break the model up (if possible) into separate models; combining them at the highest level using Model blocks. This will facilitate writing test cases (i.e. developing separate test models) for each of the individual components. Unfortunately there's no magic bullet for creating the test models (but the tone of your post seems to indicate that you are aware of this). – Phil Goddard Sep 12 '13 at 17:01
  • @PhilGoddard Do you think HIL or PIL is an acceptable, but quite expensive answer for testing large-scale system? – ha9u63a7 Sep 12 '13 at 21:31
  • @PhilGoddard And also, if you look at my problem description, the entire system is broken down into subsystem models. Those require testing - Isn't that what you are suggesting too? Or am I missing your point? – ha9u63a7 Sep 12 '13 at 21:34
  • 1
    I took your original wording to mean that you were using generic subsystems, not that you were using Model blocks to break it up into separate models. Your original post doesn't mention targeting hardware, so I wouldn't have thought about using HIL or PIL approaches. I'd still opt for breaking the model up using Model blocks; then having separate models set up as test harnesses for each of the components; and writing MATLAB code to run/exercise the test harnesses with different sets of test inputs. This is essentially a roll your own version of some of the Mathworks testing tools. – Phil Goddard Sep 13 '13 at 04:18
  • 1
    Are you based in the UK? If so, it might be a good idea to go to the MATLAB Expo (http://www.matlabexpo.com/uk/) and you can talk directly to the developers, in particular the guy giving this talk: http://www.matlabexpo.com/uk/2013/embracing-complexity.html – am304 Sep 16 '13 at 09:33
  • @am304 Thanks for reminding me +1 :)....I am aware of it...as a matter of fact some of us are going to the advanced training course....I was simply trying to get comments about users who has done testing of large and complex systems...I was interested particularly in unit testing, without involving HIL and PIL.... – ha9u63a7 Sep 16 '13 at 16:38

3 Answers3

2

I would look into Simulink Design Verifier and Simulink Verification&Validation. Specifically, they offer features such as automated test generation and coverage analysis, which I think are applicable to your problem.

As a side note...

With regards to the matlab unit-testing framework which you mentioned in your post, this is basically a MATLAB implementation of the xUnit framework. So it obviously comes in handy if you want to be able to apply xUnit design patterns to tests of MATLAB code.

The framework can be used to write tests for a Simulink model, but keep in mind that you have to be able to write the tests in MATLAB code. So this just means making use of the Simulink command-line interface (e.g. commands like set_param and sim) to setup and exercise your model, then use the verification methods of the framework (e.g. verifyEqual) to compare actual and expected results.

grungetta
  • 1,089
  • 1
  • 7
  • 8
  • Thanks for your answer. Can I use the MATLAB unit testing framework for testing SIMULINK model though. I know it may seem obvious in your answer that it tests "Code" not "Model"...but worth a shot, right? Also, SIMULINK Design Verifier and SIMUINK Verification & Validation seem more like paid blocksets (i.e. add-ons) rather than something that you will get built-in....I was trying to avoid those, but if there is no other way, then I guess I need to take a step forward. – ha9u63a7 Sep 13 '13 at 07:18
  • +1 for suggesting Simulink V&V and Simulink Design Verifier. Simulink V&V is relatively cheap and provides a lot of functionality for model testing (coverage analysis, requirements linking, etc...). Simulink Design Verifier is a fair deal more expensive, but allows you to generate test vectors, prove properties in your model, ... – am304 Sep 13 '13 at 10:33
  • Yeah, those two products are definitely add-on toolboxes. Like I said, something worth looking into, but you'll ultimately have to decide if they really make sense for you based on your specific needs and the scale of your projects. – grungetta Sep 13 '13 at 20:38
  • For the matlab unit testing framework, it just depends on what you want to test in your models. The simplest case is you just want to run a model given certain inputs and verify the outputs are as expected. It's pretty straightforward to write code to do this. I don't know that it's fair to say that it *tests* code not models. The tests are written in code, but they test anything you want. You just have to be able to write code that interfaces with the thing that you want to test and then produces some sort of resulting data set which can be compared/verified against the expected results. – grungetta Sep 13 '13 at 20:44
  • Note that as of R2017b, MathWorks has transitioned Simulink V&V product. There is now a new product called Simulink Coverage that measures test coverage in models and generated code from the models. See https://www.mathworks.com/products/simulink-coverage.html – Jay Abraham Sep 25 '17 at 14:44
0

C-code as well as Simulink models can be tested using TPT. There is no difference in the test development as well as in the test assessment but in the test execution environment. For testing C-code one can decide if the C-code shall be tested in the so called SiL-mode in Simulink where the C-Code is embedded in Simulink as so called S-function. The generation of the S-function can be done automatically by m-scripts dependent of the code generator.

For the Simulink Coder may use the following MATLAB commands to force SiL:

set_param(<testFrameName>,'RTWSystemTargetFile','rtwsfcn.tlc');
set_param(<testFrameName>,'RTWTemplateMakefile','rtwsfcn_default_tmf');
rtwbuild(<subsystem to be tested>);

For TargetLink the script uses the TargetLink commands “tl_built_host” and “tl_set_sim_mode”:

tl_build_host('Model', <testFrameName>, 'TlSubsystems', <subsystem to be tested>);
tl_set_sim_mode('Model', <testFrameName>, 'TlSubsystems', <subsystem to be tested>, 'SimMode', 'TL_CODE_HOST');

Alternatively in TPT C-code can be tested in two other ways that do not require Simulink but some manual programming and compiling.

The first alternative is using the so called EXE-platform where a test harness is built in C-code and to be compiled by the user by its own compiler. The generation of the test harness is supported in the EXE-platform configuration dialog in TPT.

The second alternative is the so called FUSION platform which is a co-simulation environment. The FUSION is an open architecture where user can adapt their own system under test C-code using a well-defined and documented API. The c-code together with the API forms a so called node that can simulated at the FUISON platform as single node or together with other nodes. In case of automatic code generation nevertheless most user use MiL and SiL testing in Simulink because it can be done fully automatic including Back to back regression testing between MiL and SiL.

Why should one test C-code instead of models? The reason is that mostly the functional development is done in floating point notation while the implementation in the target ECU is to be done in fix-point notation. The process of scaling and the fix-point calculation is at least for TargetLink done during code generation. Thus the floating point (MiL) results are to be compared with the fix-point implementation (SiL) because from the experience the scaling introduces many errors.

If 100% statement or condition coverage there is a feature called TASMO in TPT. TASMO tries to generate test cases automatically to fulfill a maximum of coverage of a Simulink or TargetLink model. The algorithm is based on optimization and is search based. Please note that automatic test case generation should not replace functional test cases. The coverage goals can be checked by using code coverage tools like CTC++, V&V toolbox or TargetLink that brings its own coverage measurement. By looking at the coverage results the user may decide on its own how to stimulate the blind spots and can identify dead code.

I’m one of the TPT developer. For further information about TPT you can visit our website.

jens
  • 26
  • 3
  • It's okay @jens I am not getting into that chapter anymore. The problem I had was a convertion of C-based real-time simulator system into MATLAB/SIMULINK-equivalent system. Testing was a part of the investigation, which I failed in miserably. Even if it succeeded, I don't think it was possible to address the fact that several modules were running at different rates (e.g. 25Hz, 60Hz, etc.). SIMULINK wasn't the tool fit for purpose. – ha9u63a7 Mar 23 '15 at 12:42
0

MathWorks changed it's V&V product offering in R2017b (September 2017). I recommend taking a look at the following links to see how they can help with unit and integration level testing of designs in Simulink/Stateflow.

  1. Simulink Test - provides an environment to test your Simulink/Stateflow design
  2. Simulink Coverage - produces coverage metrics to determine how much of your design has been tested
  3. Simulink Design Verifier - automatically generates tests to meet functional or coverage criteria

There is an overview page on the MathWorks website that provides additional detail on the overall offering for verification, validation, and test.

Jay Abraham
  • 151
  • 1
  • 4