11

I'm a software engineer who will/may be hired as a firmware test engineer. I just want to get an idea of some software tools available in the market used in testing firmware. Can you state them and explain a little about what type of testing they provide to the firmware? Thanks in advance.

LEMUEL ADANE
  • 8,336
  • 16
  • 58
  • 72
  • 1
    I see a lot of job postings for embedded-style work want familiarity with [jtag](http://en.wikipedia.org/wiki/Joint_Test_Action_Group) debugging. – sarnold Mar 19 '11 at 01:46
  • 4
    @sarnold: JTAG itself is not a "test tool"; it simply refers to the physical connection and communication between a development host debugger and a processor incorporating on-chip debug hardware, avoiding the need for expensive external in-circuit emulator hardware, or less robust software based remote debug monitors. Anyone who puts it as a specific job requirement probably has little understanding of exactly what it is, and actually simply wants hardware debugger/ICE experience in general. Either way I don't think it is hugely relevant to the question in hand. – Clifford Mar 20 '11 at 10:26

4 Answers4

13

Testing comes in a number of forms and can be performed at different stages. Apart from design validation before code is even written, code testing may be divided into unit testing, integration testing, system testing and acceptance testing (though exact terms and number of stages may very). In the V model, these would correspond horizontally with stages in requirements and design development. Also in development and maintenance you might perform regression testing - ensuring that fixed bugs remain fixed when other changes are applied.

As far as tools are concerned, these can be divided into static analysis and dynamic analysis. Static tools analyse the source code without execution, whereas dynamic analysis is concerned with the behaviour of the code during execution. Some (expensive) tools perform "abstract execution" which is a static analysis technique that determines how the code may fail during execution without actual execution, this approach is computationally expensive but can process far more execution paths and variable states than traditional dynamic analysis.

The simplest form of static analysis is code review; getting a human to read your code. There are tools to assist even with this ostensibly manual process such as SmartBear's Code Collaborator. Likewise the simplest form of dynamic analysis is to simply step through your code in your debugger or even to just run your code with various test scenarios. The first may be done by a programmer during unit development and debugging, while the latter is more suited to acceptance or integration testing.

While code review done well can remove a large amount of errors, especially design errors, it is not so efficient perhaps at finding certain types of errors caused by subtle or arcane semantics of programming languages. This kind of error lends itself to automatic detection using static analysis tools such as Gimpel's PC-Lint and FlexeLint tools, or Programming Research's QA tools, though lower cost approaches such as setting your compiler's warning level high and compiling with more than one compiler are also useful.

Dynamic analysis tools come in a number of forms such as code coverage analysis, code performance profiling, memory management analysis, and bounds checking.

Higher-end tools/vendors include the likes of Coverity, PolySpace (an abstract analysis tool), Cantata, LDRA, and Klocwork. At the lower end (in price, not necessarily effectiveness) are tools such as PC-Lint and Tessy, or even the open-source splint (C only), and a large number of unit testing tools

Clifford
  • 88,407
  • 13
  • 85
  • 165
  • 1
    It is important to emphasize that the extremely expensive static analysers are no better than the ones that cost a 10th of the price. I'm using LDRA myself and it costs a fortune, but it doesn't perform any better than other tools. The difference is that LDRA is extremely trigger-happy and even with a large number of checks removed, it gives approximately 100 false errors for every real issue. It is a good idea to check the code with several different analysers, as they all have their flaws. – Lundin Mar 21 '11 at 07:34
  • @Lundin: Absolutely. I did mentions that, though perhaps not with emphasis. @Lemual wanted to know about available tools, I presume he wants to know what he may encounter. If her were tasked with selection of such a tool, that would warrant a different kind of answer. – Clifford Mar 21 '11 at 13:56
10

Here are some firmware testing techniques I've found useful...

  1. Unit test on the PC; i.e., extract a function from the firmware, and compile and test it on a faster platform. This will let you, for example, exhaustively test a function whereas this would be prohibitively time consuming in situ.

  2. Instrument the firmware interrupt handlers using a free running hardware timer: ticks at entry and exit, and count of interrupts. Keep track of min and max frequency and period for each interrupt handler. This data can be used to do Rate Monotonic Analysis or Deadline Monotonic Analysis.

  3. Use a standard protocol, like Modbus RTU, to make an array of status data available on demand. This can be used for configuration and verification data.

  4. Build the firmware version number into the code using an automated build process, e.g., by getting the version info from the source code repository. Make the version number available using #3.

  5. Use lint or another static analysis tool. Demand zero warnings from lint and from the compiler with -Wall.

  6. Augment your build tools with a means to embed the firmware's CRC into the code and check it at runtime.

Doug Currie
  • 40,708
  • 1
  • 95
  • 119
0

I have found stress tests useful. This usually means giving the a system a lot of input in a short time and see how it handles it. Input could be

  • A files with a lot of data to process. An example would me a file with wave data that needs to analyzed by a alarm device.
  • Data received by an application running on another machine. For example a program that generates random touch screen presses/releases data and sends it to a device of a debug port.

These types of tests can shake out a lot of bugs (particularly in systems where performance is critical as well as limited). A good logging system is also good to have to track down the causes of the errors raised by a stress test.

waffleman
  • 4,159
  • 10
  • 39
  • 63
0

We use a combination of unit testing, integration testing and system testing.

  • Unit testing is concerned with logical correctness and here we use branch coverage to make sure that all code paths through the program have been verified. Unit testing must be done using mocking and a framework for example like CMock. We have a Zephyr RTOS package that enables effective CMock testing of embedded firmware here: https://github.com/swedishembedded/testing
  • Integration testing is done by writing sample apps and tests that can be executed both on real hardware and on native posix. This is done by having a build system that supports native posix as a target and then using a test assertion library like Unity or ZTest (zephyr's native) to verify that conditions are met when running the code. This tests APIs and their behaviors when they are integrated with real hardware and other software components. The measure of success here is API requirement coverage: ie test cases targetting inputs and outputs to apis. No mocking.
  • System testing is concerned with testing the final firmware that is treated as a black box. The final firmware must be executed either on real hardware with a physical test bench or on simulated hardware on CI pipeline. This kind of test verifies the firmware against actual requirements which can be expressed as Gherkin language robot framework scripts.

Together, these three types of testing cover vast majority of requirements for quality assurance of embedded firmware. If you would like examples and more code that can help you implement all of these on Zephyr RTOS then you can always go through the code provided in the Swedish Embedded Platform SDK repository here: https://github.com/swedishembedded/sdk

Martin
  • 3,509
  • 3
  • 26
  • 31