How to test reliability of my own (small) embedded operating system?

Question

I've written a small operating system for embedded project running on small to medium target.

I added some automated unit test with a high test code coverage (>95%), but the scope is only the static part.
I got some code metrics as complexity and readability.
I'm testing my code with a rule checker with MiSRA support, and of course fixed all warnings.
I'm testing the code with a static analyzer and again fixed all warnings.

What can I do now to test - and improve - the reliability of my OS ? How about the dynamic part ?

Letting people actually use it isn't an option at this point? — Tim Post, May 23 '10 at 09:23
@Tim Post: Yeah, 2 projects are running it ... but I must provide some info about reliability ! I've tested only the static part. — TridenT, May 23 '10 at 09:48
There is no shell, it's just the scheduler and tasks synchronizations (mailbox, message, queue, semaphore etc.) — TridenT, May 23 '10 at 15:45
Verify reliability and response-time for multiple events in multi-threaded environment. It works statically with unit test, but writing tests with the whole dynamic is difficult, hard to find the scenarios ! — TridenT, May 27 '10 at 11:47

score 3 · Answer 1 · answered May 23 '10 at 10:59

3

Things missing in your list:

If you are not already doing it then also run the unit tests on the target hardware, to check for compiler and hardware issues.
Code reviews, check especially for race conditions

You could also review the generated assembly code, if it is not too big.

answered May 23 '10 at 10:59

starblue

55,348
14
97
151

Code reviews are good, but it's 'one shot'. The review for assembly code was partially done, but what results should I show ? – TridenT May 25 '10 at 11:29

score 1 · Answer 2 · answered May 23 '10 at 08:16

1

Check out the software-testing entry on wikipedia. It's a rather comprehensive description of the different branches of testing. You might find a new idea or two there.

answered May 23 '10 at 08:16

bitc

1,588
13
15

It's an interesting start, but I don't know how to handle the dynamic behavior of my OS. I will follow the link for investigation ! – TridenT May 26 '10 at 19:29

score 1 · Answer 3 · answered May 24 '10 at 00:09

1

Try Atomic Object's site. Try this also..

Also James Greening.

answered May 24 '10 at 00:09

Gutzofter

2,003
23
26

Mock technic is rather difficult in C (the OS is written in C + ASM) and heavy compared to C++. But it might be the only solution ! – TridenT May 26 '10 at 19:31

score 1 · Accepted Answer · answered May 24 '10 at 06:05

1

Try writing some unit tests for the dynamic part. Then run the tests on the target hardware. Run the tests on hardware with more cores Run the tests on hardware with only one core

Vary target system clock speed and run the dynamic tests.

should shake out most timing issues.

answered May 24 '10 at 06:05

Tim Williscroft

3,705
24
37

Maybe it's the only solution ! But write unit / integration test with timing evaluation, on the target, with multiple cores ... I will write unit test during 10 years ?!? – TridenT May 26 '10 at 19:33
Run the dynamic tests on the target. (1) experiment run the dynamic tests on development hardware with more or less cores (2) experiments. Run on target with different clock speed (1) experiment. A total of 5 experiments. – Tim Williscroft May 26 '10 at 23:33

score 1 · Answer 5 · answered Oct 29 '10 at 12:58

1

Seems like you have done a lot to test your system. I think the next step would get other projects or people to use it. Other users would quickly show you were robustness problems are.

answered Oct 29 '10 at 12:58

Gerhard

6,850
8
51
81

How to test reliability of my own (small) embedded operating system?

5 Answers5

Linked