QNX - not a chance! QNX is a relatively large and sophisticated OS for 32bit devices with MMU, providing not only kernel level scheduling but also file systems, fault-tolerant networking, POSIX API, GUI etc. Its most significant feature is its support for memory protection - each thread runs in its own virtual memory segment, so only runs on devices with appropriate hardware support.
What features do you want from your OS? On an 8 bit device it is only reasonable to expect basic priority based pre-emptive scheduling and IPC. Other services such as networking, filesystem, USB etc. are usually add-ons from the RTOS vendor or must be integrated yourself from third-party code.
The obvious choice if you want to spend no money is FreeRTOS. It is competent, though in some ways unconventional architecturally, even if fairly conventional at the API level. In my tests on ARM it had slower context switch times that other kernels I compared it with others I tested, but that may not be the case on AVR, and would only be an issue if you require real-time response times in order of a few microseconds. AVR has a rather large register set, so context switches are typically expensive in any case.
Atmel have a list of third-party support including RTOS at http://www.atmel.com/products/AVR/thirdparty.asp#. They list the following:
- CMX Systems, Inc: CMX-RTX, CMX-Tiny+ (Add-ons: CMX-MicroNet, CMX-FFS)
- FreeRTOS.org: FreeRTOS
- Micriµm, Inc: µC/ OS-II
- Nut/OS: RTOS and TCP/IP stack with a Posix-like API.
- SEGGER: embOS
I have personal experience of CMX-Tiny+ (on dsPIC), embOS (on ARM), and FreeRTOS (on ARM), and uC/OS-II. They are all competent, uC-OS-II has the minor restriction of only allowing a single task at each priority level (no round-robin scheduling), but consequently probably faster context switches. In the case of embOS I have, successfully integrated third-party file-system and USB code, though the vendor has their own add-ons for these as well.