I'm struggling with AutoIt regression tests. AutoIt was chosen because requirement is 100% blackbox GUI testing (no intervention to tested project). This solution has nevertheless got some issues, with which I need help.
Changing tabs in application - Because GUI class isn't SysTabControl32, but SWT_Window, example code from GUITab.au3 UDF doesn't work. Current solution is focusing on component and then navigating through tabs with arrow keys. This has a bad influence on test performance (and I don't even want to imagine a future possibility of some tab being disabled).
Timeouts - When tabs are changing, tests have to delay before they can proceed. The shorter the delay, the bigger the probability of test failure (app wasn't ready for test to continue). This leads to big delays before actions.
Instance numbers - Identifying instances of controls is a problem. When I wrote some test button
OKit had instance number 9. When some buttons were added to the form I had to rewrite the test, because instance number ofOKbutton changed because of this.
These three are most important.
Changing technology of testing would be hard because of big amount of already written tests. But I would like to write new test in a better way. Sikuli has problems getting text from screen and SWTBot has dependencies in tested projects.
Our tests run for 20 hours and when GUI layout changes I need to edit almost every test (instance number problem). Can anybody suggest a solution or workaround for ultra reliable blackbox testing?
