5

My GUI unit-tests with Assertj Swing (version 3.17.0) all pass locally, but sometimes they fail in a CI server. If i retry long enough, eventually the test suite becomes green. I'm having a hard time figuring out how to fix these tests.

I am using Java 8 and Github Actions as CI. As suggested in a similar question, I am using VNC to run these tests headlessly on CI, with the command:

./execute-on-vnc.sh mvn -B -f pom.xml clean verify

Here's an example of a test that is failing (I am mocking the controller and just verifying that clicking a button actually calls the controller method with the correct arguments):

@Mock private ItemController controller;

@Test @GUITest
public void testAddItemButtonShouldDelegateToControllerAddItem() {
    window.textBox("itemIdTextField").enterText("1");
    window.textBox("itemNameTextField").enterText("Some Item");
    window.button(JButtonMatcher.withName("addItemButton")).click();
    verify(controller).addItem(new Item("1", "Some Item"));
}

Here's the very straight forward code for handling clicks on the JButton:

addItemButton.addActionListener(e ->
    controller.addItem(new Item(itemIdTextField.getText(), itemNameTextField.getText()))
);

The test passes locally every time with no issues, but often fails on CI with this error:

testAddItemButtonShouldDelegateToControllerAddItem(com.example.view.swing.ItemSwingViewTest)
Time elapsed: 0.985 sec  <<< FAILURE!
Wanted but not invoked:

controller.addItem(
  Item{id='1', name='Some Item'}
);

Actually, there were zero interactions with this mock.

I have not been able to figure out the cause of these flaky tests.

Things I've tried:

  • Added 5 seconds timeouts to verify() methods
  • Invalidated CI maven cache and rerun tests
  • Tested on CI with other Java versions (9, 11 and 13)

All with no luck. Am I doing something wrong?

John Doe
  • 141
  • 10
  • I'm working on your same problem, maybe the problem it is that in Github action it not allowed the UI rendering? – vincenzopalazzo Mar 27 '21 at 21:37
  • Any news on this? I'm having a similar problem https://stackoverflow.com/questions/70758763/swing-ui-tests-on-github-actions – ClaudiaR Jan 18 '22 at 18:17

0 Answers0