TestFX tests pass headful but fail headless

Question

I am trying to write GUI tests for my university project.I am trying to do this using TestFX 4.0.4, but I am having an issue. For the project we are also obliged to use continuous integration with Travis. This means that I have to make the GUI tests headless. However, when I try to run my tests headless they fail, while they pass when they are headful.

This is the code that I use to run the test headless:

@BeforeClass
public static void setupHeadlessMode() {
    System.setProperty("testfx.robot", "glass");
    System.setProperty("testfx.headless", "true");
    System.setProperty("prism.order", "sw");
    System.setProperty("prism.text", "t2k");
    System.setProperty("java.awt.headless", "true");
}

The test I run clicks on the optionsButton, which switches the scene to the options menu and then has to click the musicButton to toggle whether music is playing. The scene switch might cause the failure in this case, but I don't know how I would avoid this. This is the test code:

@Test
public void testMusicPlaying() {
    clickOn("#optionsButton");
    assertThat((boolean) ConfigLoader.get("musicPlaying")).isTrue();
    clickOn("#musicButton");
    assertThat((boolean) ConfigLoader.get("musicPlaying")).isFalse();
    clickOn("#musicButton");
    assertThat((boolean) ConfigLoader.get("musicPlaying")).isTrue();
}