Why is Spring State Machine PersistStateChangeListener component sometimes not executing during same thread?

Question

Long story short:

We have a Spring State Machine implementation that will persist with a different request (on same thread from thread pool), causing a state error within our code. Why does it not run synchronously as part of same request?

Much longer story:

Our state machine is set up such that when an event is triggered and the state transition is successful/valid and it persisted, then work is done. If it fails either, we throw an exception. Persistence ironically is not enforced in Spring State Machine, so we had to do our own thing to enforce it.

We have a config class:

@Slf4j
@Configuration
@EnableStateMachine(name = "myStateMachine")
public class StatemachineConfiguration extends StateMachineConfigurerAdapter<String, String> {

    @Override
    public void configure(StateMachineTransitionConfigurer<String, String> transitions) throws Exception {
     transitions
            .withExternal().source(States.INITIAL.name()).target(States.IN_PROGRESS.name())
            .event(Events.START.name())
            .and()
            ...more transitions
    }

    @Override
    public void configure(StateMachineStateConfigurer<String, String> states) throws Exception {
        states
            .withStates()
            .initial(States.INITIAL.name())
            .state(States.IN_PROGRESS.name())
            ...more states;
    }

    @Override
    public void configure(StateMachineConfigurationConfigurer<String, String> config) throws Exception {
        config.withConfiguration()
            .taskExecutor(new SyncTaskExecutor()) // This should be default, but I added it anyways
            .autoStartup(false);
    }
}

To trigger events we are using a wrapper to enforce persistence prior to continuing on. Without the wrapper, we were continuing without actually having successfully saved the state entity. This was causing us to run into state transition errors when we later tried to trigger events, but the database had the old state.

StateEntityWrapper wrapper = new StateEntityWrapper(stateEntity);
boolean success = persistStateMachineHandler.handleEventWithState(
    MessageBuilder.withPayload(event.name()).setHeader(Constants.PERSIST_ENTITY_WRAPPER_HEADER, wrapper).build(),
    stateEntity.getState().name()
        );

if (!(success && wrapper.isPersisted()) {
    throw new StateTransitionException();
}

// Transition saved and successful - begin work
// Trigger other events depending on what happens during work

Our PersistStateChangeListener:

@Slf4j
@Component
@RequiredArgsConstructor
public class MyPersistStateChangeListener implements PersistStateChangeListener {

    private final MyStateRepository repository;

    @Override
    public void onPersist(State<String, String> state,
                          Message<String> message,
                          Transition<String, String> transition,
                          StateMachine<String, String> stateMachine) {
        StateEntityWrapper wrapper = message.getHeaders().get(Constants.PERSIST_ENTITY_WRAPPER_HEADER, StateEntityWrapper.class);
        if (wrapper != null) {
            StateEntity entity = wrapper.getStateEntity();
            entity.setState(States.valueOf(state.getId()));
            this.persist(entity);
            wrapper.setPersisted(true);
        } else {
            String msg = "StateEntityWrapper is null. Could not retrieve " + Constants.PERSIST_ENTITY_WRAPPER_HEADER + ".";
            log.warn("{}", msg);
            throw new MyPersistRuntimeException(msg);
        }
    }

    private void persist(FulfillmentState entity) {
        try {
            repository.save(entity);
        } catch (Exception e) {
            throw new MyPersistRuntimeException("Persistence failed: " + e.getMessage(), e);
        }
    }
}

I found that exceptions thrown by the persist listener are caught by Spring State Machine in AbstractStateMachine::callPreStateChangeInterceptors and will skip the state change. However, the call to PersistStateMachineHandler::handleEventWithState will return true regardless of not being able to save. This is because the state machine interceptors (persist listener is one of these) can be set up to be asynchronous. However, according to documentation, the default is supposed to be synchronous.

If the default is synchronous, then why does it sometimes execute after the PersistStateMachineHandler returns? We use Sleuth in our logging and I noticed that usually it has the same trace id during persistence. But occassionally, I see it execute with another request and its trace id. Lastly, I found DefaultStateMachineExecutor manages a TaskExecutor and should run synchronously based on default configurations, but sometimes does not. Any ideas?