Atomikos + Spring + JMS read messages batch transactionally

Question

I am trying to design an application using spring + atomikos that reads a batch of N messages and persists them to a database within a single transaction. It has to be a batch of messages because the data is only consistent when it's in batches, i.e., a single message is not enough data for a consistent transaction. Moreover having one transaction per message would absolutely kill my performance. This is not the typical JMS + DB application so I'm having a hard time finding examples online (I tried with a MessageListener as suggested on the atomikos website but that creates one transaction per message). What's the best way to achieve this using Spring? Thanks

Giovanni

score 0 · Answer 1 · answered Feb 16 '13 at 14:31

0

I think your problem has nothing to do with Spring or Atomikos. It's more a design issue.

If you need all messages to be commited within a single transaction, merge them all in a bigger structure before sending the full message to your persistence layer.

answered Feb 16 '13 at 14:31

Frederic Conrotte

754
4
12

Technically, it should be possible to use Spring+Atomikos to perform the following steps: start a transaction, read N messages **sequentially** from a queue, persist the data in (one or more) database(s) and then commit the transaction. Obviously I expect Spring to manage the transactions: start/commit/rollback. I know how to persist data in batches in multiple databases using the @Transactional annotation, but I can't figure out how have the JMS queue be part of the same transaction. This is the technical challenge: it has nothing to do with design. – Giovanni Botta Feb 19 '13 at 17:18
About merging the batches in a single message, I thought about that. Given that I'm not sure it will be feasible for me, is there an out of the box solution for JMS to "pack" messages together? – Giovanni Botta Feb 19 '13 at 17:20
"I can't figure out how have the JMS queue be part of the same transaction." To have both database and JMS part of the same transaction, you need the XA protocol, aka Two Phase Commit http://en.wikipedia.org/wiki/Two-phase_commit_protocol See also this link on how to configure Atomikos with JMS using XA: http://www.atomikos.com/Documentation/SpringIntegration#Receiving_JMS_Messages Here is the doc regarding Atomikos with XA protocol using JDBC: http://www.atomikos.com/Documentation/ConfiguringJdbc – Frederic Conrotte Feb 20 '13 at 16:03

score 0 · Accepted Answer · answered Feb 20 '13 at 19:01

Figured out what I need to do. In my spring configuration file, I need to configure the following:

an instance of org.springframework.transaction.jta.JtaTransactionManager class (configured for example to use atomikos) with the "transactionManager" and "userTransaction" dependencies injected, e.g., with instances of the classes com.atomikos.icatch.jta.UserTransactionManager and com.atomikos.icatch.jta.UserTransactionImp;
the <tx:annotation-driven transaction-manager="txManager"> tag to use annotation-driven transactions in the client code;
register the JMS connection factory with the transaction manager, e.g., by creating an instance of com.atomikos.jms.AtomikosConnectionFactoryBean;
a queue object, e.g., an instance of com.tibco.tibjms.TibjmsQueue or similar;
a JmsTemplate with the dependency "defaultDestination" injected with the previously created queue;
register the database(s) with the transaction manager, e.g., by creating an instance of com.atomikos.jdbc.AtomikosDataSourceBean (or com.atomikos.jdbc.nonxa.AtomikosNonXADataSourceBean for a last resource gambit scenario);
one or more SqlUpdate (or BatchSqlUpdate or JdbcTemplate) instances (one per data source) each with the "dataSource" property injected with the previously defined data source(s);

an instance of a custom DAO class with injected the previously defined JmsTemplate and SqlUpdate objects: this class will have one or more methods annotated @Transactional that will perform transactions using the JmsTemplate to receive messages and the SqlUpdate(s) to persist them on the databases, e.g., it could be a class that looks like this:

public class MyDao{        
  JmsTemplate jmsTemplate; // getter/setter omitted for clarity
  BatchSqlUpdate sqlUpdate; // getter/setter omitted for clarity

  @Transactional
  public void persistMessages(int n){
    // map used for the sqlUpdate object
    Map<String, Object> params = new HashMap<>();
    // need to reset this since it's being reused
    sqlUpdate.reset();
    for(int i=0;i<n;i++){
      // retrieve a message synchronously
      Message msg = jmsTemplate.receive();
      // transform the message
      doSomeMagic(msg,params);
      // set parameters in the SQL
      sqlUpdate.updateByNamedParam(params);
    }
    // now the batch can be flushed
    sqlUpdate.flush();
  }

  private void doSomeMagic(Message msg, Map<String,Object> params){
    // implementation is application-dependent!
    // the only assumption is that somehow the
    // message can be used to set the named
    // parameters in the sqlUpdate object
  }
}

The only thing worth noting is that the DAO must be spring-managed because the use of annotations requires spring to create a proxy.

Atomikos + Spring + JMS read messages batch transactionally

2 Answers2