6

I have an architectural question about how to handle big tasks both transactional and scalable in Java/Java EE.

The general challenge

I have a web application (Tomcat right now, but that should not limit the solution space, so just take this to illustrate what I'd like to achieve). This web application is distributed over several (virtual and physical) nodes, connected to a central DBMS (MySQL in this case, but again, this should not limit the solution...) and able to handle some 1000s of users, serving pages, doing stuff, just as you'd expect from your average web-based information system.

Now, there are some tasks which affect a larger portion of data and the system should be optimized to carry out these tasks reasonably fast. (Faster than processing everything sequentially, that is). So I'd make the task parallel and distribute it over several (or all) nodes:

Vision - what I'd like to have

(Note: the data portions which are processed are independent, so there are no database or locking conflicts here).

The problem is, I'd like the (whole) task to be transactional. So if one of the parallel subtasks fails, I'd like to have all other tasks rolled back as a result. Otherwise the system would be in a potentially inconsistent state from a domain perspective.

Current implementation

As I said, the current implementation uses Tomcat and MySQL. The nodes use JMS to communicate (so there is a JMS server to which a dispatcher sends a message for each subtask; and executors take tasks from the message queue, execute them, and post the results to a result queue from which the dispatcher collects the results. The dispatcher blocks and waits for all results to come in and if anything is fine, it terminates with an OK status.

The problem here is that all the executors have their own local transaction context, so the picture would look like this:

Current implementation

If for some reason one of the subtasks fails, the local transaction is rolled back and the dispatcher gets an error result. (There is some failsafe mechanism here, which tries to repeat the failed transaction, but let's assume for some reason, the one task cannot be completed). The problem is that the system now is in a state where all transactions but one is already committed and completed. And because I cannot get the one final transaction to finish successfully, I cannot get out of this state.

Possible solutions

These are the thoughts which I have followed so far:

  • I could somehow implement a domain-specific rollback mechanism myself. Because the distributor knows which tasks have been carried out, it could revert the effects explicitly (e.g. storing old values somewhere and revert already committed values back to the previous values). Of course, in this case, I must guarantee that no other process changes something in between, so I'd also have to set the system to a read-only state, as long as the big operation is running. More or less, I'd need to simulate a transaction in business logic ...

  • I could choose not to parallelize and do everything on a single node in one big transaction (but as stated at the beginning, I need to speed up processing, so this is not an option...)

  • I have tried to find out about XATransactions or distributed transactions in general, but this seems to be an advanced Java EE feature, which is not implemented in all Java EE servers, and which would not really solve that basic problem, because there does not seem to be a way to transfer a transaction context over to a remote node in an asynchronous call. (e.g. section 4.5.3 of EJB Specification 3.1: "Client transaction context does not propagate with an asynchronous method invocation. From the Bean Developer’s view, there is never a transaction context flowing in from the client.")

The Question

Am I overlooking something? Is it not possible to distribute a task asynchronously over several nodes and at the same time have a (shared) transactional state which can be rolled back as a whole?

Thanks for any pointers, hints, propositions ...

Arjan Tijms
  • 37,782
  • 12
  • 108
  • 140
Stefan Winkler
  • 3,871
  • 1
  • 18
  • 35
  • May be the [Parallel Pipeline pattern](http://www.amazon.com/Software-Pipelines-SOA-Multi-Core-Processing/dp/0137137974) can help you in this case. – Gabriel Aramburu Jun 30 '14 at 22:28
  • 1
    Do you have any pointer which is more concise/specific than a whole book? ;-) Maybe a summary somewhere, about the parallel pipeline pattern is working (and especially dealing with transaction contexts). – Stefan Winkler Jul 02 '14 at 10:50

1 Answers1

1

If you want to distribute your application as described, JTA is your friend in Java EE context. Since it's part of the Java EE spec, you should be able to use it in any compliant container. As with all implementations of the spec, there are differences in the details or configuration, as for example with JPA, but in real life it's very uncommon to change application servers very often.

But without knowing the details and complexity of your problem, my advice is to rethink if you really need to share the task execution for one use case, or if it's not possible and better to have at least everything belonging to that one use case within one node, even though you might need several nodes for the overall application. In case you really have to use several nodes to fulfill your requirements, then I'd go for distributed tasks which do not write directly to the database, but give back results and then commit/rollback them in the one component which initiated the tasks.

And don't forget to measure first, before over-engeneering the architecture. Try to keep it simple at first, assuming that one node could handle it and then write a stress test which tries to break your system, to learn about the maximum possible load it can handle with the given architecture.

Alexander Rühl
  • 6,769
  • 9
  • 53
  • 96
  • Thank you for reading and answering ;-) Also, your remarks about over-engineering are appreciated and welcome in general. Let's--for the context of this question--assume that the load actually has to be distributed over several nodes. Regarding JTA and XATransactions--as far as I understand, these are more useful when a transaction has to be coordinated for several databases (or more general: resources), and not so much for the case where a single database transaction (or the application/framework equivalent thereof) has to be shared among several collaborators? – Stefan Winkler Jun 30 '14 at 16:27
  • So the question is, why is it necessary to start the transaction before starting the tasks? Wouldn't it be feasible to do the calculation in memory, collect the results and then have the database transaction if all came back? Alternatively, couldn't you have the task restults stored individually in a task result table and when collecting results store the actual result? In both cases it wouldn't need for several nodes taking part in one transaction. The task result table could then be cleaned by a maintenance job periodically. – Alexander Rühl Jul 01 '14 at 12:29
  • Well, actually, the software is already written (by an external company). They use Tomcat/Spring/Hibernate and there are some batch jobs which run periodically. The jobs generally get the IDs for the entities to handle, package them up and distribute the tasks to parallel workers on the nodes. Those tasks will load the hibernate entities, do some external work (e.g., create PDFs of some sort) and then update the entities (link the PDFs to the entities). We had issues with failing jobs leaving inconsistent states, so I am currently investigating if things could have been done in a better way. – Stefan Winkler Jul 02 '14 at 10:56
  • That said, I like the idea of writing an intermediate state and returning it either via JMS message or a database table. The only thing is that a job can change several 100k rows in one run. So there's quite some data which need to be updated. And you either need to bypass Hibernate and do an `UPDATE ... JOIN ...` directly on the DB level (bypassing frameworks sounds a bit fishy ...), or you'd have to iterate over Hibernate entities to update them, which could be lengthy if done sequentially ... – Stefan Winkler Jul 02 '14 at 11:01
  • Ok, I see. But don't worry about "bypassing framework" - if direct SQL suits the situation better than manipulating entities, it's perfectly fine to do so. You only have to take care that if those manipulations should be visible in other parts where entities are used, that they are reloaded. We use this method when querying for report data in our application. – Alexander Rühl Jul 02 '14 at 12:49