Spring Boot and long running tasks

Question

In my Spring Boot application I have to implement an import service. Users can submit a bunch of JSON files and application will try to import the data from these files. Depending on the data amount at JSON files the single import process can take a 1 or 2 hours.

I do not want to block the users during the import process so I plan to accept the task for importing and notify user that this data is scheduled for processing. I'll put the data into the queue and a free queue-consumer on the other end will start the import process. Also, I need to have a possibility to monitor a jobs in the queue, terminate them if needed.

Right now I'm thinking to use Embedded Apache ActiveMQ in order to introduce message producer and consumer logic but before this I'd like to ask - from the architecture point of view - is it a good choice for the described task or it can be implemented with a more appropriate tools.. like for example plain Spring @Async and so on ?

score 1 · Answer 1 · edited May 23 '17 at 12:10

It is possible to treat files concurrently with Camel like this

from("file://incoming?maxMessagesPerPoll=1&idempotent=true&moveFailed=failed&move=processed&readLock=none").threads(5).process()

Take a look at http://camel.apache.org/file2.html

But i think that it is better for your requirements to use a standalone ActiveMQ, a standalone service to move files to ActiveMQ and standalone consumer to be capable to kill or restart each one independently.

It is better to use ActiveMQ as you said and you can easily create a service to move messages to a queue with Camel like this :

        CamelContext context = new DefaultCamelContext();
        ConnectionFactory connectionFactory = new ActiveMQConnectionFactory("vm://localhost?broker.persistent=true");
        context.addComponent("test-jms", JmsComponent.jmsComponentAutoAcknowledge(connectionFactory));
        context.addRoutes(new RouteBuilder() {
            public void configure() {
            // convertBodyTo to use TextMessage or maybe send them as file to the Queue            from("file://testFolderPath").convertBodyTo(String.class).to("test-jms:queue:test.queue");
            }
        });
        context.start();

Here some examples

http://www.programcreek.com/java-api-examples/index.php?api=org.apache.camel.component.jms.JmsComponent

https://skills421.wordpress.com/2014/02/08/sending-local-files-to-a-jms-queue/

https://github.com/apache/camel/blob/master/examples/camel-example-jms-file/src/main/java/org/apache/camel/example/jmstofile/CamelJmsToFileExample.java

https://github.com/apache/camel/tree/master/examples

To monitor and manage you can use jmx with VisualVM or Hawtio http://hawt.io/getstarted/index.html

http://camel.apache.org/camel-jmx.html

To consume you can use DefaultMessageListenerContainer with concurrent consumers on the queue and for this you need to change the prefetchPolicy on the ConnectionFactory of the DefaultMessageListenerContainer , Multithreaded JMS client ActiveMQ

Spring Boot and long running tasks

1 Answers1