Consumer/Producer AWS SQS akka scala with synchrone consumer

Question

My application have a producer and a consumer. My producer produces messages irregularly. Sometime my queue will be empty, sometime I will have a few messages. I would like to have my consumer listen to the queue and when a message is in it, take it and process this message. The process could take hours and I don't want my consumer to take another message of the queue if he doesn't have finished to process the current message.

I think AKKA and AWS SQS can meet my needs. By reading documentations and examples, it seems that akka-camel can simplified my work.

I found this example on github.

I am more interested in the configuration of the consumer (the Thread.sleep is just to simulate my processing) :

class MySqsConsumer extends Consumer {

  //The SQS URI is an in-only message exchange (autoAck=true)
  override def endpointUri = "aws-sqs://sqs-akka-camel?  amazonSQSClient=#client"

  override def receive = {
    case msg: CamelMessage => {
      println("Start received %s" format msg.bodyAs[String])
      Thread.sleep(4000)
      println("Stop received %s" format msg.bodyAs[String])

    }

  }
 }

StackTrace :

...
14:38:53.335 [Camel (sqs-akka-camel) thread #0 - aws-sqs://sqs-akka-camel] DEBUG o.a.camel.processor.SendProcessor - >>>> Endpoint[akka://sqs-akka-camel/user/consumer?autoAck=true&replyTimeout=60000+milliseconds] Exchange[Message: Hello World1!]
14:38:54.051 [Camel (sqs-akka-camel) thread #0 - aws-sqs://sqs-akka-camel] DEBUG o.a.camel.processor.SendProcessor - >>>> Endpoint[akka://sqs-akka-camel/user/consumer?autoAck=true&replyTimeout=60000+milliseconds] Exchange[Message: Hello World2!]
14:38:54.753 [Camel (sqs-akka-camel) thread #0 - aws-sqs://sqs-akka-camel] DEBUG o.a.camel.processor.SendProcessor - >>>> Endpoint[akka://sqs-akka-camel/user/consumer?autoAck=true&replyTimeout=60000+milliseconds] Exchange[Message: Hello World3!]
Start received Hello World1!
Stop received Hello World1!
Start received Hello World2!
Stop received Hello World2!
Start received Hello World3!
...

My problem here is that akka-camel reads the messages from the queue before the receive method is done.

How can I make my consumer wait the end of the process, before taking a new message? If I am using the wrong tools/libraries, could you direct me towards new tools?

score 1 · Answer 1 · answered Jun 17 '16 at 20:39

Not sure if you are using akka streams, but if you are, send your source to a Sink.queue

val graph=yourSource.to(Sink.queue)

When materialized (graph.run) it will return a SinkQueue, you can pull manually items from it. It will use backpressure mechanism, so you don't have to worry about what will happen if you don't pull items.

Note: I ignored generics on purpose! but don't forget them.

score 0 · Answer 2 · answered Nov 04 '16 at 23:03

As far as I know, you can't do it. Camel consumes message from the queue all the time.

You can set override def autoAck = false - it will not send another message to receive(), until you acknowledged the current one. But the "hidden" camel actor will still be consuming.

Maybe you can provide a customized amazonSQSClient and somehow control it.

Consumer/Producer AWS SQS akka scala with synchrone consumer

2 Answers2