0

I used Apache Bahir's AMQSource connector that listens to ActiveMQ, but when I run the Flink job to consume the data from ActiveMQ no output is generated.

For example, the connector is listening to ActiveMQ which contains 4 messages, but when I run the Flink job no data get consumed.

val brokerURL = "tcp://localhost:61616"
val destinationName = "TEST.FOO"
val filePath = "C:\\output" + System.currentTimeMillis + ".csv"

val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStateBackend(new MemoryStateBackend(1000, false))


val config = new AMQSourceConfig.AMQSourceConfigBuilder[String]()
  .setConnectionFactory(new ActiveMQConnectionFactory(brokerURL))
  .setDestinationName(destinationName)
  .setDeserializationSchema(new SimpleStringSchema)
  .setDestinationType(DestinationType.QUEUE)
  .setRunningChecker(new RunningChecker).build
val amqSource = new AMQSource[String](config)

val stream = env.addSource(amqSource)

stream.map(/*Some MapFunction*/)

stream.writeAsText(filePath)

stream.print

env.execute
Justin Bertram
  • 29,372
  • 4
  • 21
  • 43
Miki
  • 1
  • How are you confirming that there are 4 messages in the `TEST.FOO` queue on ActiveMQ? – Justin Bertram Feb 28 '19 at 15:57
  • I'm following it by the ActiveMQ console : http://localhost:8161/admin/queues.jsp, in the column "Number Of Pending Messages" I saw 4. In addition when I consume the queue not by flink source but other code (see below link) it does generate the mesagges. https://stackoverflow.com/questions/10795220/how-to-get-all-enqueued-messages-in-activemq. – Miki Mar 03 '19 at 09:16

1 Answers1

0

AMQSource expects message as bytes, see code from run method under AMQSource.class:

Message message = this.consumer.receive(1000L);
if (!(**message instanceof BytesMessage**)) {
LOG.warn("Active MQ source received non bytes message: {}", message);
return;
}

When produce data to ActiveMQ instead of text message: 

val message = session.createTextMessage(text)

Use bytes message:

val message = session.createBytesMessage()
message.writeBytes(text.getBytes)
Miki
  • 1