How to use Apache Apex Malhar RabbitMQ operator in DAG

Question

I have an Apache Apex application DAG which reads RabbitMQ message from a queue. Which Apache Apex Malhar operator should I use? There are several operators but it's not clear which one to use and how to use it.

the question is ? – Gabriele Santomaggio Feb 13 '17 at 15:21 — Gabriele Santomaggio, Feb 13 '17 at 15:21

score 4 · Answer 1 · answered Feb 13 '17 at 18:07

4

Have you looked at https://github.com/apache/apex-malhar/tree/master/contrib/src/main/java/com/datatorrent/contrib/rabbitmq ? There are also tests in https://github.com/apache/apex-malhar/tree/master/contrib/src/test/java/com/datatorrent/contrib/rabbitmq that show how to use the operator

answered Feb 13 '17 at 18:07

Sanjay

141
1

Please do explain your answer. Or you can put as comment. – αƞjiβ Feb 13 '17 at 18:21

score 2 · Answer 2 · answered Feb 13 '17 at 18:16

https://github.com/apache/apex-malhar/blob/master/contrib/src/main/java/com/datatorrent/contrib/rabbitmq/AbstractRabbitMQInputOperator.java

That is the main operator code where the tuple type is a generic parameter and emitTuple() is an abstract method that subclasses need to implement.

AbstractSinglePortRabbitMQInputOperator is a simple subclass that provides a single output port and implements emitTuple() using another abstract method getTuple() which needs an implementation in its subclasses.

The tests that Sanjay pointed to show how to use these classes.

score 0 · Answer 3 · answered Aug 08 '17 at 16:51

I also had problems finding out how to read messages from RabbitMQ to Apache Apex. With the help of the provided links of Sanjay's answer (https://stackoverflow.com/a/42210636/2350644) I finally managed to get it running. Here's how it works all together:

1. Setup a RabbitMQ Server

There are lot of ways installing RabbitMQ that are described here: https://www.rabbitmq.com/download.html
The simplest way for me was using docker (See: https://store.docker.com/images/rabbitmq)

docker pull rabbitmq

docker run -d --hostname my-rabbit --name some-rabbit -p 5672:5672 -p 15672:15672 rabbitmq:3-management

To check if RabbitMQ is working, open a browser and navigate to: http://localhost:15672/. You should see the Management page of RabbitMQ.

2. Write a Producer program

To send messages to the queue you can write a simple JAVA program like this:

import com.rabbitmq.client.BuiltinExchangeType;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;

import java.util.ArrayList;

public class Send {
    private final static String EXCHANGE = "myExchange";

    public static void main(String[] args) throws Exception {
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("localhost");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();
        channel.exchangeDeclare(EXCHANGE, BuiltinExchangeType.FANOUT);

        String queueName = channel.queueDeclare().getQueue();
        channel.queueBind(queueName, EXCHANGE, "");

        List<String> messages = Arrays.asList("Hello", "World", "!");
        for (String msg : messages) {
            channel.basicPublish(EXCHANGE, "", null, msg.getBytes("UTF-8"));
            System.out.println(" [x] Sent '" + msg + "'");
        }

        channel.close();
        connection.close();
    }
}

If you execute the JAVA program you should see some outputs in the Management UI of RabbitMQ.

3. Implement a sample Apex Application

3.1 Bootstrap a sample apex application

Follow the official apex documentation http://docs.datatorrent.com/beginner/

3.2 Add additional dependencies to pom.xml

To use the classes provided by malhar add the following dependencies:

<dependency>
  <groupId>org.apache.apex</groupId>
  <artifactId>malhar-contrib</artifactId>
  <version>3.7.0</version>
</dependency>
<dependency>
  <groupId>com.rabbitmq</groupId>
  <artifactId>amqp-client</artifactId>
  <version>4.2.0</version>
</dependency>

3.3 Create a Consumer

We first need to create an InputOperator that consumes messages from RabbitMQ using available code from apex-malhar.

import com.datatorrent.contrib.rabbitmq.AbstractSinglePortRabbitMQInputOperator;

public class MyRabbitMQInputOperator extends AbstractSinglePortRabbitMQInputOperator<String> {
    @Override
    public String getTuple(byte[] message) {
        return new String(message);
    }
}

You only have to override the getTuple() method. In this case we simply return the message that was received from RabbitMQ.

3.4 Setup an Apex DAG

To test the application we simply add an InputOperator (MyRabbitMQInputOperator that we implemented before) that consumes data from RabbitMQ and a ConsoleOutputOperator that prints the received messages.

import com.rabbitmq.client.BuiltinExchangeType;
import org.apache.hadoop.conf.Configuration;

import com.datatorrent.api.annotation.ApplicationAnnotation;
import com.datatorrent.api.StreamingApplication;
import com.datatorrent.api.DAG;
import com.datatorrent.api.DAG.Locality;
import com.datatorrent.lib.io.ConsoleOutputOperator;

@ApplicationAnnotation(name="MyFirstApplication")
public class Application implements StreamingApplication
{

  private final static String EXCHANGE = "myExchange";


  @Override
  public void populateDAG(DAG dag, Configuration conf)
  {

    MyRabbitMQInputOperator consumer = dag.addOperator("Consumer", new MyRabbitMQInputOperator());
    consumer.setHost("localhost");
    consumer.setExchange(EXCHANGE);
    consumer.setExchangeType(BuiltinExchangeType.FANOUT.getType());

    ConsoleOutputOperator cons = dag.addOperator("console", new ConsoleOutputOperator());

    dag.addStream("myStream", consumer.outputPort, cons.input).setLocality(Locality.CONTAINER_LOCAL);
  }
}

3.5 Test the Application

To simply test the created application we can write a UnitTest, so there is no need to setup a Hadoop/YARN cluster. In the bootstrap application there is already a UnitTest namely ApplicationTest.java that we can use:

import java.io.IOException;
import javax.validation.ConstraintViolationException;
import org.junit.Assert;
import org.apache.hadoop.conf.Configuration;
import org.junit.Test;
import com.datatorrent.api.LocalMode;

/**
 * Test the DAG declaration in local mode.
 */
public class ApplicationTest {

  @Test
  public void testApplication() throws IOException, Exception {
    try {
      LocalMode lma = LocalMode.newInstance();
      Configuration conf = new Configuration(true);
      //conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml"));
      lma.prepareDAG(new Application(), conf);
      LocalMode.Controller lc = lma.getController();
      lc.run(10000); // runs for 10 seconds and quits
    } catch (ConstraintViolationException e) {
      Assert.fail("constraint violations: " + e.getConstraintViolations());
    }
  }

}

Since we don't need any properties for this application the only thing changed in this file is uncommenting the line:

conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml"));

If you execute the ApplicationTest.java and send messages to RabbitMQ using the Producer program as described in 2., the Test should output all the messages. You might need to increase the time of the test to see all messages (It is set to 10sec currently).