How to use Apache Streaming with DynamoDB Stream

Question

We have a requirement wherein we log events in a DynamoDB table whenever an ad is served to the end user. There are more than 250 writes into this table per sec in the dynamoDB table.

We would want to aggregate and move this data to Redshift for analytics.

The DynamoDB stream will be called for every insert made in the table i suppose. How can I feed the DynamoDB stream into some kind of batches and then process those batches. Are there any best practices around such kind of use cases ?

I was reading about apache spark and seems like with Apache Spark we can do such kind of aggregation. But apache spark stream does not read the DynamoDB stream.

Any help or pointers is appreciated.

Thanks

Answer 1

DynamoDB streams have two interfaces: low-level API, and Kinesis Adapter. Apache Spark has a Kinesis integration, so you can use them together. In case if you are wondering what DynamoDB streams interface you should use, AWS suggests that Kinesis Adapter is a recommended way.

Here is how to use Kinesis adapter for DynamoDB.

Few more things to consider:

• Instead of using Apache Spark it is worth looking at Apache Flink. It is a stream-first solution (Spark implements streaming using micro-batching), has lower latencies, higher throughput, more powerful streaming operators, and has support for cycling processing. It also has a Kinesis adapter

• It can be the case that you don't need DynamoDB streams to export data to Redshift. You can export data using Redshift commands.

Answer 2

Amazon EMR provides an implementation of this connector as part of emr-hadoop-ddb.jar, which contains the DynamoDBItemWriteable class. Using this class, you can implement your own DynamoDBInputFormat as shown below.

 public class DynamoDbInputFormat implements InputFormat, Serializable {

    @Override
    public InputSplit[] getSplits(@NonNull final JobConf job, final int numSplits) throws IOException {
        final int splits = Integer.parseInt(requireNonNull(job.get(NUMBER_OF_SPLITS), NUMBER_OF_SPLITS
            + " must be non-null"));

        return IntStream.
            range(0, splits).
            mapToObj(segmentNumber -> new DynamoDbSplit(segmentNumber, splits)).
            toArray(InputSplit[]::new);
}