1

I was trying to programmatically Load a dynamodb table into HDFS (via java, and not hive), I couldnt find examples online on how to do it, so thought I'd download the jar containing org.apache.hadoop.hive.dynamodb and reverse engineer the process.

Unfortunately, I couldn't find the file as well :(.

Could someone answer the following questions for me (listed in order of priority).

  1. Java example that loads a dynamodb table into HDFS (that can be passed to a mapper as a table input format).
  2. the jar containing org.apache.hadoop.hive.dynamodb.

Thanks!

n915
  • 81
  • 1
  • 1
  • 5

2 Answers2

1

It's in hive-bigbird-handler.jar. Unfortunately AWS doesn't provide any source or at least Java Doc about it. But you can find the jar on any node of an EMR Cluster:

/home/hadoop/.versions/hive-0.8.1/auxlib/hive-bigbird-handler-0.8.1.jar

You might want to checkout this Article:

Unfortunately, Amazon haven’t released the sources for hive-bigbird-handler.jar, which is a shame considering its usefulness. Of particular note, it seems it also includes built-in support for Hadoop’s Input and Output formats, so one can write straight on MapReduce Jobs, writing directly into DynamoDB.

Tip: search for hive-bigbird-handler.jar to get to the interesting parts... ;-)

muhqu
  • 12,329
  • 6
  • 28
  • 30
0

1- I am not aware of any such example, but you might find this library useful. It provides InputFormats, OutputFormats, and Writable classes for reading and writing data to Amazon DynamoDB tables.

2- I don't think they have made it available publically.

Community
  • 1
  • 1
Tariq
  • 34,076
  • 8
  • 57
  • 79
  • Thanks for the information Tariq! (especially 2). Helped save a few hours of my life :) – n915 Jun 13 '13 at 15:39