Unable to get AWS creds for local dynamodb

Question

I am trying to connect to local dynamodb on my system with my spark program. I am following https://aws.amazon.com/blogs/big-data/analyze-your-data-on-amazon-dynamodb-with-apache-spark/ However I am getting below error.

23/04/05 12:02:46 WARN DynamoDBFibonacciRetryer: Retry: 1 Exception: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [com.amazonaws.auth.InstanceProfileCredentialsProvider@6059fd6f: Unable to load credentials from service endpoint]

I have followed below page as well https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html but no use.

PFB the code snippet package test

import org.apache.hadoop.io.Text
import org.apache.hadoop.dynamodb.DynamoDBItemWritable
import org.apache.spark.sql.SparkSession
/* Importing DynamoDBInputFormat and DynamoDBOutputFormat */
import org.apache.hadoop.dynamodb.read.DynamoDBInputFormat
import org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat
import org.apache.hadoop.mapred.JobConf
import org.apache.hadoop.io.LongWritable

object dynamoRead {

  def main(args: Array[String]): Unit = {

    val spark  = SparkSession.builder().master("local[1]").appName("dynamo").getOrCreate()

    val sc = spark.sparkContext

    var jobConf = new JobConf(sc.hadoopConfiguration)
    jobConf.set("dynamodb.servicename", "dynamodb")
    jobConf.set("dynamodb.input.tableName", "Music")
    jobConf.set("dynamodb.endpoint", "http://localhost:8000")
    jobConf.set("dynamodb.awsAccessKeyID","")
    jobConf.set("dynamodb.awsSecretAccessKey","")

    jobConf.set("mapred.output.format.class","org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat")
    jobConf.set("mapred.input.format.class","org.apache.hadoop.dynamodb.read.DynamoDBInputFormat")

    var num_count = sc.hadoopRDD(jobConf, classOf[DynamoDBInputFormat], classOf[Text], classOf[DynamoDBItemWritable])

    num_count.count()

  }

}

Can somebody please point out what I might be missing or how to debug this?

packages used:

spark : 3.0.0 emr-dynamodb-hadoop : 4.16.0

Unable to get AWS creds for local dynamodb

0 Answers0