Issue running aws glue job locally

Question

I'm trying to run a glue job locally but I'm facing a problem, when I run my script a exception is raised:

py4j.protocol.Py4JJavaError: An error occurred while calling o47.getDynamicFrame.
: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.metrics2.lib.MutableCounterLong.<init>(Lorg/apache/hadoop/metrics2/MetricsInfo;J)V from class org.apache.hadoop.fs.s3a.S3AInstrumentation

I downloaded aws-glue-libs from here: https://github.com/awslabs/aws-glue-libs/tree/glue-1.0/awsglue.

My code:

from pyspark.sql import SparkSession

from awsglue.context import GlueContext

spark = SparkSession \
    .builder \
    .appName("GlueSparkJobExample") \
    .config("spark.jars", "AWSGlueETLPython-1.0.0-jar-with-dependencies.jar") \
    .config("spark.local.dir", "/tmp") \
    .getOrCreate()

sc = spark.sparkContext
glueContext = GlueContext(sc)

db = "database"
table = "table"

my_df = glueContext.create_dynamic_frame.from_catalog(
database=db, table_name=table)

If someone can help would be great.

I have worked on glue locally using docker if interested follow https://towardsdatascience.com/develop-glue-jobs-locally-using-docker-containers-bffc9d95bd1 — Shubham Jain, Aug 11 '20 at 05:06

Issue running aws glue job locally

0 Answers0