Questions tagged [aws-glue-spark]
244 questions
0
votes
0 answers
How to Select values from column which has array data
+--------------------+--------------------+
| _VALUE| paraarr|
+--------------------+--------------------+
|Archer, Edward Pa...|[, [[, Arbitrator...|
|Archer, Edward Pa...|[, [[, Member:],,...|
|Archer, Edward…
user190549
- 397
- 1
- 7
- 15
0
votes
1 answer
Is it possible writing down to RDS raw sql (PostgreSQL) using AWS/Glue/Spark shell?
I have a Glue/Connection for an RDS/PostgreSQL DB pre-built via CloudFormation, which works fine in a Glue/Scala/Sparkshell via getJDBCSink API to write down a DataFrame to that DB.
But also I need to write down to the same db, plain sql like create…
Randomize
- 8,651
- 18
- 78
- 133
-1
votes
1 answer
pyspark code failing with error An error occurred while calling z:com.amazonaws.services.glue.DynamicFrame.apply. list#5451 []
I am writing aws glue job (pyspark code) using SQL Transformation. I am getting error with scala.MatchError: list#5252 [] (of class org.apache.spark.sql.catalyst.expressions.ListQuery. There is one table like tbl1 when i exclude this table from…
Kaustubh K
- 11
- 5
-1
votes
1 answer
Spark AWS Glue UI - Error: Could not find or load main class $SPARK_HISTORY_OPTS - Windows Docker
Ref -
https://github.com/aws-samples/aws-glue-samples/tree/master/utilities/Spark_UI/
https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-history.html
I am trying to access glue spark ui locally on my windows 10 - (docker desktop) using…
Sarde
- 658
- 1
- 8
- 19