Questions tagged [aws-glue3.0]
22 questions
0
votes
1 answer
Access runtime parameters in AWS Glue Studio
I have created a Glue job using Glue studio Visual Editor. This job is getting data from a JDBC data source (Clickhouse Database) using a query. Doing some business transformations and then storing the data back to the database table using the same…
Prateek Rastogi
- 3
- 2
0
votes
1 answer
AWS Glue - multiple RDS tables in one job
I am trying to use AWS Glue.
My data source is in RDS(AWS Aurora) and the destination is s3.
My RDS database has many tables and I would like to sync all of them toward s3.
In the data source settings, there is an input for table name but I can…
tesnirs
- 57
- 6
0
votes
2 answers
It is possible use Spark 3.3.0 in AWS Glue 3.0
I would like to use Spark 3.3.0 version features like Trigger.availableNow in AWS Glue 3.0 with scala, but the AWS Glue 3.0 usage Apache spark version 3.1.1, Is there any way to use apache spark 3.3.0 in AWS Glue 3.0 with scala.
krishna Prasad
- 3,541
- 1
- 34
- 44
0
votes
0 answers
How to create dynamic dataframe from AWS Glue 3 catalog in local environment?
I I have performed some AWS Glue version 3.0 jobs testing using Docker containers as detailed here.
The following code outputs two lists, one per connection, with the names of the tables in a database:
import boto3
db_name_s3 =…
GDC
- 17
- 6
0
votes
1 answer
How can I optimize the read from S3?
dyf_pagewise_word_count = glueContext.create_dynamic_frame.from_options(
connection_type="s3",
format="csv",
connection_options={
"paths": ["s3://somefile.csv/"],
'recurse':True,
'groupFiles': 'inPartition',
'groupSize':…
sheetal_158
- 7,391
- 6
- 27
- 44
0
votes
1 answer
AWS Glue Data Target Upsert generates an Exception
I'm using AWS Glue to load data into a Redshift database using Glue Studio.
If the Data Target is Insert Only the data gets inserted without any problem, this is the code generated:
# Script generated for node Amazon Redshift
AmazonRedshift_node =…
saavedrah
- 183
- 14
-1
votes
1 answer
How to create script for AWS Glue Job using CloudFormation Template
I am trying to write a CloudFormation template for glue job. I see that there is no option available in official documentation to use apply mapping feature just like we see in the console. How can we use the same feature in our template?
I know that…
aws-beginner
- 1
- 2