Stack Overflow
Stack Overflow

Questions tagged [aws-glue-connection]

40 questions
1
vote
0 answers

AWS Glue not able to access database in VPC

I have AWS Glue Job which is using Spark and Scala with jdbc connections specified in the script for custom ETL and data decryption. While running the job in an environment where databases are not publicly available the jobs are failing with…
Trojan Developer
  • 13
  • 5
0
votes
1 answer

How to make AWS Glue Crawler connection to mongodb

We are having trouble getting AWS Glue Crawler to make connection to our mongodb. The mongodb is publically accessible But we get the following error [cb423c7c-b0ef-4649-92ea-c784ddd4aa96] ERROR : Unable to resolve any valid connection (Service:…
Kay
  • 17,906
  • 63
  • 162
  • 270
0
votes
2 answers

Connect to on-premise MongoDB from AWS Glue

I am trying to connect to on-premise MongoDB from AWS Glue. I have created a Database connection with connection type as MongoDB, SSL enabled true, VPC, Subnet, Security group in AWS console. I added a ETL Python script to connect the MongoDB. while…
Raja
  • 11
0
votes
0 answers

Decorator sql error when using Spark Big Query Connector

I am migrating data from gcp to aws. For this, I'm following this aws glue solution [1] utilizing the Spark Big Query Connector [2]. This approach is working fine for full loads, however I want to modify the job to load data incrementally based on…
alex
  • 1
  • 1
0
votes
0 answers

Getting error in aws lake formation,after running bluprint -Job run error : Exception :could not find source JDBC table table_name

I am using aws lake formation blueprint for migrating data from rds to S3 .I have migrated few tables from rds to S3 but after adding few tables I am not able to add new table to S3 and getting following error .enter image description here why this…
Pranita Nagapurkar
  • 1
0
votes
0 answers

AWS Glue & AWS Glue Connector For Elasticsearch

I am currently trying to set up the AWS Glue Connector for Elasticsearch I am having difficulty understanding the proper format for the output schema. There is no good documentation on setting up the output schema that I can find. Can anyone help me…
Shane Henry
  • 1
0
votes
0 answers

AWS Glue connection fails over SSL to RDS PostgreSQL database but works without SSL

I have an AWS RDS PostgreSQL database setup that I would like to connect to using AWS glue using JDBC. My setup is simple, I have 1 VPC, 3 DB private subnets with a route for 0.0.0.0/0 that goes to NAT gateway, a security group that contains a…
Alex
  • 1,293
  • 1
  • 13
  • 26
0
votes
0 answers

How to query date-partitioned Google BigQuery table using AWS Glue BigQuery Connector?

I have linked Firebase events to BigQuery and my goal is to pull the events into S3 from BigQuery using AWS Glue. When you link Firebase to BigQuery, it creates a default dataset and a date-partitioned table something like…
xedus
  • 61
  • 1
  • 7
0
votes
0 answers

Error while connecting snowflake to glue using custom jdbc connector and connection?

I am trying to connect AWS Glue with Snowflake by using JDBC custom connector and connection. However after I have created the connection and run my job and call the toDF() method to convert dynamic frame to Pyspark Dataframe I get the following…
Nand042090
  • 1
0
votes
0 answers

"communication link failure" while running AWS Glue job

I am copying data from in house Mysql database table to RDS MYSQL database table. i executed my AWS glue job on smaller data size ( 1million records) and i am able to successfully copy data to RDS Mysql database. But when i try to run the same AWS…
user8545255
  • 761
  • 3
  • 9
  • 21
0
votes
1 answer

AWS Glue JOB to get secret value from secretmanager

I'm working on the AWS glue job, can someone please help me to give me a script in AWS Glue Job Spark that would retrieve my secrets from secret manager. Help is appreciated.
Johnny Bravo
  • 41
  • 5
0
votes
1 answer

Create dynamic frame from S3 bucket AWS Glue

Summary: I've got a S3 bucket which contains list of JSON files. Bucket contains child folders which are created by date. All the files contain similar file structure. Files get added on daily basis. JSON Schema schema = StructType([ …
AIViz
  • 82
  • 9
0
votes
0 answers

Custom JDBC Driver AWS Glue Connection

It seems that specifying a JDBC_DRIVER_JAR_URI connection property when defining an aws glue connection in terraform does nothing. When I test the glue connection, the cloudwatch logs show that glue is still using version 9.4 JDBC driver or…
Steven Kitscha
  • 1
  • 1
0
votes
1 answer

Access runtime parameters in AWS Glue Studio

I have created a Glue job using Glue studio Visual Editor. This job is getting data from a JDBC data source (Clickhouse Database) using a query. Doing some business transformations and then storing the data back to the database table using the same…
Prateek Rastogi
  • 3
  • 2
0
votes
1 answer

why is the session name appended to the role name when trying to connect to AWS Glue?

I am trying to create a job with Amazon Glue, using boto3. I'm using STS to connect to the AWS account, as per the following code: session_name = 'glue_job_creation' client =…
15b3m
  • 15
  • 2
1
2
3