8

We are using AWS Glue service to convert json files to parquet. Intermittently Glue job is failing with Resource unavailable Error. Can anyone help us how to resolve the same?

Senthil Kumar V
  • 183
  • 4
  • 9

3 Answers3

4

I've also experienced the "Resource unavailable" exception a few times but it's on their side (service) and they are actively working to improve it.

I've opened a support ticket regarding this exception and they said:

Glue creates a new spark environment at the backend in order to run the jobs submitted depending on account ID, IAM role, subnetID and security groups. Running the Glue jobs under different IAM user may help to mitigate this issue. By doing this, Glue creates a new spark environment at the backend in order to run the jobs submitted by different IAM role or different accountID.

So if your account has several jobs running concurrently and the "Resource unavailable" error keeps appearing I would recommend you to create different IAM roles and splitting the jobs between them in order to mitigate this particular issue.

leovrf
  • 605
  • 1
  • 6
  • 17
1

I have recently struggled with Resource Unavailable thrown by Glue Job

Also i was not able to make a direct connection in Glue using RDS -it said "no suitable security group found"

I faced this issue while trying to connect with AWS RDS using Glue Job.

The problem was with the Security Group that the RDS was using. There is a need to place a self referencing inbound rule in the Security Group.

For those who dont know what is self referencing inbound rule, follow the steps

1) Go to the Security Group you are using (VPC -> Security Group)

2) In the Inbound Rules select Edit Inbound Rules

3) Add a Rule

a) Type - All Traffic b) Protocol - All c) Port Range - ALL d) Source - custom and in space available write the initial of your security group and select it. e) Save it.

Its done !

if you were missing this condition in your Security Group Inbound Rules

Try creating the connection you will be able to create the connection.

Also job should work this time.

kartik
  • 79
  • 3
0

This is still an ongoing issue. For my setup was the availability zone the RDS connection's subnet was in. The "fix" was to:

  1. AWS Console > Glue > Connections > Edit Connection > See which subnet the connection is using.
  2. AWS Console > VPC > Subnets > Identify (or create) a subnet in a different zone.
  3. AWS Console > Glue > Connections > Edit Connection > Switch to use the subnet from step 2.
  4. Run job.

If the job still fails with Resource Unavailable, repeat until it works.

weffey
  • 107
  • 1
  • 7