6

Sometimes when I want to run an ETL Job in AWS Glue, it is triggered immediately. But quite often it happens to me that it takes few minutes before ETL Job is doing anything - I see nothing in logs, only "pending execution". Is there any way I can influence it in configuration? Or is it totally up to AWS when it will start the job?

tommy vercetti
  • 307
  • 5
  • 16
  • 1
    Possible duplicate of [AWS Glue takes a long time to finish](https://stackoverflow.com/questions/45947087/aws-glue-takes-a-long-time-to-finish) – jbgorski Oct 29 '18 at 20:10
  • 1
    In addition to @j.b.gorski's comment, logs may not always be flushed to CloudWatch completely until some time after the ETL job has finished. – Kyle Oct 31 '18 at 06:39
  • @Kyle, I doubt that, in my case I'm constantly refreshing s3 bucket, in which I write data using Glue job and I see nothing for few minutes when it says "pending execution" – Ash Jan 17 '19 at 12:39

1 Answers1

8

I know this is an old thread, but I recently experienced the same symptoms. You'll have to confirm if the cause is/was the same. The job I originally created was to move data from my data lake in to a redshift db. When the job was created it had the redshift connection as a required resource, but it seems that glue was not able to acquire that resource prior to starting the job so the job would fail before it started every time... no logs.

I removed the connection as a requirement for the job by editing the job from the console. (Action->Edit job-> Required Connections->"X")

After I removed the requirement, the job ran just fine. It took me a long time to figure that one out. Hopefully it helps anyone else to runs into the same scenario.

user11985428
  • 81
  • 1
  • 2