0

My spark job(we are processing 100 GB of data) taking one hour to complete.in step functions i am using lambda to submit our job through Livy!! and created one more lambda function to get the job status. issue here is , after 15 mins the step function is failed!! any better solution to resolve this.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Siri Vali
  • 1
  • 1
  • What do you mean by "is failed"? Does it give any error message? – John Rotenstein Mar 15 '22 at 12:43
  • "ExecutionFailed", "details": { "cause": "Unable to apply step \"jobId\" to input [{\"MasterPublicDnsName\":\"ec2-$$$$$$$$.compute-1.amazonaws.com\",\"jobId\":3,\"jobStatus\":\"success\"},{\"MasterPublicDnsName\":\"ec2-$$$$$$$$.compute-1.amazonaws.com\",\"jobId\":4,\"jobStatus\":\"success\"}]", "error": "States.ReferencePathConflict" }, "previous_event_id": "0", "event_timestamp": "1647331002032", – Siri Vali Mar 15 '22 at 16:46
  • The Step Functions error is `States.ReferencePathConflict`. Sounds similar to: [amazon web services - How to insert the input from a step to a json struct and pass to next step in step functions? - Stack Overflow](https://stackoverflow.com/questions/50635671/how-to-insert-the-input-from-a-step-to-a-json-struct-and-pass-to-next-step-in-st) – John Rotenstein Mar 15 '22 at 21:01

0 Answers0