Stack Overflow
Stack Overflow

Questions tagged [aws-step-config]

`StepConfig` : Specification of a cluster (job flow) step of aws used for aws step functions to execute hive or spark job after defining a step using this class.

This is basically used for aws step functions. to define aws job flow steps for example : executing a spark job or hive job

4 questions
9
votes
4 answers

AWS Step Functions is not catching States.Runtime error

The below step function is executed in aws and when there is a missing of a required parameter it cancel the flow and throws States.Runtime Error. This is in catch phase of the step function but it is not catching the error as stated. Defined Step…
shamila
  • 1,280
  • 6
  • 20
  • 45
2
votes
0 answers

Return parameters from Aws glue to step function

I have a scenario of executing glue jobs from the step function. If i need to pass the parameters from first glue job to second glue job, How can i achieve that? I researched on it and didn't found any documentation on it.
Sujitha Chinnu
  • 123
  • 9
2
votes
1 answer

Is it possible to wait until an EMR cluster is terminated?

I'm trying to write a component that will start up an EMR cluster, run a Spark pipeline on that cluster, and then shut that cluster down once the pipeline completes. I've gotten as far as creating the cluster and setting permissions to allow my…
alexgolec
  • 26,898
  • 33
  • 107
  • 159
1
vote
2 answers

How do I specify the Spark configuration when running on EMR?

So I'm trying to run a Spark pipeline on EMR, and I'm creating a step like so: // Build the Spark job submission request val runSparkJob = new StepConfig() .withName("Run Pipeline") .withActionOnFailure(ActionOnFailure.TERMINATE_CLUSTER) …
alexgolec
  • 26,898
  • 33
  • 107
  • 159