1

Is it possible to install additional libs/dependencies after the cluster is already up and running?

Things I've done that are related to this:

  • I've already done the pre-creation bootstrapping process (this is a different solution altogether)
  • alternatively, SSH'd into each node and installed dependencies after the cluster comes up

I think the post-startup installation solution would involve being able to fire off a command to all the executors from the driver. Is there a way to accomplish this?

Kristian
  • 21,204
  • 19
  • 101
  • 176

1 Answers1

1

If you want additional dependencies to your steps, you can add them in the command of the step ( i.e. for Spark use the --jars option )

Harel Gliksman
  • 734
  • 1
  • 7
  • 19