So I have deployed a standalone cluster to Amazon EC2 using Terraform. It is using passwordless ssh to communicate with workers.
I start the master with the start master script, giving the public ip of the cluster to be the public dns of the ec2 instance.
I then start the slaves using the start-slaves script, having copied over a config/slaves file with the public ip addresses of the 2 EC2 instances that are available to run the workers. (they each have the spark deployment on them in the same location as the master)
In the UI the workers are registered and running:
However, when i submit any job to the cluster, it is never able to allocate resources showing the message:
Does anyone know how to solve this?
The logs show the workers starting and registering correctly, and the task i'm submitting is within the available resources. (have tried as little as 1 cpu core and 500mb).
Does anyone know why the task may not be being accepted?
Cheers
