Stack Overflow
Stack Overflow

Questions tagged [spark-ec2]

spark-ec2 is designed to manage multiple named clusters.

You can launch a new cluster (telling the script its size and giving it a name), shutdown an existing cluster, or log into a cluster. Each cluster is identified by placing its machines into EC2 security groups whose names are derived from the name of the cluster.

22 questions
0
votes
1 answer

Spark on EC2 Dedicated Tenancy

I have successfully managed to install a Spark Cluster in EC2 using the distribution from: https://github.com/amplab/spark-ec2 However, I can see there is no option for having EC2 dedicated hardware tenancy for the Spark master and cluster nodes,…
and_apo
  • 1,217
  • 3
  • 17
  • 41
0
votes
1 answer

Spark on EC2 - S3 endpoint for Scala not resolving

Hi I have been able to setup successfully a Spark Cluster on AWS EC2 for 2 ongoing months but recently I started getting the following error in the creation script. It's basically failing in setting up the Scala packages and not resolving the source…
and_apo
  • 1,217
  • 3
  • 17
  • 41
0
votes
0 answers

Amazon AWS cannot access my s3n url

I am trying to do spark-submit with my python file and in this python file, I would like to access my data.txt that is stored in my bucket. I am using the following command to access my file but I keep getting an error. dataFile = ("s3n://
Ray.R.Chua
  • 777
  • 3
  • 8
  • 27
0
votes
2 answers

Can a PySpark Kernel(JupyterHub) run in yarn-client mode?

My Current Setup: Spark EC2 Cluster with HDFS and YARN JuputerHub(0.7.0) PySpark Kernel with python27 The very simple code that I am using for this question: rdd = sc.parallelize([1, 2]) rdd.collect() The PySpark kernel that works as expected in…
and_apo
  • 1,217
  • 3
  • 17
  • 41
0
votes
1 answer

Apache Spark-ec2 script: "ERROR Unknown Spark version". Broken init.sh?

I want to launch an AWS EC2 instance with the spark-ec2 script. I get this error: Initializing spark --2016-11-18 22:33:06-- http://s3.amazonaws.com/spark-related-packages/spark-1.6.3-bin-hadoop1.tgz Resolving s3.amazonaws.com (s3.amazonaws.com)...…
Peter Pearman
  • 129
  • 1
  • 10
0
votes
2 answers

spark spark-ec2 credentials using aws_security_token

I would like to ask if it is currently possible to use spark-ec2 script https://spark.apache.org/docs/latest/ec2-scripts.html together with credentials that are consisting not only from: aws_access_key_id and aws_secret_access_key, but it also…
ziky90
  • 2,627
  • 4
  • 33
  • 47
0
votes
1 answer

Apache Spark EC2 c4.2xlarge not launching

When attempting to launch an EC2 cluster (c4.2xlarge) using the Apache Spark EC2 script I get the following error: Don't recognize c4.2xlarge, assuming type is pvm Spark AMI: ami-9a6e0daa Launching instances... ERROR:boto:400 Bad…
monster
  • 1,762
  • 3
  • 20
  • 38
1
2