0

I would like to ask if it is currently possible to use spark-ec2 script https://spark.apache.org/docs/latest/ec2-scripts.html together with credentials that are consisting not only from: aws_access_key_id and aws_secret_access_key, but it also contains aws_security_token.

When I try to run the script I am getting following error message:

ERROR:boto:Caught exception reading instance data
Traceback (most recent call last):
File "/Users/zikes/opensource/spark/ec2/lib/boto-2.34.0/boto/utils.py", line 210, in retry_url
r = opener.open(req, timeout=timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1184, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 64] Host is down>
ERROR:boto:Unable to read instance data, giving up
No handler was ready to authenticate. 1 handlers were checked. ['QuerySignatureV2AuthHandler'] Check your credentials

Does anyone has some idea what can be possibly wrong? Is aws_security_token the problem? It maybe seems to me more as boto than Spark problem.

I have tried both:

1) setting credentials in ~/.aws/credentials and ~/.aws/config

2) setting credential by commands: 

export aws_access_key_id=<my_aws_access_key>
export aws_secret_access_key=<my_aws_seecret_key>
export aws_security_token=<my_aws_security_token>

My launch command is:

./spark-ec2 -k my_key -i my_key.pem --additional-tags "mytag:tag1,mytag2:tag2" --instance-profile-name "profile1" -s 1 launch test
zero323
  • 322,348
  • 103
  • 959
  • 935
ziky90
  • 2,627
  • 4
  • 33
  • 47
  • Check the error message. It says: URLError: Open spark-ec2 script and check the URL is valid and the host is up and running. – helloV Jul 24 '15 at 00:14
  • Actually there is no url in spark-ec2 script. And I don't even know from where is this called. There was also no machine created at all at my AWS. – ziky90 Jul 24 '15 at 10:09

2 Answers2

0

you can setup your credentials & config using the command aws configure. I had the same issue but in my case my AWS_SECRET_ACCESS_KEY had a slash, I regenerated the key until there was no slash and it worked

Chenna V
  • 10,185
  • 11
  • 77
  • 104
0

The problem was that I did not use profile called default after renaming everything worked well.

ziky90
  • 2,627
  • 4
  • 33
  • 47