21

When EMR machine is trying to run a step that includes boto3 initialisation it sometimes get the following error: ValueError: Invalid endpoint: https://s3..amazonaws.com When I'm trying to set up a new machine it can suddenly work. Attached the full error:

self.client = boto3.client("s3")
  File "/usr/local/lib/python3.6/site-packages/boto3/__init__.py", line 83, in client
    return _get_default_session().client(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 861, in create_client
    client_config=config, api_version=api_version)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 76, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 285, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.6/site-packages/botocore/args.py", line 79, in get_client_args
    timeout=(new_config.connect_timeout, new_config.read_timeout))
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 297, in create_endpoint
    raise ValueError("Invalid endpoint: %s" % endpoint_url)
ValueError: Invalid endpoint: https://s3..amazonaws.com

Any idea why it happens?

(Versions: boto3==1.7.29, botocore==1.10.29)

Aviv Oron
  • 505
  • 1
  • 6
  • 10
  • @DeepSpace I can see it, but as far as i can see - its nothing in my code, it looks like it is something in boto3 – Aviv Oron Sep 15 '19 at 10:07
  • 2
    I also don't get the downvotes when it's clear that the library raised the error, not the OP's code – roganjosh Sep 15 '19 at 10:09
  • 1
    @roganjosh a downvote means lack of research. If it's clear the library raised the error then OP can/should have opened an issue with the library's repo. It's not something that can be fixed/answered here – DeepSpace Sep 15 '19 at 10:12
  • 1
    @DeepSpace we'll agree in part, but I think it also should be raised here, even if the issue is taken up on github. In my experience of such instances, the answer "Vx.y has an issue with AWS" is likely to bubble up on SO more often in Google searches than a long, protracted debate of the issue on github itself – roganjosh Sep 15 '19 at 10:14
  • Somehow region is not set I think. – Lamanus Sep 15 '19 at 10:33

3 Answers3

26

It looks like you have an invalid region. Check your ~/.aws/config

ChickenSoups
  • 937
  • 8
  • 18
13

Set the region in your ~/.aws/credentials or ~/.aws/config files. You can set the region as an environment variable as well e.g.

In bash

export AWS_REGION="eu-west-2"

or in Powershell

$Env:AWS_REGION="eu-west-2"
ubi
  • 4,041
  • 3
  • 33
  • 50
1

In my case, even though ~/.aws/config had the region set,

$ cat ~/.aws/config 
[default]
region = us-east-1

the env var AWS_REGION was set to an empty string

$ env | grep -i aws
AWS_REGION=

unset this env var and all was good again

$ unset AWS_REGION
$ aws sts get-caller-identity --output text --query Account
777***234534

(apologies for posting on a really old question, it did pop up in a Google search)

Gavin
  • 349
  • 4
  • 10
  • 4
    For me the offending env var was `AWS_DEFAULT_REGION= `. `unset AWS_DEFAULT_REGION` fixed the issue. – colllin Feb 08 '23 at 23:14