12

I am trying to push some files up to s3 with the AWS CLI and I am running into an error:

upload failed: ... An HTTP Client raised and unhandled exception: unknown encoding: idna

I believe this is a Python specific problem but I am not sure how to enable this type of encoding for my python interpreter. I just freshly installed Python 3.6 and have verified that it being used by powershell and cmd.

$> python --version Python 3.6.7

If this isn't a Python specific problem, it might be helpful to know that I also just freshly installed the AWS CLI and have it properly configured. Let me know if there is anything else I am missing to help solve this problem. Thanks.

Justin
  • 683
  • 8
  • 24
  • We need to see the source code. The aws cli is a standalone application. Are you using the python SDK or the command line CLI? If the latter, I'd recommend you take a look at the sdk as its much more convenient to use in python then "shelling out" to the cli. – erik258 Nov 04 '18 at 18:56
  • @DanFarrell well the source code that it is failing on is webpack compiled javascript that is hundreds of lines long. I am currently using the Python command line because that is what the AWS CLI depends on. I will checkout the SDK to see if I can fix this problem. – Justin Nov 04 '18 at 18:59
  • I'm confused by your wording there, and I wonder if perhaps I'm picking up on your own confusion. For example, you say "I'm using the python command line ... that is what the cli depends on " I'm not sure what you mean. The aws cli, implemented in pythion , is the _c_ommand _l_ine _i_nterface built by/for aws. it in turn built atop the `boto` libraries, which themselves _are_ the python sdk. – erik258 Nov 04 '18 at 19:01
  • @DanFarrell Well, I installed the AWS CLI and everything was working fine. I configured it to work with my accounts and all that no problem. When I went to start pushing things to S3 I get this error. After reading some about this error on other SO posts it seemed as though my Python version needed to be updated to 3.6 so that this encoding error would be fixed but even after ensuring the Python version was correct I am still seeing the error. I probably am confused by what is happening to be honest, this is the first time I've used the AWS CLI. – Justin Nov 04 '18 at 19:05
  • Ive used the aws cli and the python sdk quite a bit, from python 2 and 3, and I'd love to help you. But without _any_ example of what you're trying to do, how can I? – erik258 Nov 04 '18 at 19:07
  • @DanFarrell sorry I am trying to articulate my problem in a way that is helpful. I am trying to push a compiled angular application to s3 for a single page hosting on AWS. Webpack had compiled my Angular application and put the built files in a dist folder. I am trying to push that dist folder to s3 with the AWS CLI by using the command `aws s3 cp dist/ s3://mybucket.com --recursive` and this is the error that I am getting. – Justin Nov 04 '18 at 19:09
  • Do other commands work for you, such as `aws s3 ls`? – John Rotenstein Nov 04 '18 at 23:07
  • @JohnRotenstein yes other commands work, that one I tried specifically. Another weird thing that I've noticed when trying to figure this out is that different files in my webpack bundle fails on every push attempt. So its not consistently the same files that are causing the problem. – Justin Nov 05 '18 at 01:43
  • Same issue for me.... @Justin you ever figure this out? – HokieMike Dec 19 '18 at 20:55
  • @HokieMike No, unfortunately not. – Justin Dec 19 '18 at 22:14
  • @Justin. I think it may have something to do with either Python 3 or the Windows MSI bundle (which installs a local python 3 I think). I tried the MSI install on a second machine and was getting the same errors... so I tried similar steps to the answer below. I installed Python 2.7.15 and did a pip install of the AWS CLI and my problems went away. – HokieMike Dec 20 '18 at 00:59

5 Answers5

13

As a workaround, put the following import statement into the aws-script.py file.

import encodings.idna

Full path on 32-bit version is c:\Program Files (x86)\Amazon\AWSCLI\bin\aws-script.py

Full path on 64-bit version is C:\Program Files\Amazon\AWSCLI\bin\aws-script.py

I found this solution for a similar issue in this answer

Crispy Ninja
  • 348
  • 4
  • 11
Vereb
  • 14,388
  • 2
  • 28
  • 30
  • Thank you! This is the only solution that worked for me. In my 64 bit version the path is `C:\Program Files\Amazon\AWSCLI\bin\aws-script.py` and I stuck this line right after the `import sys`. – thund May 06 '19 at 23:06
  • 1
    Also, note that `aws-script.py` doesn't exist with older `aws-cli` installations so you may need to update to the latest one first with the msi installer. – thund May 17 '19 at 14:11
  • Worked for me on the 64-bit version as well. Well done! – Crispy Ninja May 20 '19 at 14:55
8

I had the same problem in Windows.

After investigating the problem, I realized that the problem is in the aws-cli installed using the MSI installer (x64). After removing "AWS Command Line Interface" from the list of installed programs and installing aws-cli using pip, the problem was solved.

I also tried to install MSI installer x32 and the problem was missing.

progm
  • 2,782
  • 3
  • 14
  • 32
  • 1
    Reinstalling AWS-CLI via pip worked for me, indeed. Here are the commands (Python and curl are needed) : `curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py` `python get-pip.py` `pip3 install awscli --upgrade --user` – JJP Mar 07 '19 at 10:06
  • 1
    @justin This should be the accepted answer. I had the same problem every time I did a `aws s3 cp --recursive s3://` and after using `pip` to install `awscli` I haven't had the problem since. – ClubbedAce Mar 24 '19 at 13:13
  • I will accept it for now, I haven't had time to verify for sure. – Justin Apr 03 '19 at 23:52
  • Beware that reinstalling with pip / pip3 does not work for everyone. I have a clean anaconda installation but trying to install awscli with pip gave errors about `no file association for .py`. Trying to hunt down those errors led me to numerous stackoverflow recommendations to install with msi instead. Eventually I went back to my msi installation and used @Vereb's solution below. – thund May 06 '19 at 23:16
  • 1
    Can verify this worked while the installer version was failing to upload a few JSONs at random out of several hundred, doing this fixed the issue. Running win10. – King Dedede May 15 '19 at 22:00
2

On Windows, this is probably an issue with the AWS-cli tools being installed using the MSI installer. It seems the best way to fix this issue is to use the aws-cli tools installed with python.

Prerequisites: Python3 and Pip3

Verify you have Python and Pip installed

C:\> python --version
Python 3.7.1
C:\> pip3 --version
pip 18.1 from c:\program files\python37\lib\site-packages\pip (python 3.7)

Uninstall the AWS-CLI tools MSI installer

Install the aws-cli tools with pip

C:\> pip3 install awscli

Check if aws-cli is in your path already ; if not, add it

C:\> where aws
C:\Program Files\Python37\Scripts\aws

If not, locate where the aws-cli tools are deployed and add the path to your PATH environment variable (can be set in Control panel > System > Advance system settings > Environment Variables...).

Below, common path of the deployed AWS cli tools:

Python 3 and pip3 – C:\Program Files\Python37\Scripts\

Python 3 and pip3 --user option on earlier versions of Windows – %USERPROFILE%\AppData\Local\Programs\Python\Python37\Scripts

Python 3 and pip3 --user option on Windows 10 – %USERPROFILE%\AppData\Roaming\Python\Python37\Scripts

Djidiouf
  • 786
  • 1
  • 10
  • 23
0

I've come across the same error and fixed by changing the order of AWC CLI and Python path:

Before:

PATH=C:\Program Files\Python35\Scripts\;C:\Program Files\Python35\;...;C:\Program Files\Amazon\AWSCLI\bin\;...

After:

PATH=...;C:\Program Files\Amazon\AWSCLI\bin\;...;C:\Program Files\Python35\Scripts\;C:\Program Files\Python35\

Looks like Python3 influences AWS CLI, which is just a cmd calling Python program.

Sig
  • 114
  • 4
  • hmm, it seems it doesn't work for some cases. The error occurs once every 10 times or more. Sounds like better using awscli pip package rather than Windows executable package. Will continue investigating. – Sig Jan 10 '19 at 16:44
-2

Even I was facing same issue. I was running it on Windows server 2008 R2. I was trying to upload around 500 files to s3 using below command.

aws s3 cp sourcedir s3bucket --recursive --acl bucket-owner-full-control --profile profilename

It works well and uploads almost all files, but for initial 2 or 3 files, it used to fail with error: An HTTP Client raised and unhandled exception: unknown encoding: idna

This error was not consistent. The file for which upload failed, it might succeed if I try to run it again. It was quite weird.

Tried on trial and error basis and it started working well.

Solution:

  1. Uninstalled Python 3 and AWS CLI.
  2. Installed Python 2.7.15
  3. Added python installed path in environment variable PATH. Also added pythoninstalledpath\scripts to PATH variable.
  4. AWS CLI doesnt work well with MS Installer on Windows Server 2008, instead used PIP.

Command:

pip install awscli

Note: for pip to work, do not forget to add pythoninstalledpath\scripts to PATH variable.

You should have following version:

Command:

aws --version

Output: aws-cli/1.16.72 Python/2.7.15 Windows/2008ServerR2 botocore/1.12.62

Voila! The error is gone!

Sameer D
  • 1
  • 1