botocore.excceptions.ClientError: An error occurred (InvalidTextEncoding) when calling the SelectObjectContent operation

Question

while executing below code through python

response= S3.select_object_content(Bucket=S3_bucket_name,Key=S3_file_Key,ExpressionType='SQL', Expression="select count(*) from s3object", InputSerialization={'CSV': {"FileHeaderInfo": header_usage},'CompressionType':compressformat}, OutputSerialization={'CSV': {}},)

I am getting error like

Traceback (most recent call last):

  File OutputSerialization={'CSV': {}},)
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 623, in _make_api_call
    raise error_class(parsed_response, operation_name)

**ClientError: An error occurred (InvalidTextEncoding) when calling the SelectObjectContent operation: UTF-8 encoding is required. The text encoding error was found near byte 49,152.**

I searched for Invalid text Encoding in boto3 but couldn't found. Can you please help me to check this?

Thanks in advance

Did you manage to solve your problem? And did my answer help? — Psychotechnopath, Dec 10 '18 at 17:35
Hey apologies for late reply.... I am checking for expand OutputSerialization={'CSV': {}}.. I am not getting relevant ans :( — Monika Bhajipale, Dec 11 '18 at 14:30
what happens if you put a parameter, encoding = 'utf-8' somewhere in the select_object_content? I think that is where it is going wrong. — Psychotechnopath, Dec 11 '18 at 19:10
Are you sure you are opening a CSV file? You will get similar error if the file format is parquet or gzip. In that case change the 'csv' Input Serialization. — shantanuo, Dec 16 '20 at 07:25

score 0 · Answer 1 · answered Dec 07 '18 at 10:27

The data you wish you receive has the wrong Output serialization. The output serialization Describes the format of the data that you want Amazon S3 to return in response, and you are asking it to return a format that has the wrong encoding. I cannot test your code myself because I only have small bits of it, but you need to encode your serialized output to utf-8 format, otherwise the Amazon S3 storage service can't serialize your response. Probably you need to expand OutputSerialization={'CSV': {}} to make sure that your respone is coded in UTF-8 format.

Maybe these resources can help you:

Select object content parameter guide

select Object Content description

score 0 · Answer 2 · answered Sep 28 '22 at 12:09

0

Sadly "UTF-8 encoding is required." means that object is not matching required format.

Reference: https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html

UTF-8 - UTF-8 is the only encoding type Amazon S3 Select supports.

answered Sep 28 '22 at 12:09

SantiagoRodriguez

188
2
10

botocore.excceptions.ClientError: An error occurred (InvalidTextEncoding) when calling the SelectObjectContent operation

2 Answers2