2

I have some code currently working that uses boto3 to iterate through lines of a text file in S3 by using s3_object.get()["Body"].iter_lines(). Is there any way to do something similar with a gzipped text file? I have seen examples such as https://stackoverflow.com/a/59635286/177333, but as far as I can tell they load the whole file into memory. Is there any way to do this in chunks? I was hoping boto3 would detect the gzip header and do it automatically, but when trying to decode I got the expected UnicodeDetectError: 'utf-8' codec can't decode byte 0x8b in position 1 error.

Nelson Rothermel
  • 9,436
  • 8
  • 62
  • 81

0 Answers0