I have some code currently working that uses boto3 to iterate through lines of a text file in S3 by using s3_object.get()["Body"].iter_lines()
. Is there any way to do something similar with a gzipped text file? I have seen examples such as https://stackoverflow.com/a/59635286/177333, but as far as I can tell they load the whole file into memory. Is there any way to do this in chunks? I was hoping boto3 would detect the gzip header and do it automatically, but when trying to decode I got the expected UnicodeDetectError: 'utf-8' codec can't decode byte 0x8b in position 1
error.
Asked
Active
Viewed 243 times
2

Nelson Rothermel
- 9,436
- 8
- 62
- 81