1

I'm writing "bytes" to a file on s3 remote, using boto3.client. Is there a way to write data while compressing it simultaneously? eg. using some option like codec="snappy" or something like that?

Code:

s3 = boto3.client("s3")
bytesio = io.BytesIO()
bytesio.write(bytes_data)
bytesio.seek(0)
key = OUTPUT_REMOTE_DIR + test.avro
s3.put_object(Body=bytesio.getvalue(), Bucket=OUTPUT_BUCKET, Key=key)
vish4071
  • 5,135
  • 4
  • 35
  • 65
  • Do you just mean something like `Body=gzip.compress(bytesio.getvalue())` ? – Anon Coward Apr 09 '22 at 01:00
  • @AnonCoward I tried `snappy.compress(bytes_data)`, which compresses the actual data (for my use case, 2.8GB was down to 1.0GB). But then, when I'm reading the avro file back, I'm getting error and there is no way to specify `codec` in `fastavro.reader` – vish4071 Apr 09 '22 at 12:28

0 Answers0