Reading and writing a gzip file at the same time

Question

I have seen python code where we first open a file to read it, and then again open it and write to it. When I try to do both at the same time it gives an error. From the documentation, I don't see an 'rwb' option. It's only one of create/read/write/append mode.

So what I was trying to do was:

with gzip.GzipFile(stage3_filepath, 'rwb') as file:
    data = json.loads(file.read().decode('utf-8'))
    data["articles"].extend(article_list)
    json_string = json.dumps(data, ensure_ascii=False)
    file.write(json_string.encode('utf-8'))

Just wanted to know why it is not possible to do so? And is there any alternative option?

As asked by others, my motive here is to avoid extra time taken to open a file twice. It seems less for 1 to 2 files but for a large collection of files in billion, this can be a huge improvement.

@DYZ my motive here is to avoid extra time taken to open a file twice. It seems less for 1 to 2 files but for a large collection of files in billion, this can be a huge improvement — Arjun Chaudhary, Nov 02 '20 at 05:57
Archive files are compressed files and the compression is based on the data contained inside. You cannot modify the compression. You can expand the compressed files and compress it back again after modification. For a large collection of files, you could parallelise the process but you cannot short circuit the compression. — TheGeorgeous, Nov 02 '20 at 06:05
@TheGeorgeous Actually, you _can_ extend gzipped files. Read the documentation. — DYZ, Nov 02 '20 at 06:06
AFAIK the compression is still lousy as compared to actually compressing the entire content from scratch https://stackoverflow.com/questions/18097107/python-gzip-appending-to-file-on-the-fly — TheGeorgeous, Nov 02 '20 at 06:11

score 1 · Answer 1 · answered Nov 03 '20 at 02:30

The alternative is open to read it, then open it to append. There is no option to open in read/write mode, because you cannot write a gzip file in the middle. You can only append another gzip member to the very end.

By the way, if you are appending gzip members, make sure they are big enough to benefit as they should from compression. At least several tens of K, if not megabytes or larger.

Reading and writing a gzip file at the same time

1 Answers1