3

Kind of the opposite of this question.

Is there a way to tell Python "Do not write to disk until I tell you to." (by closing or flushing the file)? I'm writing to a file on the network, and would rather write the entire file at once.

In the meantime, I'm writing to a StringIO buffer, and then writing that to the disk at the end.

Community
  • 1
  • 1
Jason Coon
  • 17,601
  • 10
  • 42
  • 50
  • Why? What possible value is there in waiting to become I/O bound when you're done computing? – S.Lott May 15 '09 at 03:17
  • The server is set up to commit the file-cache before it is closed, and it is slowing everything down. While this prevents data-loss, the file I'm writing is useless unless it is complete. Writing the file to disk at once has tested to be much faster. – Jason Coon May 15 '09 at 12:04

3 Answers3

3

No, a glance at the python manual does not indicate an option to set the buffer size to infinity.

Your current solution is basically the same concept.

You could use Alex's idea, but I would hazard against it for the following reasons:

  1. The buffer size on open is limited to 2^31-1 or 2 gigs. Any larger will result in "OverflowError: long int too large to convert to int"

  2. It doesn't seem to work:

    a = open("blah.txt", "w", 2 ** 31 - 1)
for i in xrange(10000): 
    a.write("a")

Open up the file without closing python, and you will see the text

nosklo
  • 217,122
  • 57
  • 293
  • 297
Unknown
  • 45,913
  • 27
  • 138
  • 182
3

You can open your file with as large a buffer as you want. For example, to use up to a billion bytes for buffering, x=open('/tmp/za', 'w', 1000*1000*1000) -- if you have a hundred billion bytes of memory and want to use them all, just add another *100...;-). Memory will only be consumed in the amount actually needed, so, no worry...

Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
1

I would say this partly depends on what you're trying to do.

The case where I came across this issue was when my application was a bit slow creating a file that was used by another application, the other application would get incomplete versions of the file.

I solved it by writing the file to a different place, then renaming it into the correct place once I'd finished writing.

If you want this for other reasons then maybe that doesn't help.

Colin Coghill
  • 1,549
  • 15
  • 18
  • This is worth testing. Writing the file locally and using the OS to copy it to the network might be faster. thanks – Jason Coon May 15 '09 at 12:05