2

Let's say I have a file /etc/conf1

it's contents are along the lines of 

option = banana
name = monkey
operation = eat

and let's say I want to replace "monkey" with "ostrich". How can I do that without reading the file to memory, altering it and then just writing it all back? Basically, how can I modify the file "in place"?

Neoscopio
  • 21
  • 1
  • 1
    With difficulty. What do you think you would achieve by modifying "in place"? Is it a big file and you're worried about holding it in memory? – MattH Jul 05 '10 at 08:49
  • I think you can't if you want to change the file size. Also `sed` cannot do this, it's simply not supported by current operating systems. As an alternative, you might look into the `ConfigParser` module for parsing INI-style configuration files. – Philipp Jul 05 '10 at 08:49
  • In the current case, it more of a hypothetical "what are the chances of me reading a file, modifying it, writing it and have a problem midway through and ending up with a corrupted file?" problem. I thought about a "read conf1, modify, write to conf1.temp, mv conf1.temp conf1" approach, but I thought there could be a more elegant way to do this. In place mod would be cool. – Neoscopio Jul 05 '10 at 09:36

6 Answers6

2

You can't. "ostrich" is one letter more than "monkey", so you'll have to rewrite the file at least from that point onwards. File systems do not support "shifting" file contents upwards or downwards.

If it's just a small file, there's no reason to bother with even this, and you might as well rewrite the whole file.

If it's a really large file, you'll need to reconsider the internal design of the file's contents, for example, with a block-based approach.

Thomas
  • 174,939
  • 50
  • 355
  • 478
  • This is something that has bothered me for some time and I could never figure out how to do this. I just thought I sucked at manipulating files. I actually once implemented the "read block, modify, write block" approach a while back in python and was told that was a stupid thing to do because when you f.read(), the system actually only loads up a few blocks at a time... and yeah, it was a large (100MB large) file. – Neoscopio Jul 05 '10 at 09:34
1

You should look at the fileinput module:

http://docs.python.org/library/fileinput.html

There's an option to perform inplace editing via the input method:

http://docs.python.org/library/fileinput.html#fileinput.input

UPDATE - example code:


import fileinput
import re
import sys

for line in fileinput.input(inplace=True):
    sys.stdout.write(re.sub(r'monkey', 'ostrich', line))

Using sys.stdout.write so as not to add any extra newlines in.

John Montgomery
  • 8,868
  • 4
  • 33
  • 43
  • Tried it, couldn't get it to work. The fact that fileinput exists is the reason I'm posing this question. Can you elaborate or provide an example please? – Neoscopio Jul 05 '10 at 09:30
  • I've added some example code now. NB it does technically read the file into memory (a line at a time), but this is equivalent to how sed operates anyway. The key thing is you have to output the line even if you haven't changed it. – John Montgomery Jul 06 '10 at 09:44
0

It depends on what you mean by "in place". How can you do it if you want to replace monkey with supercalifragilisticexpialidocious? Do you want to overwrite the remaining file? If not, you are going to have to read ahead and shift subsequent contents of the file forwards.

fmark
  • 57,259
  • 27
  • 100
  • 107
0

CPU instructions operate on data which come from memory.

The portion of the file you wish to read must be resident in memory before you can read it; before you write anything to disk, that information must be in memory.

The whole file doesn't have to be there at once, but to do a search-replace on an entire file, every character of the file will pass through RAM at some point.

What you're probably looking for is something like the mmap() system call. The above fileinput module sounds like a plausible thing to use.

Borealid
  • 95,191
  • 9
  • 106
  • 122
  • If you just parse until you reach the config option you want, you only read the whole file in the worst case scenario. My problem was related to shifting the contents of the file, I guess... – Neoscopio Jul 05 '10 at 09:40
0

In-place modifications are only easy if you don't alter the size of the file or only append to it. The following example replaces the first byte of the file by an "a" character:

fd = os.open("...", os.O_WRONLY | os.O_CREAT)
os.write(fd, "a")
os.close(fd)

Note that Python's file objects don't support this, you have to use the low-level functions. For appending, open file file with the open() function in "a" mode.

Philipp
  • 48,066
  • 12
  • 84
  • 109
  • `fp= open("…", "r+b"); fp.write("a"); fp.close()` Now, have I misunderstood you when you said “Python's `file` objects don't support this”? – tzot Aug 02 '10 at 21:15
0

sed -i.bak '/monkey$/newword/' file

ghostdog74
  • 327,991
  • 56
  • 259
  • 343