(Mass) text processing in place (in bash)

Question

How do I process a file or multiple files in place with bash?

So: read from file x, do some processing (e.g. search-replace) and write to file x.

I know that with sed you can do: sed -i "" "s/original/replacement/g", but sometimes sed doesn't cut it and I need a different tool, which lacks an -i ""-like option.

I recently discovered all by myself that I could do the following:

( BUFFER="`cat FILENAME`"; echo "$BUFFER" > FILENAME )

which uses an environment variable to store the contents of the file. For very large files this is probably not efficient and environment variables might be limited in capacity.

Is there a different way?

I also read this answer, but I'm looking for a bash solution. Open source tools that somehow fit the task perfectly are also welcome.

of course... [/facepalm], tell you what, you make it into an answer, and I'll accept it :D — pancake, Aug 29 '13 at 15:24

score 2 · Answer 1 · answered Aug 29 '13 at 15:27

There are many scripting tools around like awk, perl, ruby, python, but with large files in bash it would just be better to store the output first on a temporary file. Then save it back:

while IFS= read -r LINE; do
    case "$LINE" in
    something_to_exclude|another_to_exclude)
        ;;
    yet_another_to_exclude)
        ;;
    *)
        # This is fine to include.
        echo "$LINE"
        ;;
    eac
done < "$FILENAME" > "$FILENAME".temp

cat "$FILENAME".temp > "$FILENAME"
rm "$FILENAME".temp

+1 for the `while IFS=read line ... done < file > otherfile`, didn't know that was possible! — pancake, Aug 29 '13 at 15:35

user000001 · Accepted Answer · 2013-08-29T15:43:15.120

0

You can store the output of the command in a temp file, and then move the temp file over the original. Example:

command <file >tmp && mv tmp file

edited Aug 29 '13 at 15:43

answered Aug 29 '13 at 15:28

user000001

32,226
12
81
108

1

Sometimes it's better to use cat + rm if something in the original file has to be preserved like permissions. Or when re-creating the file is not possible, or if you can't do chmod for some reason and that you only have write access. Note that such conditionally can't always apply with the temporary file for it could be placed somewhere. – konsolebox Aug 29 '13 at 15:43

(Mass) text processing in place (in bash)

2 Answers2