Is there a simple way to do bulk file text substitution in place?

Question

I've been trying to code a Perl script to substitute some text on all source files of my project. I'm in need of something like:

perl -p -i.bak -e "s/thisgoesout/thisgoesin/gi" *.{cs,aspx,ascx}

But that parses all the files of a directory recursively.

I just started a script:

use File::Find::Rule;
use strict;

my @files = (File::Find::Rule->file()->name('*.cs','*.aspx','*.ascx')->in('.'));

foreach my $f (@files){
    if ($f =~ s/thisgoesout/thisgoesin/gi) {
           # In-place file editing, or something like that
    }
}

But now I'm stuck. Is there a simple way to edit all files in place using Perl?

Please note that I don't need to keep a copy of every modified file; I'm have 'em all subversioned =)

Update: I tried this on Cygwin,

perl -p -i.bak -e "s/thisgoesout/thisgoesin/gi" {*,*/*,*/*/*}.{cs,aspx,ascx

But it looks like my arguments list exploded to the maximum size allowed. In fact, I'm getting very strange errors on Cygwin...

Answer 1

If you assign @ARGV before using *ARGV (aka the diamond <>), $^I/-i will work on those files instead of what was specified on the command line.

use File::Find::Rule;
use strict;

@ARGV = (File::Find::Rule->file()->name('*.cs', '*.aspx', '*.ascx')->in('.'));
$^I = '.bak';  # or set `-i` in the #! line or on the command-line

while (<>) {
    s/thisgoesout/thisgoesin/gi;
    print;
}

This should do exactly what you want.

If your pattern can span multiple lines, add in a undef $/; before the <> so that Perl operates on a whole file at a time instead of line-by-line.

Answer 2

You may be interested in File::Transaction::Atomic or File::Transaction

The SYNOPSIS for F::T::A looks very similar with what you're trying to do:

  # In this example, we wish to replace 
  # the word 'foo' with the word 'bar' in several files, 
  # with no risk of ending up with the replacement done 
  # in some files but not in others.

  use File::Transaction::Atomic;

  my $ft = File::Transaction::Atomic->new;

  eval {
      foreach my $file (@list_of_file_names) {
          $ft->linewise_rewrite($file, sub {
               s#\bfoo\b#bar#g;
          });
      }
  };

  if ($@) {
      $ft->revert;
      die "update aborted: $@";
  }
  else {
      $ft->commit;
  }

Couple that with the File::Find you've already written, and you should be good to go.

Answer 3

You can use Tie::File to scalably access large files and change them in place. See the manpage (man 3perl Tie::File).

Answer 4

Change

foreach my $f (@files){
    if ($f =~ s/thisgoesout/thisgoesin/gi) {
           #inplace file editing, or something like that
    }
}

To

foreach my $f (@files){
    open my $in, '<', $f;
    open my $out, '>', "$f.out";
    while (my $line = <$in>){
        chomp $line;
        $line =~ s/thisgoesout/thisgoesin/gi
        print $out "$line\n";
    }
}

This assumes that the pattern doesn't span multiple lines. If the pattern might span lines, you'll need to slurp in the file contents. ("slurp" is a pretty common Perl term).

The chomp isn't actually necessary, I've just been bitten by lines that weren't chomped one too many times (if you drop the chomp, change print $out "$line\n"; to print $out $line;).

Likewise, you can change open my $out, '>', "$f.out"; to open my $out, '>', undef; to open a temporary file and then copy that file back over the original when the substitution's done. In fact, and especially if you slurp in the whole file, you can simply make the substitution in memory and then write over the original file. But I've made enough mistakes doing that that I always write to a new file, and verify the contents.

---

Note, I originally had an if statement in that code. That was most likely wrong. That would have only copied over lines that matched the regular expression "thisgoesout" (replacing it with "thisgoesin" of course) while silently gobbling up the rest.

Answer 5

You could use find:

find . -name '*.{cs,aspx,ascx}' | xargs perl -p -i.bak -e "s/thisgoesout/thisgoesin/gi"

This will list all the filenames recursively, then xargs will read its stdin and run the remainder of the command line with the filenames appended on the end. One nice thing about xargs is it will run the command line more than once if the command line it builds gets too long to run in one go.

Note that I'm not sure whether find completely understands all the shell methods of selecting files, so if the above doesn't work then perhaps try:

find . | grep -E '(cs|aspx|ascx)$' | xargs ...

When using pipelines like this, I like to build up the command line and run each part individually before proceeding, to make sure each program is getting the input it wants. So you could run the part without xargs first to check it.

It just occurred to me that although you didn't say so, you're probably on Windows due to the file suffixes you're looking for. In that case, the above pipeline could be run using Cygwin. It's possible to write a Perl script to do the same thing, as you started to do, but you'll have to do the in-place editing yourself because you can't take advantage of the -i switch in that situation.

Answer 6

Thanks to ephemient on this question and on this answer, I got this:

use File::Find::Rule;
use strict;

sub ReplaceText {
    my $regex = shift;
    my $replace = shift;

    @ARGV = (File::Find::Rule->file()->name('*.cs','*.aspx','*.ascx')->in('.'));
    $^I = '.bak';
    while (<>) {
        s/$regex/$replace->()/gie;
        print;
    }
}

ReplaceText qr/some(crazy)regexp/, sub { "some $1 text" };

Now I can even loop through a hash containing regexp=>subs entries!