perl to delete consecutive duplicate lines

Question

I want to delete the consecutive duplicate lines. i.e. for example

**test.txt**
car
speed is good
bike 
slower than car
plane
super fast
super fast
bullet train 
super fast

This removes all the duplicate lines except the 1st occurance.

perl -ne 'print unless $a{$_}++'

But i want the ouput to be

    **test.txt**
    car
    speed is good
    bike 
    slower than car
    plane
    super fast
    bullet train 
    super fast

I tried this oneliner but this doesnt do anything but just prints the input.

perl -00 -F'<\w+>|</\w+>' -i.bak -lane 'foreach(@F){if ($_=~/\w+/ && ($a ne $_)){print "$_";$a=$_;}}'

How to do this???

If you're on Linux (or have Cygwin on Windows) use [uniq(1)](http://unixhelp.ed.ac.uk/CGI/man-cgi?uniq) instead. — Jim Garrison, Sep 17 '12 at 03:07

Steve · Answer 1 · 2014-06-11T22:40:33.567

10

Why don't you just use uniq?

uniq file.txt

Results:

car
speed is good
bike 
slower than car
plane
super fast
bullet train 
super fast

You can also do this with awk:

awk 'line != $0; { line = $0 }' file.txt

edited Jun 11 '14 at 22:40

answered Sep 17 '12 at 03:08

Steve

1

Time for a file with 1M lines: uniq: 1.579 seconds. awk: 10.615 seconds. – erik Jan 31 '13 at 11:31

score 5 · Answer 2 · edited Sep 17 '12 at 07:40

5

$ perl -ne 'print $_ unless $_ eq $prev; $prev = $_'

edited Sep 17 '12 at 07:40

Zaid

answered Sep 17 '12 at 03:04

Vipul Ved Prakash

This will warn about `$prev` being undefined – epsalon Sep 17 '12 at 03:05
1

No, it won't. "perl -nwe" would. – Vipul Ved Prakash Sep 17 '12 at 03:06
1

Time for a file with 1M lines: 0.646 seconds. After several tests: Always a bit faster than epsalons answer. – erik Jan 31 '13 at 11:33

score 5 · Accepted Answer · answered Sep 17 '12 at 03:04

5

Try:

perl -ne 'print unless (defined($prev) && ($_ eq $prev)); $prev=$_'

answered Sep 17 '12 at 03:04

epsalon

score 0 · Answer 4 · answered Jul 22 '15 at 14:18

I also wanted to keep track of how many duplicates were suppressed and only skip consecutive duplicates.

While this is not exactly what the OP asked, it is a variant which others may find useful:

perl -ne 'if (defined($pr) && ($_ eq $pr)) {$cnt++;} else {print "... (+$cnt)\n" if ($cnt); print; $cnt=0; $pr=$_;}'

It produced something like this with my data (a database restore log):

COPY 9
COPY 0
... (+2)
COPY 5
COPY 0
... (+1)
COPY 24
ALTER TABLE
... (+23)
CREATE INDEX
... (+73)

4 Answers4