How to delete partial duplicate lines with AWK?

Question

I have files with these kind of duplicate lines, where only the last field is different:

OST,0202000070,01-AUG-09,002735,6,0,0202000068,4520688,-1,0,0,0,0,0,55
ONE,0208076826,01-AUG-09,002332,316,3481.055935,0204330827,29150,200,0,0,0,0,0,5
ONE,0208076826,01-AUG-09,002332,316,3481.055935,0204330827,29150,200,0,0,0,0,0,55
OST,0202000068,01-AUG-09,003019,6,0,0202000071,4520690,-1,0,0,0,0,0,55

I need to remove the first occurrence of the line and leave the second one.

I've tried:

awk '!x[$0]++ {getline; print $0}' file.csv

but it's not working as intended, as it's also removing non duplicate lines.

Are your (near) duplicates always adjacent or can they be interspersed? — Dennis Williamson, Oct 19 '09 at 17:42
They are always adjacent, as they already are the result of another awk operations. — zedascouves, Oct 20 '09 at 10:18

Steven Huwig · Answer 1 · 2009-10-19T17:56:07.280

2

#!/bin/awk -f
{
    s = substr($0, 0, match($0, /,[^,]+$/))
    if (!seen[s]) {
        print $0
        seen[s] = 1
    }
}

edited Oct 19 '09 at 17:56

answered Oct 19 '09 at 17:19

Steven Huwig

20,015
9
55
79

This one needs an asterisk after the closing square bracket to match the correct substring. Apart from that, it is identical to `awk '!x[substr($0, 1,16)]++ ' file.csv`. They both suffer in that they print the first of a set of near duplicates, rather than the last. – Ewan Todd Oct 19 '09 at 17:40
Identical wrt this training data, that is – Ewan Todd Oct 19 '09 at 17:41
Thanks for the correction, and good catch on the OP's requirements – Steven Huwig Oct 19 '09 at 17:56
You can make this work "correctly" by sandwiching it between invocations of `tac`, e.g. `tac | script.awk file.txt | tac`. If you're lucky enough to have tac, of course. :) – Steven Huwig Oct 19 '09 at 18:04
I meant `tac | script.awk | tac file.txt` – Steven Huwig Oct 19 '09 at 18:05
tac file.csv|script.awk | tac – Ewan Todd Oct 19 '09 at 18:18
Nice solution! Can be combined with Dennis' solution in the case he identified. – Ewan Todd Oct 19 '09 at 18:18

score 1 · Answer 2 · answered Oct 19 '09 at 17:04

As a general strategy (I'm not much of an AWK pro despite taking classes with Aho) you might try:

Concatenate all the fields except the last.
Use this string as a key to a hash.
Store the entire line as the value to a hash.
When you have processed all lines, loop through the hash printing out the values.

This isn't AWK specific and I can't easily provide any sample code, but this is what I would first try.

Dennis Williamson · Accepted Answer · 2009-10-20T10:47:52.803

1

If your near-duplicates are always adjacent, you can just compare to the previous entry and avoid creating a potentially huge associative array.

#!/bin/awk -f
{
    s = substr($0, 0, match($0, /,[^,]*$/))
    if (s != prev) {
        print prev0
    }
    prev = s
    prev0 = $0
} 
END {
    print $0
}

Edit: Changed the script so it prints the last one in a group of near-duplicates (no tac needed).

edited Oct 20 '09 at 10:47

answered Oct 19 '09 at 18:02

Dennis Williamson

346,391
90
374
439

How to delete partial duplicate lines with AWK?

3 Answers3

Linked