Remove duplicate numbers within parentheses using sed

Question

I am trying to remove duplicate numbers within parentheses using sed.

So I have the following string:

Abdc 1234 1234 (5678) (5678) (9012) (9012) (3456)

I want to use sed to remove any 4-digit numbers within the parentheses, including the parentheses. So my string should look like this:

Abdc 1234 1234 (5678) (9012) (3456)

In this case the "(5678)" and "(9012)" were removed because they were 4-digit numbers within parentheses that repeated. The "1234" numbers were not removed because they were not within parenthesis. The "(3456)" was not removed because it was not repeating.

do you have multiple lines like that example? do any repeated groups always immediately follow their first occurrence on the line? I'm not sure any regex matching tool is appropriate for this job... — Greg A. Woods, Nov 14 '12 at 00:35
No, just one line so no CR / NL. Yes, the repeats will always be consecutive. — user1822391, Nov 14 '12 at 09:36
I had found this post to remove repeating words. [link](http://stackoverflow.com/questions/10600056/remove-duplicate-words-in-a-line-with-sed) I got it to work easily for groups of repeating numbers: `# echo "Testing 1234 1234 5678" | sed -e 's/\b\([0-9]\+\)[ ,\n]\1/\1/g'` However, not knowing a lot about regex/sed, I could not get it to work with the number-group enclosed in parentheses. — user1822391, Nov 14 '12 at 09:44

score 5 · Accepted Answer · answered Nov 14 '12 at 00:42

I do not know how to do this with sed but you could try the following with awk:

$  echo "Abdc 1234 1234 (5678) (5678) (9012) (9012) (3456)" | awk '
   {
     for(i=1;i<=NF;i++) { 
       if(substr($i,0,1) != "(" || (seen[$i] != 1)) {
         seen[$i]=1;
         printf "%s ",$i
       }
     };
     print ""
   }'

Output:

Abdc 1234 1234 (5678) (9012) (3456)

This loops through the line fields then prints each field only if it has never been seen before or if it is not starting with (.

Thor · Answer 2 · 2012-11-14T07:43:31.553

2

This works for your input:

echo 'Abdc 1234 1234 (5678) (5678) (9012) (9012) (3456)' | 
  sed 's/\(([0-9][0-9]*)\) \1/\1/g'

It assumes duplicates follow each other, if that is not the case, use this version:

echo 'Abdc 1234 1234 (5678) (5678) (9012) (9012) (3456)' | 
  sed 's/\(([0-9][0-9]*)\) \(.*\)\1/\1\2/g'

Or a bit shorter with GNU sed extended expressions:

echo 'Abdc 1234 1234 (5678) (5678) (9012) (9012) (3456)' | 
  sed -r 's/(\([0-9]+\)) (.*)\1/\1\2/g'

Output in all cases:

Abdc 1234 1234 (5678) (9012) (3456)

Edit - handle situation where more than two identical items exist

This can be done by looping over the pattern until it no longer matches:

echo 'Abdc 1234 1234 (5678) (5678) (9012) (9012) (3456) (5678) (5678)' | 
  sed -r ':a; s/(\([0-9]+\))(.*)\1 ?/\1\2/g; ta'

edited Nov 14 '12 at 07:43

answered Nov 14 '12 at 00:49

Thor

45,082
11
119
130

Great idea! But none of these would work if the duplicated entries are present more than twice (such as `(5678) (5678) (5678)` as input). This once again depends on the input format constraints. – Julien Vivenot Nov 14 '12 at 00:57
@jvivenot: I see, you could handle this by looping the `s///`, see edit. – Thor Nov 14 '12 at 01:04

Gilles Quénot · Answer 3 · 2012-11-14T01:43:15.827

Using Perl :

$ echo "Abdc 1234 1234 (5678) (5678) (9012) (9012) (3456)" |
    perl -ne '
        my (@arr, %hash);
        for (split) {
            if (/^\(.*\)/) {
                $hash{$_}++;
                push @arr, $_ if $hash{$_} == 1;
            }
            else {
                push @arr, $_; 
            }
        }
        print join " ", @arr, "\n";
    '

That will works with multi line as input and N occurrences of repeated stuff with parenthesis.

score 0 · Answer 4 · answered Nov 14 '12 at 07:19

0

This might work for you (GNU sed):

sed ':a;s/\(\(([0-9]\+) *\).*\)\2/\1/g;ta' file

answered Nov 14 '12 at 07:19

potong

55,640
6
51
83

Vijay · Answer 5 · 2012-11-14T07:28:14.723

awk -F"(" '{for(i in a)delete a[i];for(i=2;i<=NF;i++){if($i in a){$i="";}else{a[$i];$i="("$i}}print $0}' your_file

Tested below: input:

> cat temp
Abdc 1234 1234 (5678) (5678) (9012) (9012) (3456)
1234 1234 (1234) (5678) (9012) (1234) (3456)
 (5678) (6467) (6467) (9012) (5678)

Now the execution:

> awk -F"(" '{for(i in a)delete a[i];for(i=2;i<=NF;i++){if($i in a){$i="";}else{a[$i];$i="("$i}}print $0}' temp
Abdc 1234 1234  (5678)   (9012)   (3456)
1234 1234  (1234)  (5678)  (9012)   (3456)
  (5678)  (6467)   (9012)  (5678)
>

Remove duplicate numbers within parentheses using sed

5 Answers5

Edit - handle situation where more than two identical items exist