0

Let's say I have an input file that looks like this:

2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:15 > user1 has connected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:29 > user2 has connected.
2016-06-03 21:00:29 > user2 has connected.
2016-06-03 21:00:29 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.

I could remove all of the duplicate consecutive lines ignoring first 2 columns with uniq -f2 file.txt but I'm looking for a way to remove only the duplicates that have has connected. in them so the output will look like this:

2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:29 > user2 has connected.
2016-06-03 21:00:29 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.

I suppose this could be done just by matching a fixed string ("has connected.") but I'm also interested in a command that would work with a regex.

I took a look at the answers to this question but couldn't modify the commands so they would work with the input I have.

user2044638
  • 423
  • 1
  • 7
  • 18

4 Answers4

1
$ awk -F'>' '!(/has connected/ && seen[$2]++)' file
2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:29 > user2 has connected.
2016-06-03 21:00:29 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.
Ed Morton
  • 188,023
  • 17
  • 78
  • 185
1

A one-line Perl solution

perl -nE 'print unless /has connected/ && @s{/>\s+(.+)/}++' myfile.log

output

2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:29 > user2 has connected.
2016-06-03 21:00:29 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.

Note that the use of a hash slice @s{/>\s+(.+)/}++ is deliberate. It would normally be an error, but here it serves to put the regex in list context


If you want something cute like Chris Charley wrote that will report connected only if the user was previously disconnected, then it's not sensibly possible in a one-liner. This script will do that for you

If you're unfamiliar with Perl, then to run this on a file you should change <DATA> to <> and run the program like this

$ perl filter.pl myfile.log

use strict;
use warnings;

my %online;

while ( <DATA> ) {

    next unless my ($name, $op) = />\s+(.+)\s+(disconnected|has connected)\./;

    if ( $op eq 'disconnected' ) {
        delete $online{$name};
        print;
    }
    else {
        print unless $online{$name}++;
    }
}

__DATA__
2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:15 > user1 has connected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:15 > user1 disconnected.
2016-06-03 21:00:29 > user2 has connected.
2016-06-03 21:00:29 > user2 has connected.
2016-06-03 21:00:29 > user2 has disconnected.
2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:30 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.

output

2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:15 > user1 disconnected.
2016-06-03 21:00:29 > user2 has connected.
2016-06-03 21:00:29 > user2 has disconnected.
2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:30 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.
Community
  • 1
  • 1
Borodin
  • 126,100
  • 9
  • 70
  • 144
0

with awk:

awk -F">" '!($2 in a) || $2 ~ /disconnected/ {a[$2]=$2; print}' < file.txt

check if a value is already present in a array or bypasses if a string has 'disconnected' in it

!($2 in a) || $2 ~ /disconnected/

output

2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:29 > user2 has connected.
2016-06-03 21:00:29 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.
Chet
  • 1,205
  • 1
  • 12
  • 20
0

I think this perl solution might be what you want. I added some more lines in the data.

#!/usr/bin/perl
use strict;
use warnings;

my %seen;
while (<DATA>) {
    if (/ > (.+? connected)/) {
        print unless $seen{$1}++;
    }
    else {
        %seen = ();
        print;  
    }   
}

__DATA__
2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:15 > user1 has connected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:29 > user2 has connected.
2016-06-03 21:00:29 > user2 has connected.
2016-06-03 21:00:29 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.
2016-06-03 21:00:31 > user1 has connected.
2016-06-03 21:00:31 > user1 has connected.
2016-06-03 21:00:34 > user1 has connected.
2016-06-03 21:00:50 > user2 has connected.
2016-06-03 21:00:51 > user2 has connected.

This prints

2016-06-03 21:00:14 > user1 has connected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:22 > foobar disconnected.
2016-06-03 21:00:29 > user2 has connected.
2016-06-03 21:00:29 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.
2016-06-03 21:00:30 > user2 has disconnected.
2016-06-03 21:00:31 > user1 has connected.
2016-06-03 21:00:50 > user2 has connected.
Chris Charley
  • 6,403
  • 2
  • 24
  • 26