2

I try to move from (stars are just here to indicate which lines I'm keeping)

*2020-12-15 19:10:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
2020-12-15 19:20:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
2020-12-15 19:30:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
*2020-12-15 19:40:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);100.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
2020-12-15 19:50:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);100.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
*2020-12-15 20:00:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm

to

2020-12-15 19:10:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
2020-12-15 19:40:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);100.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
2020-12-15 20:00:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm

I want to remove the duplicate lines from a group of identical lines without taking into account the date but I don't know how to go about it. I tried sorting from the second argument but it doesn't take into account the "group" of lines

cat <my file> | sort -t";" -k2 -u

Give me that

2020-12-15 19:40:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);100.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
2020-12-15 19:10:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm

Anyone have an idea ?

oguz ismail
  • 1
  • 16
  • 47
  • 69
Maxou
  • 59
  • 7

3 Answers3

4

Given the -m flag, sort assumes the input is already sorted and doesn't sort it again; which is exactly what you're looking for here.

$ sort -m -t';' -k2 -u <file
2020-12-15 19:10:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
2020-12-15 19:40:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);100.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
2020-12-15 20:00:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
oguz ismail
  • 1
  • 16
  • 47
  • 69
3

This is not really sorting. You are attempting to print a row if it is different from previous one, ignoring date-time column. You may try this awk:

awk -F ';' '{s=$0; sub(/^[^;]+;/, "", s)} p != s; {p=s}' file

2020-12-15 19:10:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
2020-12-15 19:40:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);100.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
2020-12-15 20:00:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
anubhava
  • 761,203
  • 64
  • 569
  • 643
3

Just tell uniq to skip the date field:

$ uniq -s 20 file
2020-12-15 19:10:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
2020-12-15 19:40:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);100.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
2020-12-15 20:00:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm
Ed Morton
  • 188,023
  • 17
  • 78
  • 185