uniq command not detected duplicated lines

Question

I'm facing a difficulty with the unix uniq command. I have a file containing a list of ids that looks like this (output from head -5 list.txt) :

IBNUKWG02JZU4E
IBNUKWG02JZULO
IBNUKWG02JZUMG
IBNUKWG02JZUZS
IBNUKWG02JZV0R

The files contains 619142 lines (cat list.txt | wc -l ), and contains duplicate, for example if I run the command ( -c flag returns the number of occurrence of the line )

cat list.txt | grep IBNUKWG02JZULO | uniq -c

it returns

  2 IBNUKWG02JZULO

but if I run the command ( -u flag to only print unique lines)

   cat list.txt | uniq -u | wc -l

it returns 619142, as if duplicated lines weren't detected. How is it possible ?

The lines need to be sorted, uniq only checks adjacent lines. — 123, May 15 '17 at 11:13
ok thanks I lot, looks liek I didn't pay enough attention to the `uniq` documentation — felix, May 15 '17 at 11:16

score 6 · Answer 1 · answered May 15 '17 at 11:11

6

before using uniq use sort.

cat list.txt | sort | uniq -u | wc -l

answered May 15 '17 at 11:11

tso

4,732
2
22
32

2

The useless use of `cat` can be avoided here, just `sort list.txt | uniq -c` would be good – Inian May 15 '17 at 11:20
yep, i know, just copied from his question. :))) – tso May 15 '17 at 11:21

uniq command not detected duplicated lines

1 Answers1