Stack Overflow
Stack Overflow

Questions tagged [uniq]

uniq is a Unix/POSIX/Linux utility to remove or filter duplicate lines from a sorted file. It is also the name of a method to remove duplicates from an array in Ruby.

uniq is a Unix/POSIX/Linux utility to remove or filter duplicate lines from a sorted file. It is typically applied to the output of sort.

In Ruby , uniq is a method of the Array class to remove duplicates from an array. uniq creates a new array whereas uniq! modifies the array in place.

For questions about unique identifiers, keys, names, etc., see or more specific tags such as , , , , etc.

Documentation

454 questions
3
votes
2 answers

Ruby 1.8.6 Array#uniq not removing duplicate hashes

I have this array, in a ruby 1.8.6 console: arr = [{:foo => "bar"}, {:foo => "bar"}] both elements are equal to each other: arr[0] == arr[1] => true #just in case there's some "==" vs "===" oddness... arr[0] === arr[1] => true But, arr.uniq…
Max Williams
  • 32,435
  • 31
  • 130
  • 197
3
votes
1 answer

Removing blankspace at the start of a line (size of blankspace is not constant)

I am a beginner to using sed. I am trying to use it to edit down a uniq -c result to remove the spaces before the numbers so that I can then convert it to a usable .tsv. The furthest I have gotten is to use: $ sed 's|\([0-9].*$\)|\1|'…
Rob Sellers
  • 31
  • 4
3
votes
2 answers

Remove semi-unique string in file

I am writing a script to track changes in my routing tables. The script pulls down a snapshot of the routing table every 5 minutes and diffs the table against the version that is 5 minutes old. If there is a deviation in the file, the script…
FORCED-INDUCTN
  • 31
  • 2
3
votes
5 answers

How to remove lines appear only once in a file using bash

How can I remove lines appear only once in a file in bash? For example, file foo.txt has: 1 2 3 3 4 5 after process the file, only 3 3 will remain. Note the file is sorted already.
user200340
  • 3,301
  • 13
  • 52
  • 74
3
votes
3 answers

Display unique values in each column using awk

I'm relatively new to using awk/grep etc and want to filter some data. I have a large spreadsheet which I want to display the unique values column by column. For example I want to change this: DS571187 DS571220 DS571200 DS571194 contig1 …
A. Leckenby
  • 31
  • 3
3
votes
5 answers

How to count the amount of unique lines, duplicate lines and lines that appear three times in a text file

I have a list of names, one name per line saved as a .txt file. I'm trying to use bash to determine how many different names appear once, two times or three times. For example: names.txt looks like Donald Donald Lisa John Lisa Donald In this case…
WhatAmIDoing
  • 33
  • 1
  • 3
3
votes
1 answer

uniq -c without additional spaces

Is there an option in uniq -c (or an alternative) that doesn't add additional whitespaces around the count number? Currently I generally pipe it through sed, like so: sort | uniq -c | sed 's/^ *\([0-9]*\) /\1 /' But this seems kinda redundant,…
JohnnyTooBad
  • 77
  • 2
  • 6
3
votes
1 answer

PHP Array permutation uniq ordered

TURNS OUT THAT MY MATH OPERATION WAS THE WRONG ONE! READ PHP array combinations FOR AN EXAMPLE OF COMBINATION I Am searching the Web now for 3 weeks ( including SO) i dont find someone who has already asked or solved it. I need a Permutation of an…
KikiTheOne
  • 523
  • 4
  • 13
3
votes
2 answers

Using awk to get the maximum value of a column, for each unique value of another column

So I have a file such as: 10 1 abc 10 2 def 10 3 ghi 20 4 elm 20 5 nop 20 6 qrs 30 3 tuv I would like to get the maximum value of the second column for each value of the first column, i.e.: 10 3 ghi 20 6 qrs 30 3 tuv How can I do using awk or…
leonard vertighel
  • 1,058
  • 1
  • 18
  • 37
3
votes
3 answers

Advanced `uniq` with "unique part regex"

uniq is a tool that enables once to filter lines in a file such that only unique lines are shown. uniq has some support to specify when two lines are "equivalent", but the options are limited. I'm looking for a tool/extension on uniq that allows one…
Willem Van Onsem
  • 443,496
  • 30
  • 428
  • 555
3
votes
1 answer

`uniq` not working as expected

I have file called "test.txt" which looks like this: 10 10 10 8 10 9 10 10 9 10 8 For some reason, when I ran uniq test.txt, I got this output: 10 8 10 9 10 9 10 8 Why am I getting this output? I am using BSD uniq. Is there some sort of bug in the…
lutaoact
  • 4,149
  • 6
  • 29
  • 41
3
votes
3 answers

shell: select unique row of flat file

I have a flat file looks like this cat file ID1, VALUE1_1 ID1, VALUE1_2 ID1, VALUE1_3 ID2, VALUE2_1 ID2, VALUE2_1 ID3, VALUE3_1 ID3... As you can see from the data sample, for each ID, there are several values for that ID and they could be…
B.Mr.W.
  • 18,910
  • 35
  • 114
  • 178
3
votes
3 answers

calling uniq and sort in different orders in shell

Is there a difference in the order of uniq and sort when calling them in a shell script? I’m talking here about time- and space-wise. grep 'somePattern' | uniq | sort vs. grep 'somePattern' | sort | uniq a quick test on a 140 k lines textfile…
knittl
  • 246,190
  • 53
  • 318
  • 364
3
votes
4 answers

Unix uniq, sort & cut command remove duplicate lines

If we have the following result: Operating System,50 Operating System,40 Operating System,30 Operating System,23 Data Structure,87 Data Structure,21 Data Structure,17 Data Structure,8 Data Structure,3 Crypo,33 Crypo,31 C++,65 C Language,39 C…
eleven
  • 359
  • 1
  • 5
  • 14
3
votes
2 answers

How to use uniq -cd in bash scripting and extract only the count and not the line?

I have a .sh file that takes a log file and extracts data and makes reports. I would like to calculate what percentage of the total lines does an error pop-up (top talkers). So far I have this: awk '// {print $4, substr($0, index($0,$9))}' | sort…
MikeyC343
  • 139
  • 2
  • 11
1 2 3
…
30 31