Stack Overflow
Stack Overflow

Questions tagged [uniq]

uniq is a Unix/POSIX/Linux utility to remove or filter duplicate lines from a sorted file. It is also the name of a method to remove duplicates from an array in Ruby.

uniq is a Unix/POSIX/Linux utility to remove or filter duplicate lines from a sorted file. It is typically applied to the output of sort.

In Ruby , uniq is a method of the Array class to remove duplicates from an array. uniq creates a new array whereas uniq! modifies the array in place.

For questions about unique identifiers, keys, names, etc., see or more specific tags such as , , , , etc.

Documentation

454 questions
10
votes
2 answers

using Linux cut, sort and uniq

I have a list with population, year, and county and I need to cut the list, and then find the number of uniq counties. The list starts off like this: #Population, Year, County 3900, 1969, Beaver 3798, 1970, Beaver 3830, 1971, …
user2615699
10
votes
4 answers

Output whole line once for each unique value of a column (Bash)

This must surely be a trivial task with awk or otherwise, but it's left me scratching my head this morning. I have a file with a format similar to this: pep> AEYTCVAETK 2 genes ADUm.1024,ADUm.5198,ADUm.750 pep> AIQLTGK 1 genes…
Bede Constantinides
  • 2,424
  • 3
  • 25
  • 28
9
votes
1 answer

"isn't numeric" error in "sort" after "uniq"

use List::MoreUtils 'uniq'; print join ", ", sort uniq ("b", "a", "a"); results in Argument "a" isn't numeric in sort at ... print join ", ", uniq sort ("b", "a", "a"); works as expected. print join ", ", sort {$a cmp $b} uniq ("b", "a",…
André
  • 405
  • 5
  • 16
9
votes
2 answers

sort: string comparison failed Invalid or incomplete multibyte or wide character

I'm trying to use the following command on a text file: $ sort m.dict However I get the following error message: sort: string comparison failed: Invalid or incomplete multibyte or wide character sort: Set LC_ALL='C' to…
hjalpmig
  • 702
  • 1
  • 13
  • 39
8
votes
3 answers

bash tail on a live log file, counting uniq lines with same date/time

I'm looking for a good way to tail on a live log file, and display number of lines with the same date/time. Currently this is working: tail -F /var/logs/request.log | [cut the date-time] | uniq -c BUT the performance is not good enough. There is a…
zapp
  • 1,533
  • 1
  • 14
  • 18
8
votes
2 answers

Unix uniq command to CSV file

I have a text file (list.txt) containing single and multi-word English phrases. My goal is to do a word count for each word and write the results to a CSV file. I have figured out the command to write the amount of unique instances of each word,…
Abundnce10
  • 2,150
  • 3
  • 26
  • 41
8
votes
7 answers

Why uniq -c output with space instead of \t?

I use uniq -c some text file. Its output like this: 123(space)first word(tab)other things 2(space)second word(tab)other things .... So I need extract total number(like 123 and 2 above), but I can't figure out how to, because if I split this line…
MoreFreeze
  • 2,856
  • 3
  • 24
  • 34
7
votes
4 answers

how to sort based on a column but uniq based on another column?

He all, I have a file having some columns. I would like to do sort for column 2 then apply uniq for column 1. I found this post talking about sort and uniq for the same column but my problem is a little bit different. I am thinking of using…
Ken
  • 3,922
  • 9
  • 39
  • 40
7
votes
2 answers

easy way to change the uniq -c output?

I have a simple file like this: Term1 column2 column3 Term2 column2 column3 Term3 column2 column3 Term2 column2 column3 Term1 column2 column3 Term2 column2 column3 If I sort on the first column and get a count for the terms: cut -f1 -d…
Linda-Veronique
  • 103
  • 5
7
votes
5 answers

Find unique values from an array in React/js

I'm a beginner in JavaScript and an even bigger beginner in ReactJS. Most of the following I've taken from tutorials and am trying to adapt it. So the codes displays all the values from the "tag" (people, places, places, etc.). It displays it as…
Wasteland
  • 4,889
  • 14
  • 45
  • 91
7
votes
3 answers

Bash- is it possible to use -uniq for only one column of a line?

1.gui Qxx 16 2.gu Qxy 23 3.guT QWS 18 4.gui Qxr 21 i want to sort a file depending a value in the 3rd column, so i use: sort -rnk3 myfile 2.gu Qxy 23 4.gui Qxr 21 3.guT QWS 18 1.gui Qxx 16 now i have to output…
teutara
  • 605
  • 4
  • 12
  • 24
7
votes
2 answers

Case Insensitive Unique Array Elements in Perl

I am using the uniq function exported by the module, List::MoreUtils to find the uniq elements in an array. However, I want it to find the uniq elements in a case insensitive way. How can I do that? I have dumped the output of the Array using…
Neon Flash
  • 3,113
  • 12
  • 58
  • 96
7
votes
4 answers

Why does "uniq" count identical words as different?

I want to calculate the frequency of the words from a file, where the words are one by line. The file is really big, so this might be the problem (it counts 300k lines in this example). I do this command: cat .temp_occ | uniq -c | sort -k1,1nr -k2 >…
Epi
  • 682
  • 3
  • 10
  • 16
6
votes
4 answers

How to Remove duplication of words from both sentences using shell script?

I have a two sentences containing duplicate words, for example, the input data in file my_text.txt: The Unix and Linux operating system. The Unix and Linux system was to create an environment that promoted efficient program. I used this…
Abdallah_98
  • 1,331
  • 7
  • 17
6
votes
4 answers

BASH - Tell if duplicate lines exist (y/n)

I am writing a script to manipulate a text file. First thing I want to do is check if duplicate entries exist and if so, ask the user whether we wants to keep or remove them. I know how to display duplicate lines if they exist, but what I want to…
DMS
  • 327
  • 1
  • 3
  • 11
1
2
3
…
30 31