Stack Overflow
Stack Overflow

Questions tagged [uniq]

uniq is a Unix/POSIX/Linux utility to remove or filter duplicate lines from a sorted file. It is also the name of a method to remove duplicates from an array in Ruby.

uniq is a Unix/POSIX/Linux utility to remove or filter duplicate lines from a sorted file. It is typically applied to the output of sort.

In Ruby , uniq is a method of the Array class to remove duplicates from an array. uniq creates a new array whereas uniq! modifies the array in place.

For questions about unique identifiers, keys, names, etc., see or more specific tags such as , , , , etc.

Documentation

454 questions
-1
votes
1 answer

Linux: Dedupe based on specific fields

I have a CSV file where I need to dedupe entries where the FIRST field matches, even if the other fields don't match. In addition, the line that is left should be the one where one of the other fields with the highest date. This what my data looks…
hamayoun
  • 159
  • 1
  • 8
-1
votes
2 answers

sort or unique command bash

This is a file . I want to remove the repetition in the name of patch [ppande@server-1 —]$egrep 'Patch[0-9].*.*:' content1 Patch1001 : snmp fixl.org Patch1002 : dhcp tmp fix Patch1003 : qemu-img-9.0.58 Patch001 : snmp fixl.org Patch002 : dhcp…
Unixquest945
  • 103
  • 2
  • 10
-1
votes
2 answers

grep just uniq data with latest time-stamp

I have a file test.txt A 2018-11-28 10:04:59.286 A 2018-11-28 10:07:59.286 B 2018-11-28 10:04:58.993 B 2018-11-28 10:06:58.993 C 2018-11-28 10:04:55.714 C 2018-11-28 10:04:59.714 I would like to pick just uniq data per each (A,B,C) based on latest…
PeterH
  • 27
  • 6
-1
votes
1 answer

Count unique words in all text files in directory, and delete those having less than 2?

This gets me the count. But how to delete those files having count < 2? $ cat ./a1esso.doc | grep -o -E '\w+' | sort -u -f | wc --words 1 $ cat ./a1brit.doc | grep -o -E '\w+' | sort -u -f | wc --words 4 How to grab the filenames of those that have…
Geoffrey Anderson
  • 1,534
  • 17
  • 25
-1
votes
1 answer

Unix awk - count of occurrences for each unique value

In Unix, I am printing the unique value for the first character in a field. I am also printing a count of the unique field lengths. Now I would like to do both together. Easy to do in SQL, but I'm not sure how to do this in Unix with awk (or…
user3439308
  • 485
  • 1
  • 6
  • 11
-1
votes
4 answers

Bash script that counts IDs in square brackets every ten minutes

Having this logfile 20180917084726:- 20180917085418:[111783178, 111557953, 111646835, 111413356, 111412662, 105618372, 111413557] 20180917115418:[111413432, 111633904, 111783198, 111792767, 111557948, 111413225, 111413281] 20180917105419:[111413432,…
Learner
  • 3
  • 2
-1
votes
1 answer

File is not sort after sort

I have a problem with sorting my file. My file look like this geom-10-11.com 1 geom-1-10.com 9 geom-1-11.com 10 geom-1-2.com 1 geom-1-3.com 2 geom-1-4.com 3 geom-1-5.com 4 geom-1-6.com 5 geom-1-7.com 6 geom-1-8.com 7 geom-1-9.com 8 geom-2-10.com…
McF44
  • 33
  • 6
-1
votes
3 answers

find file with highest alphabet character in file extension

How can I find file with highest alphabet character in the file's extension? Example of files my application creates: $ find . -name…
mauek unak
  • 702
  • 2
  • 11
  • 28
-1
votes
3 answers

How do I sort lines by number of appearances UNIX?

I want to sort input by number of appearances. However I don't want to delete either the unique or non-unique lines. For instance if I was given the following input: Not unique This line is unique Not unique Also not unique Also unique Also not…
trysofter
  • 11
  • 1
-1
votes
3 answers

awk remove mirrored duplicates from 2 columns

Big question: I want a list of the unique combinations between two fields in a data frame. Example data: A B C D E F B A C F E F I would like to be able to get the result of 4 unique combinations: AB, CD, EF, and CF. Since BA and and…
user4670961
  • 127
  • 2
  • 13
-1
votes
1 answer

How to sort content of arrays?

Ultimately, I want to get rid of the possibility of duplicate entries showing up my array. The reason I'm doing this is because I'm working on a script that compares two directories, searches for, and deletes duplicate files. The potential duplicate…
Alphatron
  • 83
  • 2
  • 10
-1
votes
1 answer

uniq on 3 columns and retain the lines based on fourth column in linux/unix

I have some data with four columns each and I want to keep the unique lines based on only the first 3 columns and retain their respective fourth column of maximum value also. My data looks looks like the following: chr1 5 10 1.5 chr1 5 10…
Naresh DJ
  • 91
  • 1
  • 9
-1
votes
1 answer

Uniq -c sort by column?

Can uniq -c only get the number of occurrences of column 2 and then sort it according the number of occurrence? Also, after sorting, I don't want to show the number of occurrences. How to do it? Here is my example input: 111111111 40 222222222…
WILLIAM
  • 457
  • 5
  • 28
-1
votes
1 answer

how to get second column by some simple cut or awk command after uniq -c option

I used following command on a file with 10 columns: cut -f3 f1.tsv | sort | uniq -c | sort -k1,1 -nr > f2.tsv which gave me a file like: 1230 ABC 1059 DEF 846 GHI 845 JKL but now for some other work, i need just the second column. So…
Shweta
  • 1,111
  • 3
  • 15
  • 30
-1
votes
2 answers

Linux/ unix duplicate names

What I need to do is, to check for duplicate domain names and find if there is some. So far I tried many commands with grep, awk ,sort, uniq but couldn't work this out, I am feeling its very simple, but can't reach it. P.s. If i use uniq -c I get a…
DevyDev
  • 846
  • 3
  • 14
  • 37
1 2 3
…
30 31