Stack Overflow
Stack Overflow

Questions tagged [uniq]

uniq is a Unix/POSIX/Linux utility to remove or filter duplicate lines from a sorted file. It is also the name of a method to remove duplicates from an array in Ruby.

uniq is a Unix/POSIX/Linux utility to remove or filter duplicate lines from a sorted file. It is typically applied to the output of sort.

In Ruby , uniq is a method of the Array class to remove duplicates from an array. uniq creates a new array whereas uniq! modifies the array in place.

For questions about unique identifiers, keys, names, etc., see or more specific tags such as , , , , etc.

Documentation

454 questions
3
votes
2 answers

RoR ActiveRecord::Base uniq undefined when trying to uniq.pluck(:column_name)

so I have a very simple model called Movies. I'm trying to add a class method which returns a tidy list of the ratings assigned to movies in the database. It seems I'd want to call Movie.uniq.pluck(:rating) So I've added the method like so: class…
BigglesB
  • 123
  • 2
  • 8
3
votes
3 answers

How can I emulate `uniq -d` in awk?

I've got a busybox system which doesn't have uniq and I'd like to generate a unique list of duplicated lines. A plain uniq emulated in awk would be: sort | awk '!($0 in a){a[$0]; print}' How can I use awk (or sed for that matter, not…
Jamie
  • 7,075
  • 12
  • 56
  • 86
2
votes
2 answers

merging 2 files into a third one, using columns as index and merging lines too

I've been studying awk and i've come upon a problem i'm not being able to solve, please help if you can. I have 2 files I generated using awk, sort and uniq -c. File 1 is in the format: 1 aaa.c 10/10/2010 1 bbb.h 1/1/2011 3 ccc.c 2/2/2012 1 ccc.c…
Jonas Tadeu
  • 23
  • 2
2
votes
1 answer

How to force uniq to distinquish between em- and en-dashes?

uniq (GNU coreutils 8.5) does not seem to distinguish between em- and en-dashes: $ echo -e "a–b\na—b" | uniq -c 2 a–b Is there any way to force this distinction? I've tried various settings for LC_COLLATE with no luck.
user918938
2
votes
3 answers

explain one expression of Ruby Array uniq method

c = %w(a b c d) 1.8.7 :025 > c.uniq {|x|x[/^a/]} => ["a", "b"] 1.8.7 :026 > c.uniq {|x|x[/^b/]} => ["a", "b"] 1.8.7 :027 > c.uniq {|x|x[/^c/]} => ["a", "c"] 1.8.7 :029 > c.uniq {|x|x =~ [/^c/]} => ["a"] 1.8.7 :030 > c.uniq {|s|s[/[^abc]/]} =>…
Race
  • 923
  • 10
  • 14
2
votes
1 answer

How can I use grep with pipe to sort uniq lines from a gff file

I am taking a fourth year bioinformatics course. In this current assignment, the prof has given us a gff file with all the miRNA genes in the human genome annotated as gene-MIR. We are supposed to use grep, along with a regular expression and other…
bioinformaticsnewbe
  • 23
  • 2
2
votes
5 answers

Word count and it output

I have the following lines: 123;123;#rss 123;123;#site #design #rss 123;123;#rss 123;123;#rss 123;123;#site #design and need to count how many times each tag appears, do the following: grep -Eo '#[a-z].*' ./1.txt | tr "\ " "\n" | uniq -c i.e. first…
KarlsD
  • 649
  • 1
  • 6
  • 12
2
votes
1 answer

uniq -u returns nothing

I ve been trying to use uniq -u to print the unique lines of a file with a single column uniq -u samples3 but I get nothing! No output! The original file is pretty big (ca. 40mil lines) but still it makes no sense that i get no output... I have…
Panos
  • 179
  • 2
  • 13
2
votes
3 answers

Remove duplicate lines repeating immediately ignoring the first field

I try to move from (stars are just here to indicate which lines I'm keeping) *2020-12-15 19:10:00;34min 8s (min: 32s; max 34min 8s);normal;4;2;91EECB0E;1us (-24);78.688ms (max: 5s);-50.923ms;103.234ms;25.637ms;70;-5,050ppm 2020-12-15 19:20:00;34min…
Maxou
  • 59
  • 7
2
votes
4 answers

Ruby, get uniq slice from combination

What I have a = [1,2,3,4] => [1, 2, 3, 4] b = a.combination(2).to_a => [[1, 2], [1, 3], [1, 4], [2, 3], [2, 4], [3, 4]] b.each_slice(2).to_a => [[[1, 2], [1, 3]], [[1, 4], [2, 3]], [[2, 4], [3, 4]]] What I'm trying to achieve is a unique…
highlight
  • 31
  • 3
2
votes
3 answers

Change the order of columns in the output of `uniq -c`

i am using this script that read the file and show me the unique Name from the file and also show me how many times this Name exist(number of count) in the…
Kashif
  • 27
  • 6
2
votes
5 answers

Counting the number of unique values based on two columns in bash

I have a tab-separated file looking like this: A 1234 A 123245 A 4546 A 1234 B 24234 B 4545 C 1234 C 1234 Output: A 3 B 2 C 1 Basically I need counts of unique values that belong to the first column, all in one commando with pipelines. As you may…
ta4le
  • 93
  • 1
  • 7
2
votes
0 answers

generate Uniq Transaction Id only Number in Laravel6+

I am posting this question here because i could't find the right solution: I have to make a uniq transaction id which i am generating with the time() + rand() In this function still collision can happen.. My question is is there any better way to…
Abir Adak
  • 21
  • 3
2
votes
3 answers

How to obtain only repeated lines for a specific column in bash

Imagine I have this file in bash: 1 3 6 name1 1 2 7 name2 3 4 2 name1 2 2 2 name3 7 8 2 name2 1 2 9 name4 How could I extract just those lines which present the field "name" repeated and sort them? My expected output would be: 1 3 6 name1 3 4 2…
Jeni
  • 918
  • 7
  • 19
2
votes
1 answer

How do you count consecutive strings in file using powershell?

So I want to know how I could get content from a file and count the consecutive occurrences of a string within that file? So my file has the following strings: 1 1 1 0 0 0 0 1 1 1 0 1 1 0 0 0 1 0 1 1 1 0 0 Now the thing is I know next to nothing…
Lha
  • 33
  • 3
1 2 3
…
30 31