What did I do wrong? Not sorting properly with awk

Question

Hi so basically I have a 'temp' text file that I'm using that has a long list of various email addresses (some repeats). What I'm trying to output is the email addresses in order of highest frequency and then the total number of unique email addresses at the end.

awk '{printf "%s %s\n", $2, $1} END {print "total "NR}' temp | sort -n | uniq -c -i

So far I got the output I wanted except for the fact that it's not ordered in terms of highest frequency. Instead, it's in alphabetical order.

I've been stuck on this for a few hours now and have no idea why. I know I probably did something wrong but I'm not sure. Please let me know if you need more information and if the code I provided was not the problem. Thank you in advance.

edit: I've also tried doing sort -nk1 (output has frequency in first column) and even -nk2

edit2: Here is a sample of my 'temp' file

aol.com
netscape.net
yahoo.com
yahoo.com
adelphia.net
twcny.rr.com
charter.net
yahoo.com

edit 3:

expected output:

33 aol.com
24 netscape.net
18 yahoo.com
5 adelphia.net
4 twcny.rr.com
3 charter.net
total 6

(no repeat emails, 6 total unique email addresses)

Answer 1

Sample input modified to include an email with two instances

$ cat ip.txt 
aol.com
netscape.net
yahoo.com
yahoo.com
adelphia.net
twcny.rr.com
netscape.net
charter.net
yahoo.com

Using perl

$ perl -lne '
$c++ if !$h{$_}++;
END
{
    @k = sort { $h{$b} <=> $h{$a} } keys %h;
    print "$h{$_} $_" foreach (@k);
    print "total ", $c;
}' ip.txt
3 yahoo.com
2 netscape.net
1 adelphia.net
1 charter.net
1 aol.com
1 twcny.rr.com
total 6

• $c++ if !$h{$_}++ increment counter for unique input lines, increment hash value with input line as key. Default initial value is 0 for both
• After processing all input lines:
  • @k = sort { $h{$b} <=> $h{$a} } keys %h get keys sorted by descending numeric values of hash
  • print "$h{$_} $_" foreach (@k) print each hash value and key based on sorted keys @k
  • print "total ", $c print total unique lines

Can be written in single line if preferred:

perl -lne '$c++ if !$h{$_}++; END{@k = sort { $h{$b} <=> $h{$a} } keys %h; print "$h{$_} $_" foreach (@k); print "total ", $c}' ip.txt

Reference: How to sort perl hash on values and order the keys correspondingly

Answer 2

In Gnu awk using @Sundeep's data:

$ cat program.awk
{ a[$0]++ }                                # count domains 
END {
    PROCINFO["sorted_in"]="@val_num_desc"  # sort in desc order in for loop
    for(i in a) {                          # this for in desc order
        print a[i], i 
        j++                                # count total
    } 
    print "total", j
}

Run it:

$ awk -f program.awk ip.txt
3 yahoo.com
2 netscape.net
1 twcny.rr.com
1 aol.com
1 adelphia.net
1 charter.net
total 6

Answer 3

Updated / Summary

Summarising a few tested approaches here for this handy sorting tool:

Using bash (In my case v4.3.46)

sortedfile="$(sort temp)" ; countedfile="$(uniq -c <<< "$sortedfile")" ; uniquefile="$(sort -rn <<< "$countedfile")" ; totalunique="$(wc -l <<< "$uniquefile")" ; echo -e "$uniquefile\nTotal: $totalunique"

Using sh/ash/busybox (Though they aren't all the same, they all worked the same for these tests)

time (sort temp > /tmp/sortedfile ; uniq -c /tmp/sortedfile > /tmp/countedfile ; sort -rn /tmp/countedfile > /tmp/uniquefile ; totalunique="$(cat /tmp/uniquefile | wc -l)" ; cat /tmp/uniquefile ; echo "Total: $totalunique")

Using perl (see this answer https://stackoverflow.com/a/40145395/3544399)

perl -lne '$c++ if !$h{$_}++; END{@k = sort { $h{$b} <=> $h{$a} } keys %h; print "$h{$_} $_" foreach (@k); print "Total: ", $c}' temp

---

What was tested

A file temp was created using a random generator:

• @domain.com was different in the unique addresses
• Duplicated addresses were scattered
• File had 55304 total addresses
• File has 17012 duplicate addresses

A small sample of the file looks like this:

24187@9674.com
29397@13000.com
18398@27118.com
23889@7053.com
24501@7413.com
9102@4788.com
16218@20729.com
991@21800.com
4718@19033.com
22504@28021.com

Performance:

For the sake of completeness it's worth mentioning the performance;

perl:               sh:                 bash:

Total: 17012        Total:    17012     Total:    17012

real    0m0.119s    real    0m0.838s    real    0m0.973s
user    0m0.061s    user    0m0.772s    user    0m0.894s
sys     0m0.027s    sys     0m0.025s    sys     0m0.056s

---

Original Answer (Counted total addresses and not unique addresses):

tcount="$(cat temp | wc -l)" ; sort temp | uniq -c -i | sort -rn ; echo "Total: $tcount"

• tcount="$(cat temp | wc -l)": Make Variable with line count
• sort temp: Group email addresses ready for uniq
• uniq -c -i: Count occurrences allowing for case variation
• sort -rn: Sort according to numerical occurrences and reverse the order (highest on top)
• echo "Total: $tcount": Show the total addresses at the bottom

Sample temp file:

john@domain.com
john@domain.com
donald@domain.com
john@domain.com
sam@domain.com
sam@domain.com
bill@domain.com
john@domain.com
larry@domain.com
sam@domain.com
larry@domain.com
larry@domain.com
john@domain.com

Sample Output:

   5 john@domain.com
   3 sam@domain.com
   3 larry@domain.com
   1 donald@domain.com
   1 bill@domain.com
Total:       13

Edit: See comments below regarding use of sort