Sublime duplicate removal (Array)

Question

Ran into an issue that I couldnt solve. Say I have a text file with thousands of entries such as:

12.04.2013 krispy
11.2.2013 krispy
11.2.2013 peter
11.2.2013 william
23.4.2014 krispy

How can I select and permute unique so that only 1 date for krispy is selected (doesnt matter which), so that the output is:

12.04.2013 krispy
11.2.2013 peter
11.2.2013 william

Meaning somehow I'm selecting the 2nd value after the " " space character and permuting it to remove the entire line.

Any help would be great, thanks!

Sublime Text 2 on windows, but if there is a solution on linux then I'd be glad to hear it — , Aug 16 '14 at 09:25
Because on bash I would have issued `sort -k2 -u test -o test`, could you use python? — enrico.bacis, Aug 16 '14 at 09:27
@enrico.bacis why don't you make that an answer? Maybe include a line about installing gitbash for Windows to get command-line utilities like `sort`... — MattDMo, Aug 16 '14 at 19:00
Actually because it was requested for sublimetext, but since the OP agreed on checking also other solutions, let's do it. — enrico.bacis, Aug 16 '14 at 19:01

score 0 · Accepted Answer · edited Jun 20 '20 at 09:12

This is (normally) not the job of an editor, you should do it in a programming language, but since you agreed to check also other solutions, let's go for it.

bash

Just use the sort method:

sort -k2 -u filename -o filename

This will sort lines based on the second column (-k2) and return only the ones that are unique for that column (-u). You read the file filename and output on the file filename (to overwrite it).

If you are on a non-unix system you could use git-bash or cygwin to use unix commands.

python

Otherwise you could use the omnipresent python to accomplish this. Actually sublime text is written in python, so it's trivial to turn this code in a plugin for sublime.

removedups.py

from fileinput import input
import sys

seen = set()
filename = sys.argv[1]

for line in input(filename, inplace=True):
    date, name = line.split()
    if name not in seen:
        seen.add(name)
        print line,

Then you can use it like this:

python removedups.py filename

Ill just perform it on one of my Debian machines. Thanks for the sort modifiers to use, silly that I didn't Google that but I had become dependent on sublime there for a while. +1. — , Aug 17 '14 at 07:22

Sublime duplicate removal (Array)

1 Answers1

bash

python

removedups.py