Find maximally similar strings between two lists

Question

I have two lists. The first list contains values from the database; the second list contains a set of words from a file that came from the user.

list_with_name_in_BD = [
    'LOPEZ TRANSPORT', 
    'LOPEZ TRANSPORTATION', 
    'Lopez Transport', 
    'napora inc', 
    'Narain Transport Ltd'
]

The total number of items in this list is about 300.000

list_with_words_in_file = [
    'Description', 'llc', 'Load', 'Rate', 'Amount', '$1950.00', 
    'Please', 'BBE', 'note', 'Lopez transport', 'LLC'
]

The total number of elements in this list is about 1000

My task is to find the most similar (or completely identical) elements between two lists. That is, as you can see, there are no identical elements between the two lists (although they may be), but there are similar ones. So, at the output, I would like to get a list with the most similar (or identical elements) between the two lists. That is, the result should be as follows.

output = [
    'Lopez Transport', 
    'LOPEZ TRANSPORT', 
    'LOPEZ TRANSPORTATION'
]

I will be grateful for any help and advice you can give me.

Answer 1

rapidfuzz might be useful here. extract in particular can take a preprocessing function (in your case str.lower), a cutoff score and a scorer as arguments.

See if the following brings some improvement to nested loops:

from rapidfuzz import process

list_with_name_in_DB = ["LOPEZ TRANSPORT", "LOPEZ TRANSPORTATION", "Lopez Transport", "napora inc", "Narain Transport Ltd"]
list_with_words_in_file = ['Description', 'Load', 'Rate', 'Amount', '$1950.00', 'Please', 'note', 'Lopez transport']

res = []
for e in list_with_words_in_file:
    r = process.extract(e, list_with_name_in_DB, processor=str.lower, score_cutoff=80)
    res += r

print(res)

Output:

[('LOPEZ TRANSPORT', 100.0, 0), ('Lopez Transport', 100.0, 2), ('LOPEZ TRANSPORTATION', 85.71428571428572, 1)]

Answer 2

Here's a way to do the search with soundex. Note that I have included a simple soundex conversion here from Is there a soundex function for python?, but there are several Python modules that can do it (soundex, fuzzy, jellyfish).

This has only one loop; the other loop becomes the "in" operator searching for a match in the set.

list_with_name_in_BD = [
    'LOPEZ TRANSPORT', 
    'LOPEZ TRANSPORTATION', 
    'Lopez Transport', 
    'napora inc', 
    'Narain Transport Ltd'
]

list_with_words_in_file = [
    'Description', 'Load', 'Rate', 'Amount', '$1950.00', 
    'Please', 'note', 'Lopez transport'
]

import unicodedata
def soundex(s):

    if not s:
        return ""

    s = unicodedata.normalize("NFKD", s)
    s = s.upper()

    replacements = (
        ("BFPV", "1"),
        ("CGJKQSXZ", "2"),
        ("DT", "3"),
        ("L", "4"),
        ("MN", "5"),
        ("R", "6"),
    )
    result = [s[0]]
    count = 1

    # find would-be replacment for first character
    for lset, sub in replacements:
        if s[0] in lset:
            last = sub
            break
    else:
        last = None

    for letter in s[1:]:
        for lset, sub in replacements:
            if letter in lset:
                if sub != last:
                    result.append(sub)
                    count += 1
                last = sub
                break
        else:
            if letter != "H" and letter != "W":
                # leave last alone if middle letter is H or W
                last = None
        if count == 4:
            break

    result += "0" * (4 - count)
    return "".join(result)

def soundexphrase(s):
    words = s.split()
    return ''.join(soundex(w) for w in words)

# Encode the names to search for:


keywords = set(map(soundexphrase,list_with_name_in_BD))
print(keywords)

for possible in list_with_words_in_file:
    s = soundexphrase(possible)
    print(f"{possible} encoded as {s} found: {s in keywords}")

Output:

{'N650T652L300', 'L120T652', 'N160I520'}
Description encoded as D261 found: False
Load encoded as L300 found: False
Rate encoded as R300 found: False
Amount encoded as A553 found: False
$1950.00 encoded as $000 found: False
Please encoded as P420 found: False
note encoded as N300 found: False
Lopez transport encoded as L120T652 found: True