fuzzywuzzy returning single characters, not strings

Question

I'm not sure where I'm going wrong here and why my data is returning wrong. Writing this code to use fuzzywuzzy to clean bad input road names against a list of correct names, replacing the incorrect with the closest match.

It's returning all lines of data2 back. I'm looking for it to return the same, or replaced lines of data1 back to me.

My Minimal, Reproducible Example:

import pandas as pd
import os
import csv
import usaddress
from fuzzywuzzy import fuzz
from fuzzywuzzy import process

data1 =('3176 DETRIT ROAD')
data2 =('DETROIT RD')

try:
    data1 = usaddress.tag(data1)
except usaddress.RepeatedLabelError:
    pass

roaddnum2 = data1[0].get('AddressNumber', '')
roadir2 = data1[0].get('StreetNamePreDirectional', '')
roadname2 = data1[0].get('StreetName', '')
roaddsg2 = data1[0].get('StreetNamePostType', '')

street2 = (roadir2, roadname2, roaddsg2)
street2 = " ".join(street2)
street2 = street2.strip()
data2 = data2.split(',')

if street2 not in data2:
    street2 = process.extract(street2, data2)
    print(street2[0])

My full code

import pandas as pd
import os
import csv
import usaddress
from fuzzywuzzy import fuzz
from fuzzywuzzy import process

def convert_tolist(string):
    li = list(string.split(" "))
    return li

with open(r"Cass_Howard - Copy.csv") as csv_file,\
        open("Final_Test_Clean.csv", "w") as f,\
        open(r"TEST_no_dups12.csv") as ul:
    csv_reader = csv.reader(csv_file, delimiter=',')
    next(csv_reader)
    csv_reader = csv.reader(f, delimiter=',')
    file_1 = csv_file
    file_2 = ul

    for data1, data2 in zip(file_1, file_2):
        data1 = data1.split(',')
        data1 = data1[18]
        data1 = data1.upper()
        data2 = data2.strip()
        data2 = data2.split(',')
        data2 = ''.join(data2)
        try:
            data1 = usaddress.tag(data1)
        except usaddress.RepeatedLabelError:
            pass

        roaddnum2 = data1[0].get('AddressNumber', '')
        roadir2 = data1[0].get('StreetNamePreDirectional', '')
        roadname2 = data1[0].get('StreetName', '')
        roaddsg2 = data1[0].get('StreetNamePostType', '')

        street2 = (roadir2, roadname2, roaddsg2)
        street2 = " ".join(street2)
        street2 = street2.strip()
        data1 = list(data1)
        convert_tolist(data2)


        if street2 not in data2:
            street2 = process.extract(street2, data2)
            print(street2)

street2 query data(around 950 lines)

DETROIT ROAD
DETROIT ROAD
MANNIX ST
MANNIX ST

data2 choices data(around 200 lines)

ACRES
ADERSON RD
AIRPORT RD
ALGONQUIN

You need to create a [Minimal, Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example). Your code doesn't seem to be actually using `fuzz`. Also, what is the purpose of the `convert_tolist` function? It is longer to call that than just `.split(" ")`. — theherk, Jan 25 '22 at 15:14
Thanks for the info! I added the Minimal, Reproducible Example to the top of the post and will take your other comments into account when revising. — Michael Winston, Jan 25 '22 at 15:41
Some more suggestions: `fuzz` seems unused. Overwriting `data1` only when `usaddress.tag()` doesn't fail is questionable. What is the outcome of this? What are the results in `roaddnum2`, `roadir2` etc? Can you hardcode those instead of using `usaddress.tag()`? What happens in the last `if ...` clause, which path does it take? — Ulrich Eckhardt, Jan 25 '22 at 15:57
this is the outcome from `usaddress.tag()` `(OrderedDict([('AddressNumber', '3176'), ('StreetName', 'DETROIT'), ('StreetNamePostType', 'ROAD')]), 'Street Address')` and `roadnum2'` would call `3176` and `roaddir2` would call a cardinal direction if there was one and so on. I can't hardcode these because some addresses have a cardinal direction and some don't, some have DR left off the end and some don't. `usadress.tag()` is smart enough to parse the address into the proper category and allows me to work with nasty user input address easier. — Michael Winston, Jan 25 '22 at 16:15
Okay, I'm halfway there. fuzzywuzzy requires you to pass an iterable so I had to add `data2 = data2.split(',')` to get the full strings to return. But now it's clear that I'm returning the exact lines from `data2`. I'll update my Minimal, reproduceable example to reflect. — Michael Winston, Jan 25 '22 at 17:08

theherk · Answer 1 · 2022-01-25T18:21:49.283

Okay, I'm not certain I've fully understood your issue, but modifying your reprex, I have produced the following solution.

import usaddress
from fuzzywuzzy import process

data1 = "3176 DETRIT ROAD"
choices = ["DETROIT RD"]

try:
    data1 = usaddress.tag(data1)
except usaddress.RepeatedLabelError:
    pass

parts = [
    data1[0].get("StreetNamePreDirectional"),
    data1[0].get("StreetName"),
    data1[0].get("StreetNamePostType"),
]

street = " ".join([x for x in parts if x])

if street not in choices:
    street = " ".join(
        [data1[0].get("AddressNumber"), process.extract(street, choices)[0][0]]
    )

print(street)

This yields:

3176 DETROIT RD

So basically, it has replaces the street name bits with the best match from the choices list using the process.extract() function.

And here it is as a callable function:

import usaddress
from fuzzywuzzy import process


def best_street(addr: str, choices: list[str]) -> str:
    try:
        usaddr = usaddress.tag(addr)
    except usaddress.RepeatedLabelError:
        pass
    street_parts = [
        usaddr[0].get("StreetNamePreDirectional"),
        usaddr[0].get("StreetName"),
        usaddr[0].get("StreetNamePostType"),
    ]
    street = " ".join([x for x in street_parts if x])
    return " ".join(
        [usaddr[0].get("AddressNumber"), process.extract(street, choices)[0][0]]
    )


if __name__ == "__main__":
    choices = ["AIRPORT RD", "DETROIT RD"]
    print(best_street("123 Detrt", choices))
    print(best_street("9876 AIRPUMP DR", choices))

yielding:

123 DETROIT RD
9876 AIRPORT RD

Michael Winston · Accepted Answer · 2022-01-28T02:42:11.487

My issue here was fuzzywuzzy requires you to pass an iterable so I had to add data2 = data2.split(',') to get the full strings to return.

My other issue was thinking I needed to use zip() to compare my files when zip() is for comparing parallel and not cross products.

Here is what I came up with that works. There is no issue with the code but fuzzywuzzy is not accurate enough for this tool to be practical to clean my data with the amount of typos in my address data.

If you can think of a way for me to clean up the if/else statement i'd be willing to hear it.

import os
import csv
import shutil
import usaddress
import pandas as pd
from fuzzywuzzy import process

with open(r"TEST_Cass_Howard.csv") as csv_file, \
        open(".\Scratch\Final_Test_Clean.csv", "w") as f, \
        open(r"TEST_Unique_List.csv") as ul:
    csv_reader = csv.reader(csv_file, delimiter=',')
    next(csv_reader)
    csv_reader1 = csv.reader(f, delimiter=',')
    correct = list(ul)
    for line in csv_reader:
        line = line[18]
        line = line.upper()
        if line == '' or line == ' ':
            line = ''
        else:
            try:
                addressbrk = usaddress.tag(line)
            except usaddress.RepeatedLabelError:
                addressbrk = line
            line = addressbrk  # alt output: ('Please fix the incorect format of: %s,' % addressbrk)
        if line != '':
            roadnum2 = line[0].get('AddressNumber', '')
            roadir2 = line[0].get('StreetNamePreDirectional', '')
            roadname2 = line[0].get('StreetName', '')
            roaddsg2 = line[0].get('StreetNamePostType', '')
        else:
            line = ''
        if line != '':
            street2 = (roadir2, roadname2, roaddsg2)
            street2 = " ".join(street2)
            street2 = street2.strip()
        else:
            street2 = ''
        if street2 != '':
            if street2 not in correct:
                street2 = process.extractOne(street2, correct)
            else:
                street2 = '\n'
        if street2 != '':
            tgthr = (roadnum2, street2[0])
        else:
            tgthr = ''
        if tgthr != '':
            final = (' '.join(tgthr))
        else:
            final = ('Null' + '\n')
        f.writelines(final)
original = r"TEST_Cass_Howard.csv" 
target = (r'.\Scratch\Cass_Howard_Clean.csv')
shutil.copyfile(original, target)
df1 = pd.read_csv('.\Scratch\Final_Test_Clean.csv', header=None)
df1.columns = ["WELL_ADDR_CLN"]
df = pd.read_csv('.\Scratch\Cass_Howard_Clean.csv')
df = df.join(df1)
new_data = df['WELL_ADDR_CLN']
df = df.drop(columns=['WELL_ADDR_CLN'])
df.insert(loc=19, column='WELL_ADDR_CLN', value=new_data)
os.remove(".\Scratch\Cass_Howard_Clean.csv")
df.to_csv("Cass_Howard_Clean.csv", index=False)
os.remove('.\Scratch\Final_Test_Clean.csv')

fuzzywuzzy returning single characters, not strings

2 Answers2