Remove words from list but keep the ones only made up from the list

Question

I have one dataframe containing strings and one list of words that I want to remove from the dataframe. However, I would like to also keep the strings from the df which are entirely made up of words from the list.

Here is an example:

strings_variable
Avalon Toyota loan
Blazer Chevrolet
Suzuki Vitara sales
Vauxhall Astra
Buick Special car
Ford Aerostar
car refund
car loan

data = {'strings_variable': ['Avalon Toyota loan', 'Blazer Chevrolet', 'Suzuki Vitara sales', 'Vauxhall Astra', 'Buick Special car', 'Ford Aerostar', 'car refund', 'car loan']}
df = pd.DataFrame(data)

words_to_remove = ('car','sales','loan','refund')

The final output should look like this:

strings_variable
Avalon Toyota
Blazer Chevrolet
Suzuki Vitara
Vauxhall Astra
Buick Special
Ford Aerostar
car refund
car loan

data= {'strings_variable': ['Avalon Toyota', 'Blazer Chevrolet', 'Suzuki Vitara', 'Vauxhall Astra', 'Buick Special', 'Ford Aerostar', 'car refund', 'car loan']}
df = pd.DataFrame(data)

Note, the words that I want to remove are in addition to the car names however I would like to keep the rows where the strings are only made of words in words_to_remove

Here is my code (Python) so far:

def remove_words(df):
   df = [word for words in df if word not in words_to_remove]
   return df

strings_variable = strings_variable.apply(remove_words)

I hope it makes sense - thank you in advance!

score 1 · Answer 1 · answered Mar 06 '23 at 14:40

You can use set differences to decide whether all the words are included or not in words_to_remove.

def remove_words(text, words=words_to_remove):
    if set(text.split()).difference(set(words)):
        for word in words:
            text = text.replace(word, '')
        text = text.strip()
    return text

df['strings_variable'].map(remove_words)

0       Avalon Toyota
1    Blazer Chevrolet
2       Suzuki Vitara
3      Vauxhall Astra
4       Buick Special
5       Ford Aerostar
6          car refund
7            car loan
Name: strings_variable, dtype: object

score 0 · Answer 2 · answered Mar 06 '23 at 14:54

You could create a temporary list of words and if it ends up empty, you just use the original word. Something like this:

df= {'strings_variable': ['Avalon Toyota loan', 'Blazer Chevrolet', 'Suzuki Vitara sales', 'Vauxhall Astra', 'Buick Special car', 'Ford Aerostar', 'car refund', 'car loan']}
words_to_remove = ('car','sales','loan','refund')

new_df = []
for word in df["strings_variable"]:
    temp = []
    for w in word.split():
        if w.lower() not in words_to_remove: temp.append(w)
    new_df.append(" ".join(temp) if temp else word)
print(new_df)

Result:

['Avalon Toyota', 'Blazer Chevrolet', 'Suzuki Vitara', 'Vauxhall Astra', 'Buick Special', 'Ford Aerostar', 'car refund', 'car loan']

I also considered not case sensitive, hence the lower() in the condition.

score 0 · Accepted Answer · answered Mar 06 '23 at 14:57

I'm assuming you're using pandas, because of your use of df and the .apply() method. However, you need to create the DataFrame itself. Then you can create a function to apply to the Series (if you're only changing the column) or to apply-map to the whole DataFrame (probably not what you're looking for).

import pandas as pd

df = pd.DataFrame({
    'strings_variable': [
        'Avalon Toyota loan',
        'Blazer Chevrolet',
        'Suzuki Vitara sales', 
        'Vauxhall Astra', 
        'Buick Special car', 
        'Ford Aerostar', 
        'car refund', 
        'car loan'
    ]
})

words_to_remove = ('car', 'sales', 'loan', 'refund')

def remove_words(text: str) -> str:
    """Remove stop words if string composed made entirely of them"""
    
    new_text = ' '.join([
        word
        for word in text.split()
        if word not in words_to_remove
    ])
    
    if not new_text:
        new_text = text
        
    return new_text

df['strings_variable'] = df['strings_variable'].apply(remove_words)
# or
df = df.applymap(remove_words) # probably not this one

Thank you, this does the job. Very close to Ignatius Reilly's answer however for his answer I find that some strings don't get removed properly. For example my real words_to_remove list has also words like sale, sales . Whenever using that code for 'Suzuki Vitara sales' for example it returns 'Suzuki Vitara s'. It's probably only a minor thing but pashri's answer works better for my problem. — RoyalPotatoe, Mar 06 '23 at 15:46

Remove words from list but keep the ones only made up from the list

3 Answers3