How to apply frozenset on polars dataframe?

Question

I have a pandas dataframe as:

df_names = pd.DataFrame({'last_name':['Williams','Henry','XYX','Smith','David','Freeman','Walter','Test_A'],
                        'first_name':['Henry','Williams','ABC','David','Smith','Walter','Freeman','Test_B']})

Here i have applied a frozen set on last_name and first_name columns to see if the names are interchanged i.e williams henry, henry williams.

df_names[['last_name','first_name']].apply(frozenset,axis=1)

Here the same kind of implementation is required on polars dataframe. How to get it done ?

Could you tell me what you are trying to achieve on a higher level? Turning a column into python frozenset objects is almost always a bad idea and will be terribly slow. — ritchie46, Aug 18 '22 at 10:40
I would like to find a set of names which are interchanged i.e Joe Doe, Doe Joe — myamulla_ciencia, Aug 18 '22 at 10:42
@ritchie46 if you have any other idea on it please share it here. — myamulla_ciencia, Aug 18 '22 at 10:43
https://stackoverflow.com/questions/73079297/how-to-check-similarity-between-two-name-pairs-interchangeably-and-give-them-a-u — myamulla_ciencia, Aug 18 '22 at 10:43
Here i had asked about this requirement on SO earlier. it can be implemented in pandas. i wanted to have it in polars as well — myamulla_ciencia, Aug 18 '22 at 10:44

score 4 · Accepted Answer · answered Aug 18 '22 at 14:39

As @ritchie46 mentioned, you'll want to avoid embedding Python objects (like frozensets) into a Polars DataFrame. The performance is not good, and columns of type object have limited functionality.

Here's a Polars algorithm that will be very performant and will accomplish what you need.

(
    df_names
    .with_column(
        pl.concat_list([
            pl.col("first_name").str.replace_all(r'\s','').str.to_uppercase(),
            pl.col("last_name").str.replace_all(r'\s','').str.to_uppercase(),
        ])
        .arr.sort()
        .arr.join('|')
        .alias('name_key')
    )
    .filter(pl.count().over('name_key') > 1)
)

shape: (6, 3)
┌───────────┬────────────┬────────────────┐
│ last_name ┆ first_name ┆ name_key       │
│ ---       ┆ ---        ┆ ---            │
│ str       ┆ str        ┆ str            │
╞═══════════╪════════════╪════════════════╡
│ Williams  ┆ Henry      ┆ HENRY|WILLIAMS │
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Henry     ┆ Williams   ┆ HENRY|WILLIAMS │
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Smith     ┆ David      ┆ DAVID|SMITH    │
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ David     ┆ Smith      ┆ DAVID|SMITH    │
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Freeman   ┆ Walter     ┆ FREEMAN|WALTER │
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Walter    ┆ Freeman    ┆ FREEMAN|WALTER │
└───────────┴────────────┴────────────────┘

To help with matching, I've converted names to all uppercase and eliminated white space. (You can drop that if you don't find it useful.)

score 0 · Answer 2 · answered Aug 18 '22 at 06:45

0

dfl_names.apply (docs) will do. You'll need to wrap frozenset in tuple.

answered Aug 18 '22 at 06:45

user1635327

1,469
3
11

How to apply frozenset on polars dataframe?

2 Answers2

Linked