I have a pyspark dataframe with a column of MapType(StringType(), FloatType()) and I would get a list of all keys appearing in the column. For example having this dataframe:
+---+--------------------+
| ID| map|
+---+--------------------+
| 0|[a -> 3.0, b -> 2...|
| 1|[a -> 1.0, b -> 4...|
| 2|[a -> 6.0, c -> 5...|
| 3|[a -> 6.0, f -> 8...|
| 4|[a -> 2.0, c -> 1...|
| 5|[c -> 1.0, d -> 1...|
| 6|[a -> 4.0, c -> 1...|
| 7|[a -> 2.0, d -> 1...|
| 8| [a -> 2.0]|
| 9|[e -> 1.0, f -> 1.0]|
| 10| [g -> 1.0]|
| 11|[e -> 2.0, b -> 3.0]|
+---+--------------------+
I am expecting to get the following list:
['a', 'b', 'c', 'd', 'e', 'f', 'g']
I already tried
df.select(explode(col('map'))).groupby('key').count().select('key').collect()
df.select(explode(col('map'))).select('key').drop_duplicates().collect()
df.select(explode(col('map'))).select('key').distinct().collect()
df.select(explode(map_keys(col('map')))).select('key').distinct().collect()
...
But for each of these commands I get differing results, not only across the different commands, but also wenn I execute the exact same command on the same dataframe.
For example:
keys_1 = df.select(explode(col('map'))).select('key').drop_duplicates().collect()
keys_1 = [row['key'] for row in keys_1]
And:
keys_2 = df.select(explode(col('map'))).select('key').drop_duplicates().collect()
keys_2 = [row['key'] for row in keys_2]
Then it happens quite often that len(keys_1) != len(keys_2).
My dataframe has around 10e7 rows and there are around 2000 different keys for my map column
NOTE that on the small example dataset it works without a problem, but unfortunatly it is quite hard to find a large example dataset.
Small Dataset example code:
df = spark.createDataFrame([
(0, {'a':3.0, 'b':2.0, 'c':2.0}),
(1, {'a':1.0, 'b':4.0, 'd':6.0}),
(2, {'a':6.0, 'e':5.0, 'c':5.0}),
(3, {'f':8.0, 'a':6.0, 'g':4.0}),
(4, {'a':2.0, 'c':1.0, 'd':3.0}),
(5, {'d':1.0, 'g':5.0, 'c':1.0}),
(6, {'a':4.0, 'c':1.0, 'f':1.0}),
(7, {'a':2.0, 'e':2.0, 'd':1.0}),
(8, {'a':2.0}),
(9, {'e':1.0, 'f':1.0}),
(10, {'g':1.0}),
(11, {'b':3.0, 'e':2.0})
],
['ID', 'map']
)
df.select(explode(col('map'))).groupby('key').count().select('key').collect()
