I have a dictionary like this below
dictionary_Tag = {'A':'unitA&',
'B':'B&',
'C':'unitC',
'D':'D#' }
and a table like this
|item_name|item_value|timestamp |idx|
+---------+----------+----------------------------+---+
|A |0.25 |2023-03-01T17:20:00.000+0000|0 |
|B |0.34 |2023-03-01T17:20:00.000+0000|0 |
|A |0.3 |2023-03-01T17:25:00.000+0000|1 |
|B |0.54 |2023-03-01T17:25:00.000+0000|1 |
|A |0.3 |2023-03-01T17:30:00.000+0000|2 |
|B |0.54 |2023-03-01T17:30:00.000+0000|2 |
I want to add one more column named "description" to the table like this
|item_name|item_value|timestamp |idx|description|
+---------+----------+----------------------------+---+-----------+
|A |0.25 |2023-03-01T17:20:00.000+0000|0 |unitA& |
|B |0.34 |2023-03-01T17:20:00.000+0000|0 |B& |
|A |0.3 |2023-03-01T17:25:00.000+0000|1 |unitA& |
|B |0.54 |2023-03-01T17:25:00.000+0000|1 |B& |
|A |0.3 |2023-03-01T17:30:00.000+0000|2 |unitA& |
|B |0.54 |2023-03-01T17:30:00.000+0000|2 |B& |
How can I do this using Pyspark? would appreciate any help