How to reverse geocode PySpark dataframe? I have VehicleID, eventdatetime, latitude, longitude, vehicle speed columns and over a million rows

Asked May 18 '23 at 08:18

Active May 18 '23 at 08:21

Viewed 114 times

I have VehicleID, eventdatetime, latitude, longitude, vehicle speed columns and over a million rows. I need to fetch city, state, district for the same.

here is the working python code snippet for the same using geopy and reverse-geooder library. I am not able to do the same in pyspark.

if __name__ == "__main__":
    # Coordinates tuple.Can contain more than one pair.
    coordinates = list(zip(pdVehicles['gpsLatitude'],pdVehicles['gpslongitude']))#generates pairs of latlong
    data = reverseGeocode(coordinates)
    pdVehicles['City_Town'] = [i['name']for i in data]
    pdVehicles['State'] = [i['admin1'] for i in data]
    pdVehicles['District'] = [i['admin2'] for i in data]

I tried using UDF but it didn't work.

edited May 18 '23 at 08:21

samkart

6,007
2
14
29

asked May 18 '23 at 08:18

Aayush Sinha

how did your udf look like? – samkart May 18 '23 at 08:28
https://github.com/aayushs7ha/Pyspark-Tutorial/blob/main/Pyspark_reverseGeocode.py – Aayush Sinha May 18 '23 at 08:39
UDFs return a single column, not multiple columns. so, try to return your values as an array or struct, and then create 3 columns using that result – samkart May 18 '23 at 10:04
Give us a sample input and output – arudsekaberne May 18 '23 at 16:41
https://stackoverflow.com/questions/59099762/python-reverse-geocoding-to-get-city-name-and-state-name-in-pandas -> I need to do the same with PySpark – Aayush Sinha May 22 '23 at 08:50

How to reverse geocode PySpark dataframe? I have VehicleID, eventdatetime, latitude, longitude, vehicle speed columns and over a million rows

0 Answers0