How can I solve this MemoryError?

Question

I'm trying to download and store in a csv file a big object using the Traffic library

from traffic.data import opensky, airports
low_memory=False
airport = airports['LEMD']
flight = opensky.history(start="2021-06-01 00:00",stop="2021-06-16 00:00",arrival_airport=airport.icao)
flight.to_csv(r"C:\Users\peter\Desktop\cosicas.csv")

But short after it begins running, this comes up:

DtypeWarning: Columns (18) have mixed types.Specify dtype option on import or set low_memory=False.
  flight = opensky.history(start="2021-05-15 00:00",stop="2021-06-16 00:00",arrival_airport=airport.icao)

And finally this:

MemoryError: Unable to allocate 2.54 GiB for an array with shape (7, 48728454) and data type float64

I have read a lot and this answer seemed to be the best solution but I do not know how to assign the dtype before it downloads (?).

Have you tried fetching your data in smaller chunks? [The documentation](https://traffic-viz.github.io/opensky_impala.html?highlight=history#traffic.data.adsb.opensky_impala.Impala.history) suggests fetching one day at a time. Pandas uses the most amount of memory when it is reading in the data. — Nick ODell, Jun 17 '21 at 18:44

score 0 · Answer 1 · answered Jun 18 '21 at 12:02

As commented by Nick ODell, it is better to do a smaller search, or even a loop of small searchs like this one:

from traffic.data import opensky, airports
import pickle, pandas
airport = airports['LEMD']

fechas = pandas.date_range(start="2021-05-30",end="2021-06-14")

VUELO = opensky.history(start=fechas[0],stop=fechas[1],arrival_airport=airport.icao)

for x in range(1,len(fechas)-1):
    
    vuelo = opensky.history(start=fechas[x],stop=fechas[x+1],arrival_airport=airport.icao)

    VUELO = vuelo + VUELO
    del vuelo
    
file_to_store = open(r"C:\Users\Shadow\Desktop\stored_object2.pickle", "wb")
pickle.dump(VUELO, file_to_store)
file_to_store.close()

The pickle thing is for later using the dowload objects to generate an altair figure.

How can I solve this MemoryError?

1 Answers1