0

This was not an issue prior to Pandas 1.5.x.

Last version working fast as expected: 1.4.4.

The below tests demonstrate a conversion of a dict to a dataframe (df = pd.DataFrame(dict)), however, the issue exists also when performing a simple value retrieval via print(dict.values()).

The dict used for testing is a simple one where the keys are integers 0-999, and the values are pandas serieses, where each series' index is a DatetimeIndex with a 'US/Eastern' TZ and each series' values are all zeros.

The cause of the slowness is the TZ.

Tested on both Python 3.10.x and 3.11.x.

Here are the test results:

import pandas as pd
from timeit import default_timer as timer

dict = {}
index = pd.date_range(start='2020-01-01', end='2020-12-31', freq='D')

for i in range(1000):
    dict[i] = pd.Series(0, index=index)

start = timer()
df = pd.DataFrame(dict)
print(f"Pandas > 1.4.4, no TZ: {timer() - start} seconds.")

Pandas > 1.4.4, no TZ: 0.014551600441336632 seconds.

import pandas as pd
from timeit import default_timer as timer

dict = {}
index = pd.date_range(start='2020-01-01', end='2020-12-31', freq='D', tz='US/Eastern')

for i in range(1000):
    dict[i] = pd.Series(0, index=index)

start = timer()
df = pd.DataFrame(dict)
print(f"Pandas > 1.4.4, with TZ: {timer() - start} seconds.")

Pandas > 1.4.4, with TZ: 28.93334749992937 seconds.

import pandas as pd
from timeit import default_timer as timer

dict = {}
index = pd.date_range(start='2020-01-01', end='2020-12-31', freq='D')

for i in range(1000):
    dict[i] = pd.Series(0, index=index)

start = timer()
df = pd.DataFrame(dict)
print(f"Pandas 1.4.4, no TZ: {timer() - start} seconds.")

Pandas 1.4.4, no TZ: 0.023447500076144934 seconds.

import pandas as pd
from timeit import default_timer as timer

dict = {}
index = pd.date_range(start='2020-01-01', end='2020-12-31', freq='D', tz='US/Eastern')

for i in range(1000):
    dict[i] = pd.Series(0, index=index)

start = timer()
df = pd.DataFrame(dict)
print(f"Pandas 1.4.4, with TZ: {timer() - start} seconds.")

Pandas 1.4.4, with TZ: 0.06350550008937716 seconds.

Aerodyne
  • 11
  • 3

0 Answers0