This was not an issue prior to Pandas 1.5.x.
Last version working fast as expected: 1.4.4.
The below tests demonstrate a conversion of a dict to a dataframe (df = pd.DataFrame(dict)
), however, the issue exists also when performing a simple value retrieval via print(dict.values())
.
The dict used for testing is a simple one where the keys are integers 0-999, and the values are pandas serieses, where each series' index is a DatetimeIndex with a 'US/Eastern' TZ and each series' values are all zeros.
The cause of the slowness is the TZ.
Tested on both Python 3.10.x and 3.11.x.
Here are the test results:
import pandas as pd
from timeit import default_timer as timer
dict = {}
index = pd.date_range(start='2020-01-01', end='2020-12-31', freq='D')
for i in range(1000):
dict[i] = pd.Series(0, index=index)
start = timer()
df = pd.DataFrame(dict)
print(f"Pandas > 1.4.4, no TZ: {timer() - start} seconds.")
Pandas > 1.4.4, no TZ: 0.014551600441336632 seconds.
import pandas as pd
from timeit import default_timer as timer
dict = {}
index = pd.date_range(start='2020-01-01', end='2020-12-31', freq='D', tz='US/Eastern')
for i in range(1000):
dict[i] = pd.Series(0, index=index)
start = timer()
df = pd.DataFrame(dict)
print(f"Pandas > 1.4.4, with TZ: {timer() - start} seconds.")
Pandas > 1.4.4, with TZ: 28.93334749992937 seconds.
import pandas as pd
from timeit import default_timer as timer
dict = {}
index = pd.date_range(start='2020-01-01', end='2020-12-31', freq='D')
for i in range(1000):
dict[i] = pd.Series(0, index=index)
start = timer()
df = pd.DataFrame(dict)
print(f"Pandas 1.4.4, no TZ: {timer() - start} seconds.")
Pandas 1.4.4, no TZ: 0.023447500076144934 seconds.
import pandas as pd
from timeit import default_timer as timer
dict = {}
index = pd.date_range(start='2020-01-01', end='2020-12-31', freq='D', tz='US/Eastern')
for i in range(1000):
dict[i] = pd.Series(0, index=index)
start = timer()
df = pd.DataFrame(dict)
print(f"Pandas 1.4.4, with TZ: {timer() - start} seconds.")
Pandas 1.4.4, with TZ: 0.06350550008937716 seconds.