Concatenate different dataframes into one with for loop

Question

I want to concatenate different financial dataframes into one. So when doing this for 2-3 stocks manually I do:

#getting dataframe from .csv file
df1 = pd.read_csv('C:\\Users\\Enric\\Desktop\\python\\tickers\\MSFT.csv',index_col=None, header=0)
df2 = pd.read_csv('C:\\Users\\Enric\\Desktop\\python\\tickers\\MMM.csv',index_col=None, header=0)

#selecting time period
df1_ = df1[(df1['date'] > '2017-01-01') & (df1['date'] <= '2018-01-01')]
df2_= df2[(df2['date'] > '2017-01-01') & (df2['date'] <= '2018-01-01')]

#concatenate
list=[]
list.append(df1_)
list.append(df2_)
df_all=pd.concat(list) #this is the final dataframe (I skipped some column set code)

On the other hand I've got a list of each sector cointaining all its companies:

healthcare=[u'ABT', u'ABBV', u'AET', u'A', u'ALXN', u'ALGN', u'AGN', u'AMGN', u'ANTM', u'BAX', u'BDX', u'BIIB', u'BSX', u'BMY', u'CELG', u'CNC', u'CI', u'COO', u'CVS', u'DVA', u'XRAY', u'EW', u'EVHC', u'ESRX', u'GILD', u'HCA', u'HOLX', u'HUM', u'IDXX', u'ILMN', u'INCY', u'ISRG', u'JNJ', u'LH', u'LLY', u'MDT', u'MRK', u'MTD', u'MYL', u'PKI', u'PRGO', u'PFE', u'DGX', u'REGN', u'RMD', u'SYK', u'TMO', u'UNH', u'UHS', u'VAR', u'VRTX', u'WAT', u'ZBH', u'ZTS']

And I want to concatenate all pricing dataframes from tickers appear in this list:

for i in healthcare:
  df=pd.read_csv('C:\\Users\\Enric\\Desktop\\python\\tickers\\'+i+'.csv',index_col=None, header=0)
  df1_ = df[(df['date'] > '2017-01-01') & (df['date'] <= '2018-01-01')]
  list=[]
  list.append(df1_)

But that won't work since each loop I'm duplicating "df" inside the list, so when trying to use pd.concat(list) I get an error.

I'm new to python and this seems to be an easy mistake but I dont find how. I tought about changing df1_name for each company ticker inside the loop but Idon't know how.

Thanks

jezrael · Accepted Answer · 2018-03-04T17:21:53.980

Define list out of loop:

L = []
for i in healthcare:
  df=pd.read_csv('C:\\Users\\Enric\\Desktop\\python\\tickers\\'+i+'.csv',index_col=None)
  df1_ = df[(df['date'] > '2017-01-01') & (df['date'] <= '2018-01-01')]
  L.append(df1_)

Or use list comprehension:

p = 'C:\\Users\\Enric\\Desktop\\python\\tickers\\'
d1 = '2017-01-01'
d2 = '2018-01-01'
L = [pd.read_csv(p+i+'.csv',index_col=None).query("@d1 < date <= @d2") for i in healthcare]

df = pd.concat(L)

Concatenate different dataframes into one with for loop

1 Answers1