Resample timeseries with panda

Question

i have to downsample the following time series with panda:

Time                            ch1     ,ch2   ,ch3     ,ch4
09/19/2019 22:00:00.000000000  ,0       ,0     ,8675601 ,0
09/19/2019 22:00:00.000976562  ,8028645 ,0     ,8662525 ,7706467
09/19/2019 22:00:00.001953124  ,8027705 ,0     ,0       ,7704373
09/19/2019 22:00:00.001953125  ,0       ,0     ,8685515 ,0
09/19/2019 22:00:00.002929687  ,8028089 ,0     ,8699659 ,7704202
09/19/2019 22:00:00.003906249  ,8027918 ,0     ,0       ,7705569
09/19/2019 22:00:00.003906250  ,0       ,0     ,8703334 ,0

Since the csv file is big i use the following code to read each chunk, and then try to downsaple it:

def resampleSignal(inPath,separator,firstDataLine,chunkSize):
    columsDataFrame=[]
    tempIndex=0
    for chunk in tqdm(pd.read_csv(inPath,skiprows=range(0,firstDataLine),chunksize=chunkSize,sep=separator)):
        columsDataFrame=chunk.columns
        chunk.index = pd.to_datetime(chunk.index, unit='ns')
        resampled = pd.DataFrame()
        resampled=chunk.resample('1S').last()
        resampled_np=(resampled).values
        if tempIndex==0:
            finalDataSet=np.array(resampled_np)
        else:
            finalDataSet=np.append(finalDataSet,np.array(resampled_np),axis=0)
        tempIndex+=1
    return finalDataSet

The problem is that no matter if i change the parameter '1S' the output will always be:

Time                          ,ch1       ,ch2       ,ch3       ,ch4
09/19/2019 12:03:21.906250000 ,8471473.0 ,5633804.0 ,8578007.0  ,7515027.0
09/19/2019 12:16:20.657226562 ,8463397.0 ,5616594.0 ,8582878.0  ,7536395.0
09/19/2019 12:28:45.581054687 ,7711094.0 ,0.0       ,16777215.0 ,7773021.0
09/19/2019 12:41:04.551757812 ,7690984.0 ,5697459.0 ,16777215.0 ,7795462.0

Basically it always get the last row of the chunk instead of a row each second. My current verion of panda is 0.25.3 and if i print the resampler object chunk.resample('1S') i get the following output:

DatetimeIndexResampler [freq=<Second>, axis=0, closed=left, label=left, convention=start, base=0]

so i know that it's using the right axis. What am i doing wrong? Thanks in advance for the help!!

What does it mean to downsample? Do you want to get N random lines out of your files? — alec_djinn, Feb 03 '20 at 13:29

score 1 · Accepted Answer · answered Feb 03 '20 at 13:13

Ok i realized my mistake, i was trying to time sample on the wrong chunk.index i modified the code as follow:

def resampleSignal(inPath,separator,firstDataLine,chunkSize):
    columsDataFrame=[]
    tempIndex=0
    for chunk in tqdm(pd.read_csv(inPath,skiprows=range(0,firstDataLine),chunksize=chunkSize,sep=separator)):
        columsDataFrame=chunk.columns
        chunk.Time = pd.Index(pd.to_datetime(chunk.Time, unit='ns'))
        resampled = pd.DataFrame()
        resampled=chunk.resample('100L', on='Time').last()
        resampled_np=(resampled).values
        if tempIndex==0:
            finalDataSet=np.array(resampled_np)
        else:
            finalDataSet=np.append(finalDataSet,np.array(resampled_np),axis=0)
        tempIndex+=1
    return finalDataSet

Resample timeseries with panda

1 Answers1