3

I am currently trying to model a Multivariate Random Forest on time series data. The only way I get decent test accuracies on the model is to detrend the data before modelling using the scipy detrend type=constant (the type=linear does not give good accuracies) with the following code:

detrended =signal.detrend(feature, axis=-1, type='constant', bp=0, overwrite_data=True)

After I model the data I want to show the results of the model as the original units before detrending.

I have two questions:

  1. Is there a way to reverse signal.detrend?
  2. What exactly does the signal.detrend (type=constant) do to the data?

In the Scipy documentation I am aware of the following:

type{‘linear’, ‘constant’}, optional The type of detrending. If type == 'linear' (default), the result of a linear least-squares fit to data is subtracted from data. If type == 'constant', only the mean of data is subtracted.

However when I look at the data after performing signal.detrend (type=constat) it does not simply subtract the mean from the data as the documentation suggests.

Any guidance on how to reverse the detrend function and what the detrend type=constant does to the data would be greatly appreciated.

HoldOffHunger
  • 18,769
  • 10
  • 104
  • 133
Kirsty
  • 33
  • 3

1 Answers1

3

I had a similar question and figured I'd look in the source files. For your case it seems it literally substracts the np.mean() of the dataset:

    if type in ['constant', 'c']:
        ret = data - np.expand_dims(np.mean(data, axis), axis)
        return ret

Further in the source code there seems to be no difference whether you do overwrite_data=True/False. It runs the code above without checking this attribute, for as far as I can see.

Also, if I manually substract the mean vs. signal.detrend with type='constant' I see no differences in my data. Therefore I think it could be something you're doing.

enter image description here

If the above answers your question, obviously to reverse the detrending adding the original mean should do the job.

Daan Koetsenruijter
  • 86
  • 5