0

I try to use Kalman filtering for my one dimensional data. So, assume that I have the following dataset:

 Variable
 250.1
 248.5
 262.3
 265.3
 270.2

I do know that there is a noise in my data and hence, I want to clean this data by using Kalman filtering. Which way can produce the most efficient result for me?

I run the following code:

 from pykalman import KalmanFilter
 import numpy as np
 kf = KalmanFilter(transition_matrices = [[1, 1], [0, 1]], 
 observation_matrices = [[0.1, 0.5], [-0.3, 0.0]])
 measurements = np.asarray([(250.1),(248.5),(262.3),(265.3), (270.2)])
 kf = kf.em(measurements, n_iter=5)
 (filtered_state_means, filtered_state_covariances)=kf.filter(measurements)
 (smoothed_state_means, smoothed_state_covariances)=kf.smooth(measurements)

As you can see, I try to use pykalman, however I cannot install this module. I try to use easy_install pykalman direction, and the error is invalid syntax. Another problem is, I have a huge data set, so I have more than one hundred thousand rows in my variable column. So, I cannot write all observations one by one.

Khalid
  • 621
  • 1
  • 7
  • 14
  • So whats your question? Also show some code. – miindlek May 07 '18 at 15:22
  • I want to remove noise from my data by using Kalman filtering by using python. My question is how to apply Kalman filtering to my data by employing python. I follow the codes from the mentioned link : https://stackoverflow.com/questions/43377626/how-to-use-kalman-filter-in-python-for-location-data . I cannot get any results. Firstly, I cannot install pykalman. Secondly, I have a huge number of rows. I cannot include the values one by one. – Khalid May 07 '18 at 15:38
  • So you have no python code at all? – miindlek May 07 '18 at 15:42
  • I have. As I said, I employ the code from the mentioned question. I explained in my question that I try to use pykalman and get invalid syntax. If I do no have code, how can I get that error? – Khalid May 07 '18 at 15:46
  • So if you have code, why aren't you able to post it? – miindlek May 07 '18 at 15:50
  • I added it to the question. – Khalid May 07 '18 at 16:01

1 Answers1

0

To install pykalman I used:

pip install pykalman --user

The --user flag installs to my home directory, avoiding having to use sudo to install. I was told that scipy was missing, so I pip installed that as well. On the project github page there is a list of dependent libraries, so you may be asked to install any one of those.

You are using single values for each of your readings. Most examples have more than this, for instance position and velocity for each reading. To get something to plot with the transition and observation matrices you supplied, I added a second bogus reading of '1' to each of your measurements. The following Jupyter notebook script will produce a plot, but the output is poor as the matrices values need to be adjusted for your data set.

%matplotlib inline
from pykalman import KalmanFilter
import numpy as np
kf = KalmanFilter(transition_matrices = [[1, 1], [0, 1]], 
 observation_matrices = [[0.1, 0.5], [-0.3, 0.0]])
# measurements = np.asarray([(250.1),(248.5),(262.3),(265.3), (270.2)])
measurements = np.array([[250.1,1],[248.5,1],[262.3,1],[265.3,1], [270.2,1]])
kf = kf.em(measurements, n_iter=5)
filtered_state_estimates = kf.filter(measurements)[0]
(smoothed_state_estimates, smoothed_state_covariances)=kf.smooth(measurements)
# draw estimates
pl.figure()
lines_true = pl.plot(measurements, color='b')
lines_filt = pl.plot(filtered_state_estimates, color='r')
lines_smooth = pl.plot(smoothed_state_estimates, color='g')
pl.legend((lines_true[0], lines_filt[0], lines_smooth[0]),
          ('true', 'filt', 'smooth'),
          loc='lower right'
)
pl.show()

For the data set that you propose, a fast and simpler way to produce a filtered output would be to use a one minus alpha filter. Have a look at this link for more details on this type of filter: http://stats.stackexchange.com/questions/44650/a-simpler-way-to-calculate-exponentially-weighted-moving-average

Oppy
  • 2,662
  • 16
  • 22