1

Problem:

I'm currently parsing a time series dataset, of [x,y] coordinates. The data isn't complete - it contains gaps and jitter, and I would like to fill these gaps / normalise the jitter using statistical analysis.

Background:

I'm currently reading up on non-linear regression (specifically polynomial regression -> PR) - which seems to be the best fit (pun intended) for my problem.

I realise that PR deals with arcs "turning in one direction" so, I'm going to try to refactor my code to work with smaller sample sizes - and work my way along the time series.

Questions:

  1. Am I on the right track?
  2. Is there a name for what I'm trying to do? (curve fitting? trendline? continuous regression?)
  3. Is there another technique I can/should use, that provides a better "fit" for my data?
Nick Grealy
  • 24,216
  • 9
  • 104
  • 119
  • 2
    sooo.... this code is highly bound to how things might be drawn on a canvas. It seems like your question is more about how to un-jitter a jittery set of x/y data over time. So your actual question might be "how do I smooth a ton of data points where I'm missing chunks of time", but your code is more like "how do I smooth out this junky looking jigsaw chart into a curve". I think you need to decide whether you're trying to solve the actual underlying math question (what the missing data is) or the graphics question (how to make it look smooth on a chart). Just my 2p here. – joshstrike Jul 27 '20 at 11:24
  • If *restoring* missing chunks of data is your goal, you may look to [piecewise linear approximation](https://optimization.mccormick.northwestern.edu/index.php/Piecewise_linear_approximation#:~:text=A%20piecewise%20linear%20approximation%20is,to%20reformulate%20the%20original%20problem.), e.g. [like that one](https://stackoverflow.com/a/59264756/11299053) – Yevhen Horbunkov Jul 27 '20 at 11:45
  • thanks @joshstrike, I've dropped the code snippet as it was misleading. Just wanted to show effort. – Nick Grealy Jul 27 '20 at 12:38

1 Answers1

2

It seems to me that you are in the right direction. Using a fitting technique like PR can help you smooth the data and thus get rid of the jtters, and resample in the missing gaps. Although that finding a proper function could complete both of the tasks, if you don't have a clear view of what you kind of function to fit, you can try other methods:

  1. In order to deal with short term jitters and fluctuation you can use smoothing techniques like moving average (like in here) or Cubic smoothing splines which uses polynomial regression on short window of the data (see [here]) 2.

  2. In order to fill the missing gap" you could interpolation techniques like Nearest-Neighbor Interpolation, Regression, or Auto-Regressive Methods, it really depends what kind of data do you have. I will suggest look here and find the right method for you.

libyair
  • 21
  • 2