1

I have a signal that looks like this (given in blue), and what I want is to remove the unwanted "spikes" to yield the signal that I drew by hand shown in black. Simple tools like median filter, etc, do not work since the spacing of these peaks can vary widely. Bandpass tools also will distort the edge of the desired black signal, and also the spikes do not appear in any consistent frequency range (i.e., they're not particularly periodic, as can be seen). Does anyone have any other ideas as to how to achieve this? One thing I've recognized is a "spike" has both a positive and negative edge, while the underlying signal does not necessarily have that corresponding negative edge, although I haven't figured out how to make use of that in any meaningful way yet. Possible tools I've considered are machine learning models, but I think that's overkill, although maybe not. Another thing that is useful is it is safe to assume after that initial rise the true signal is monotonically decreasing

enter image description here

user2551700
  • 89
  • 1
  • 6

1 Answers1

0

Beyond a single pass of median filtering, sometimes multiple passes of median filtering can do a better job at removing spikes. The justification is that iterated median filtering has a close connection to total variation (TV) denoising, which is an excellent technique for removing spikes, and doing a few median filtering passes is easier to implement and faster than TV denoising while giving qualitatively similar results.

I don't have the original data from your figure, but here is an approximation of what I would get by running 5 passes of median filtering with window size 25.

Plot of filtered data

It does a reasonable job removing the spikes while preserving the jump at around index 50. Increase the number of passes and/or median filter window size to increase the overall strength of the filtering.

Code:

# Copyright 2023 Google LLC.
# SPDX-License-Identifier: Apache-2.0
import scipy.ndimage
for _ in range(5):
  x = scipy.ndimage.median_filter(x, 25)
Pascal Getreuer
  • 2,906
  • 1
  • 5
  • 14