3

I need to detect singular points (extremas, trend change, sharp changes) on a given curve plotted from a dataset. The first thing to come in mind is the inflexion point detection with derivation( but i dont have the mathematical expression of the plotted curve), second is how to detect angular points. so if possible can i build (using python) a sliding window which detects these kind of SP(singular points), if possible what are the libraries and function used ?

Thanks

Singular point detection

skull95
  • 33
  • 3
  • Wouldn’t it be simpler to determine these points from the dataset directly, instead of from a graphical representation? – mkrieger1 Mar 27 '19 at 07:50
  • 2
    There's very little information present here so far. Do you have access to the array of values that is plotted in the graph? Can you prepare a [Minimal, Complete and Verifiable](https://stackoverflow.com/help/mcve) Example? – Paritosh Singh Mar 27 '19 at 07:51
  • Indeed it's easier to do it from the dataset (that i have access to) , i just wanted to verify if there is already an existing algorithm of this "sliding window". Thanks – skull95 Mar 27 '19 at 08:16

1 Answers1

2

I just scraped some of your data to show you that you can find points on the whole dataset, without using a sliding window (but you could, in theory):

  1. Local extrema (find peaks in raw data)
  2. Max Steepness (find peaks in 1st derivative)
  3. Inflexion points (find peaks in 2nd derivative)

First, let's have a look on calculating the derivatives:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("Default Dataset.csv", 
                 sep=';', 
                 decimal=",", 
                 header=None)

### Interpolate linearily ###
x_new = np.linspace(0, df[0].iloc[-1], 2000)
y_new = np.interp(x_new, df[0], df[1])

### First and second derivative ###
diff1 = np.insert(np.diff(y_new), 0, 0)
diff2 = np.insert(np.diff(diff1), 0, 0)

### Plot everything ###
plt.figure(figsize=(12,3))
plt.subplot(131)
plt.plot(x_new, y_new)
plt.subplot(132)
plt.plot(x_new, diff1)
plt.subplot(133)
plt.plot(x_new, diff2)
plt.tight_layout()

Here, I also interpolate to have an equal spacing between the datapoints. Further, I insert a 0 at position 0 using the np.insert function after the differentiation, to ensure the same shape as the raw data.

Comparison of raw, 1st deriv and 2nd deriv

Next, we will find the peaks:

import peakutils as pu

ix_abs   = pu.indexes(y_new, thres=0.5, min_dist=15)
ix_diff1 = pu.indexes(diff1, thres=0.5, min_dist=15)
ix_diff2 = pu.indexes(diff2, thres=0.5, min_dist=15)

plt.scatter(x_new[ix_abs], y_new[ix_abs], color='g', label='abs')
plt.scatter(x_new[ix_diff1], y_new[ix_diff1], color='r', label='first deriv')
plt.scatter(x_new[ix_diff2], y_new[ix_diff2], color='purple', label='second deriv')

plt.plot(x_new, y_new)
plt.legend(loc='best')

peaks in data

I am using the peakutils package, because it works nicely in almost all cases. You see that not all points that were indicated in your example were found. You can play around with different parameters for threshold and minimum distance to find a better solution. But this should be a good starting point for further research. Indeed, the minimum distance parameter would give you the desired sliding window.

anki
  • 765
  • 5
  • 14
  • Hi, Can you please let me know what this line in your code does, x_new = np.linspace(0, df[0].iloc[-1], 2000). What is x_new / y_new?. – imantha Nov 28 '19 at 08:44
  • The data was not spaced equally, so I interpolated them with equal spacing. – anki Nov 28 '19 at 15:20