5

I have a dataframe with 30 columns and >10,000 rows.

How can I run an outlier analysis for a set of variables that will return a TRUE if ANY of the variables exceed the particular threshold (for that given variable), or FALSE if the respective outlier thresholds (3SDs) are not met for any of the variables, with the TRUE/FALSE values displaying in a new column?

I have used quantile to find the 3 standard deviation cut-off values for each variable:

i.e.: 

quantile(df$a, 0.003, na.rm = T) #and 

quantile(df$a, 0.997, na.rm = T)

say the first value is 2.5 and the upper value is 10.5 for this variable, I then have created a new variable:

df$outliers <- (df$a <- df$a <2.5 | df$a > 10.5)

which gives TRUE values when values in column a are less than 2.5 or greater than 10.5.

What I would like to do, is have df$outliers represent the outlier status for a set of columns, not just one, i.e columns d, e, f, g, l, m etc, which will all have their own threshold values to meet.

What is the best way to do this?

Ronak Shah
  • 377,200
  • 20
  • 156
  • 213
stat.chat
  • 53
  • 3

2 Answers2

3

Let's assume your dataframe is called df and the columns in which you are interested to check outliers are a, b and c (stored in cols). We can use sapply on those columns find out which value lie in the outlier range. This will return a matrix of TRUE/FALSE values indicating if that particular value is an outlier or not. We take rowSums on it and assign value TRUE if any one column has TRUE value in that row or FALSE otherwise.

cols <- c("a", "b", "c")

df$outliers <- rowSums(sapply(df[cols], function(x) 
                       x < quantile(x, 0.003) | x > quantile(x, 0.997))) > 0

df
#             a          b          c random outliers
#1  -0.56047565  1.2240818 -1.0678237      1    FALSE
#2  -0.23017749  0.3598138 -0.2179749      2    FALSE
#3   1.55870831  0.4007715 -1.0260044      3    FALSE
#4   0.07050839  0.1106827 -0.7288912      4    FALSE
#5   0.12928774 -0.5558411 -0.6250393      5    FALSE
#6   1.71506499  1.7869131 -1.6866933      6     TRUE
#7   0.46091621  0.4978505  0.8377870      7    FALSE
#8  -1.26506123 -1.9666172  0.1533731      8     TRUE
#9  -0.68685285  0.7013559 -1.1381369      9    FALSE
#10 -0.44566197 -0.4727914  1.2538149     10     TRUE

data

set.seed(123)
df <- data.frame(a = rnorm(10), b = rnorm(10), c = rnorm(10), random = 1:10)
Ronak Shah
  • 377,200
  • 20
  • 156
  • 213
  • Hi Ronak, Thanks for your response - this is exactly what I am hoping to achieve. Unfortunately I'm getting the following error: "Error in `[.data.table`(df, cols) : When i is a data.table (or character vector), the columns to join by must be specified either using 'on=' argument (see ?data.table) or by keying x (i.e. sorted, and, marked as sorted, see ?setkey). Keyed joins might have further speed benefits on very large data due to x being sorted in RAM." Do you know what the problem might be? Cheers! – stat.chat Sep 06 '19 at 01:16
  • @stat.chat you have `data.table`, can you change it to data.frame `df1 <- data.frame(df)` and then try above? – Ronak Shah Sep 06 '19 at 01:22
  • Hi Ronak, thanks for this. I have another error though. I'm sorry I'm still very new to R! The error is: "missing values and NaNs not allowed if na.rm is FALSE." I have tried adding na.rm = TRUE into the code, both after [cols], or before the last bracket, however it doesn't work and pops up with an 'unexpected symbol error'. Where should I place the na.rm= T? – stat.chat Sep 08 '19 at 00:23
  • I've added the na.rm after sappy, i.e.: sapply, na.rm = T(df[cols], function ... however now I'm getting this error: Error: attempt to apply non-function – stat.chat Sep 09 '19 at 02:51
  • @stat.chat you need to add that in `quantile` function. Try `df$outliers <- rowSums(sapply(df[cols], function(x) x < quantile(x, 0.003, na.rm = TRUE) | x > quantile(x, 0.997, na.rm = TRUE))) > 0` – Ronak Shah Sep 09 '19 at 13:45
0

In general an observation is an outlier if it is outlier for one or more feature. But I dont know what your dealing with so it could be different you have to find how the problem your working on define an outlier then you can choose the features that are important and the thresholds.

Going back to the first definition you can create your column as intersection of the results of the same process you made for all variables.

However you should avoid doing this manually, so you create a table of all variable's thresholds then create a function that returns trur if the observation is an outlier for at least one variable

benaou mouad
  • 392
  • 3
  • 10