Remove highly correlated variables

Question

I have a huge dataframe 5600 X 6592 and I want to remove any variables that are correlated to each other more than 0.99 I do know how to do this the long way, step by step i.e. forming a correlation matrix, rounding the values, removing similar ones and use the indexing to get my "reduced" data again.

cor(mydata)
mydata <- round(mydata,2)
mydata <- mydata[,!duplicated (mydata)]
## then do the indexing...

I would like to know if this could be done in short command, or some advanced function. I'm learning how to make use of the powerful tools in the R language, which avoids such long unnecessary commands

I was thinking of something like

mydata <- mydata[, which(apply(mydata, 2, function(x) !duplicated(round(cor(x),2))))]

Sorry I know the above command doesn't work, but I hope I would be able to do this.

a play-data that applies to the question:

mydata <- structure(list(V1 = c(1L, 2L, 5L, 4L, 366L, 65L, 43L, 456L, 876L, 
78L, 687L, 378L, 378L, 34L, 53L, 43L), V2 = c(2L, 2L, 5L, 4L, 
366L, 65L, 43L, 456L, 876L, 78L, 687L, 378L, 378L, 34L, 53L, 
41L), V3 = c(10L, 20L, 10L, 20L, 10L, 20L, 1L, 0L, 1L, 2010L, 
20L, 10L, 10L, 10L, 10L, 10L), V4 = c(2L, 10L, 31L, 2L, 2L, 5L, 
2L, 5L, 1L, 52L, 1L, 2L, 52L, 6L, 2L, 1L), V5 = c(4L, 10L, 31L, 
2L, 2L, 5L, 2L, 5L, 1L, 52L, 1L, 2L, 52L, 6L, 2L, 3L)), .Names = c("V1", 
"V2", "V3", "V4", "V5"), class = "data.frame", row.names = c(NA, 
-16L))

Many thanks

score 55 · Accepted Answer · edited Oct 11 '21 at 03:49

55

I'm sure there are many ways to do this and certainly some better than this, but this should work. I basically just set the upper triangle to be zero and then remove any rows that have values over 0.99.

tmp <- cor(data)
tmp[upper.tri(tmp)] <- 0
diag(tmp) <- 0

# Above two commands can be replaced with 
# tmp[!lower.tri(tmp)] <- 0

 
data.new <- 
  data[, !apply(tmp, 2, function(x) any(abs(x) > 0.99, na.rm = TRUE))]
head(data.new)

   V2 V3 V5
1   2 10  4
2   2 20 10
3   5 10 31
4   4 20  2
5 366 10  2
6  65 20  5

edited Oct 11 '21 at 03:49

Kim

4,080
2
30
51

answered Aug 16 '13 at 14:33

David

9,284
3
41
40

Thanks David, it does the Job, although I don't know what the upper triangle is! I found the R help page but I cannot really understand what it does! :) – Error404 Aug 16 '13 at 14:38
@Error404 `upper.tri` just fills that part of a matrix with "TRUE" (and the rest is zero aka FALSE) , so `tmp[upper.tri(tmp)]` selects only the upper triangle portion of `tmp` . – Carl Witthoft Aug 16 '13 at 14:51
12

Might be clearer if you do `data[, apply(tmp,2,function(x) all(x<=0.99))]` Don't use no double negatives :-) – Carl Witthoft Aug 16 '13 at 14:52
Interesting simplification of the command :) I'll play around with the upper triangle. Thanks buddy – Error404 Aug 16 '13 at 14:59
2

Hi can anyone help when i use norm.num[, apply(tmp,2,function(x) any(x > 0.99))] on my data set, i get error message saying Error in `[.data.frame`(norm.num, , !apply(tmp, 2, function(x) any(abs(x) > : undefined columns selected – alily Sep 22 '16 at 08:50
I edited the answer so that high degree of negative correlation would be also accounted + also deal with missing values. – Kim Oct 11 '21 at 03:50

score 52 · Answer 2 · edited Sep 14 '16 at 07:10

This is my R code this would be helpfull for you

library('caret')

df1 = read.csv("stack.csv")

print (df1)

     GA     PN     PC   MBP    GR    AP
1 0.033  6.652  6.681 0.194 0.874 3.177
2 0.034  9.039  6.224 0.194 1.137 3.400
3 0.035 10.936 10.304 1.015 0.911 4.900
4 0.022 10.110  9.603 1.374 0.848 4.566
5 0.035  2.963 17.156 0.599 0.823 9.406
6 0.033 10.872 10.244 1.015 0.574 4.871
7 0.035 21.694 22.389 1.015 0.859 9.259
8 0.035 10.936 10.304 1.015 0.911 4.500


df2 = cor(df1)
hc = findCorrelation(df2, cutoff=0.3) # putt any value as a "cutoff" 
hc = sort(hc)
reduced_Data = df1[,-c(hc)]
print (reduced_Data)

     GA     PN    GR    AP
1 0.033  6.652 0.874 3.177
2 0.034  9.039 1.137 3.400
3 0.035 10.936 0.911 4.900
4 0.022 10.110 0.848 4.566
5 0.035  2.963 0.823 9.406
6 0.033 10.872 0.574 4.871
7 0.035 21.694 0.859 9.259
8 0.035 10.936 0.911 4.500

and to write down a reduced data into new csv just use:

write.csv(reduced_Data, file = "outfile.csv", row.names = FALSE)

@AnkitDhingra - `findCorrelation` is a function built into the `caret` package that jax loaded on his first line. — n1k31t4, Dec 14 '15 at 21:28

Vamshidhar H.K. · Answer 3 · 2016-04-05T10:13:41.627

18

@David A small change in your code make it more robust to negative correlation , by providing

abs(x) > 0.99

instead of only

x > 0.99

data.new <- data[,!apply(tmp,2,function(x) any(abs(x) > 0.99))]

cheers..!!!

edited Apr 05 '16 at 10:13

answered Apr 05 '16 at 10:07

Vamshidhar H.K.

198
1
7

Remove highly correlated variables

3 Answers3

Linked