Print cases with correlations above a threshold

Question

I have a relatively large correlation matrix (77x77). I need to identify cases with correlations above |.60|. Below I produced the correlation dataframe and made all values < |.60| show as NA.

cor_relation = cor(mydata, use="all.obs", method="pearson") cor_relation[abs(cor_relation) < 0.6] <- NA

However, it is still difficult to manually search for the cases > |.60|. I tried using Boolean operators, such as below but it only gives me a truncated summary of each case (i.e., Na or True), where I can only see a summary of the first 1000 cases

cor_relation[abs(cor_relation)] >= 0.6

Please, help out with the correct code that will only print a summary of cases with correlations > |.60|

Thank you

Answer 1

A dplyr way:

library(dplyr)
data(mtcars)
correl <- round(cor(mtcars),2)
cor_df <- as.data.frame(as.table(correl))

And now cor_df can be sorted and filtered:

> cor_df %>%  arrange(desc(Freq)) %>% filter(Freq>0.6)
   Var1 Var2 Freq
1   mpg  mpg 1.00
2   cyl  cyl 1.00
3  disp disp 1.00
4    hp   hp 1.00
... and so on

> cor_df %>%  arrange(desc(Freq)) %>% filter(Freq< -0.6)
   Var1 Var2  Freq
1  carb qsec -0.66
2  qsec carb -0.66
3    am   wt -0.69
4    wt   am -0.69
... and so on

And you can summarize the data:

> cor_df %>%  arrange(desc(Freq)) %>% filter(Freq< -0.6) %>% 
     summarise(mean=mean(Freq), sd=sd(Freq))
        mean         sd
1 -0.7515385 0.06949488

Answer 2

Here is an example of how you could do it (using dummy data as no data was provided)

# creating dummy corr matrix
corr <- matrix(runif(25, -1, 1), nrow = 5, ncol = 5)
# selecting indices where corr[i,j] >= 0.6 
S <- which(abs(corr) >= .6, arr.ind = T)
# saving the result in a new vector
val <- corr[S]
val
[1] -0.9645378 -0.6560426 -0.7547376  0.7940875 -0.7449578 -0.8844122  0.6800387
# and finally printing a summary
summary(val)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
-0.9645 -0.8196 -0.7450 -0.3615  0.0120  0.7941