Calculate pairwise correlation in R using dplyr::mutate

Question

I have a large data frame with on every rows enough data to calculate a correlation using specific columns of this data frame and add a new column containing the correlations calculated.

Here is a summary of what I would like to do (this one using dplyr):

example_data %>%
mutate(pearsoncor = cor(x = X001_F5_000_A:X030_F5_480_C, y = X031_H5_000_A:X060_H5_480_C))

Obviously it is not working this way as I get only NA's in the pearsoncor column, does anyone has a suggestion? Is there an easy way to do this?

Best,

Example data frame

Answer 1

With tidyr, you can gather separately all x- and y-variables, you'd like to compare. You get a tibble containing the correlation coefficients and their p-values for every combination you provided.

library(dplyr)
library(tidyr)

example_data %>%
  gather(x_var, x_val, X001_F5_000_A:X030_F5_480_C) %>% 
  gather(y_var, y_val, X031_H5_000_A:X060_H5_480_C) %>% 
  group_by(x_var, y_var) %>% 
  summarise(cor_coef = cor.test(x_val, y_val)$estimate,
            p_val = cor.test(x_val, y_val)$p.value)

edit, update some years later:

library(tidyr)
library(purrr)
library(broom)
library(dplyr)

longley %>%
  pivot_longer(GNP.deflator:Armed.Forces, names_to="x_var", values_to="x_val") %>% 
  pivot_longer(Population:Employed, names_to="y_var", values_to="y_val") %>% 
  nest(data=c(x_val, y_val)) %>%
  mutate(cor_test = map(data, ~cor.test(.x$x_val, .x$y_val)),
         tidied = map(cor_test, tidy)) %>% 
  unnest(tidied)

Answer 2

Here is a solution using the reshape2 package to melt() the data frame into long form so that each value has its own row. The original wide-form data has 60 values per row for each of the 6 genes, while the melted long-form data frame has 360 rows, one for each value. Then we can easily use summarize() from dplyr to calculate the correlations without loops.

library(reshape2)
library(dplyr)

names1 <- names(example_data)[4:33]
names2 <- names(example_data)[34:63]

example_data_longform <- melt(example_data, id.vars = c('Gene','clusterFR','clusterHR'))

example_data_longform %>%
  group_by(Gene, clusterFR, clusterHR) %>%
  summarize(pearsoncor = cor(x = value[variable %in% names1],
                             y = value[variable %in% names2]))

You could also generate more detailed results, as in Eudald's answer, using do():

detailed_r <- example_data_longform %>%
  group_by(Gene, clusterFR, clusterHR) %>%
  do(cor = cor.test(x = .$value[.$variable %in% names1],
                    y = .$value[.$variable %in% names2]))

This outputs a tibble with the cor column being a list with the results of cor.test() for each gene. We can use lapply() to extract output from the list.

lapply(detailed_r$cor, function(x) c(x$estimate, x$p.value))

Answer 3

I had the same problem a few days back, and I know loops are not optimal in R but that's the only thing I could think of:

df$r = rep(0,nrow(df))
df$cor_p = rep(0,nrow(df))

for (i in 1:nrow(df)){
  ct = cor.test(as.numeric(df[i,cols_A]),as.numeric(df[i,cols_B]))
df$r[i] = ct$estimate
df$cor_p[i] = ct$p.value
}