select columns from dataframe where groups of samples are nonzero

Question

I have a sample (rows) by species (columns) dataframe. And a column in another dataframe that codes the samples into groups. I want to select all of the columns where all of the samples in any of the groups have a nonzero value.

species frame:

structure(list(Otu000132 = c(0L, 56L, 30L, 52L, 1L, 4L, 31L, 4L, 17L, 9L, 4L), 
               Otu000144 = c(191L, 14L, 58L, 137L, 127L, 222L, 26L, 175L, 133L, 107L, 43L),
               Otu000146 = c(0L, 0L, 0L, 0L, 16L, 62L, 41L, 16L, 60L, 32L, 0L), 
               Otu000147 = c(0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L), 
               Otu000151 = c(2L, 9L, 4L, 1L, 0L, 4L, 4L, 2L, 3L, 0L, 0L),
               Otu000162 = c(2L, 1L, 0L, 0L, 1L, 1L, 0L, 2L, 1L, 0L, 0L), 
               Otu000164 = c(2L, 0L, 1L, 2L, 0L, 0L, 0L, 0L, 0L, 0L, 0L),
               Otu000174 = c(0L, 0L, 3L, 1L, 0L, 2L, 0L, 1L, 2L, 1L, 0L), 
               Otu000176 = c(1L, 9L, 0L, 1L, 2L, 5L, 3L, 3L, 8L, 2L, 2L), 
               Otu000186 = c(1L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L),
               Otu000190 = c(1L, 1L, 1L, 0L, 0L, 5L, 1L, 2L, 7L, 0L, 0L)),
          .Names = c("Otu000132", "Otu000144", "Otu000146", "Otu000147", 
                     "Otu000151", "Otu000162", "Otu000164", "Otu000174", 
                     "Otu000176", "Otu000186", "Otu000190"),
          row.names = 30:40, class = "data.frame")

grouping frame:

structure(c(30, 31, 32, 33, 34, 35, 36, 37, 38, 39,
            40, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3), 
          .Dim = c(11L, 2L))

desired output:

structure(list(Otu000132 = c(0L, 56L, 30L, 52L, 1L, 4L, 31L, 4L, 17L, 9L, 4L), 
               Otu000144 = c(191L, 14L, 58L, 137L, 127L, 222L, 26L, 175L, 133L, 107L, 43L), 
               Otu000151 = c(2L, 9L, 4L, 1L, 0L, 4L, 4L, 2L, 3L, 0L, 0L), 
               Otu000176 = c(1L, 9L, 0L, 1L, 2L, 5L, 3L, 3L, 8L, 2L, 2L),
               Otu000190 = c(1L, 1L, 1L, 0L, 0L, 5L, 1L, 2L, 7L, 0L, 0L)), 
          .Names = c("Otu000132", "Otu000144",  "Otu000151", 
                     "Otu000176", "Otu000190"),
          row.names = 30:40, class = "data.frame")

I feel like this should be something that I could do with dplyr select, but I can't figure it out. Anyone have suggestions for starting me on a path?

It is not very clear. You have the third column as 'Otu000146' which have 4 0's i.e. the 30, 31, and 32 are 0. Should that column be included in the desired output? Otherwise `sp1[!Reduce(`&`,lapply(split(gp1[,1], gp1[,2]), function(x) {x1 <- sp1[match(x, row.names(sp1)),]; colSums(x1==0)>0}))]` would give all the other columns in the desired. — akrun, Oct 04 '16 at 15:41
error on my part, I thought it was present in all of group 2 but it's not — thermophile, Oct 04 '16 at 15:43

score 1 · Accepted Answer · edited Oct 04 '16 at 18:41

This can indeed be done with dplyr, and in a fairly straightforward way. As others have pointed out, "Otu000146" does not meet your described criteria and would not be included in the final column selection.

library(dplyr)
library(tidyr)

df.species <- cbind(species, group = grouping[,2]) %>% # merge the grouping variable into the main data set
    gather(variable, value, -group) %>%  # gather the columns into 'long' format
    group_by(variable, group) %>% # group by column name and group
    summarize(keep = all(value != 0)) %>% # variables and groups where all values are non-zero
    ungroup %>% group_by(variable) %>%  # reset grouping
    summarize(keep = any(keep)) %>%  # variables where at least 1 group met the aforementioned criterion
    dplyr::filter(keep) # final list

   variable  keep
      <chr> <lgl>
1 Otu000132  TRUE
2 Otu000144  TRUE
3 Otu000151  TRUE
4 Otu000176  TRUE
5 Otu000190  TRUE

# retrieve only the matching columns
df.desired <- species[df.species$variable]

   Otu000132 Otu000144 Otu000151 Otu000176 Otu000190
30         0       191         2         1         1
31        56        14         9         9         1
32        30        58         4         0         1
33        52       137         1         1         0
34         1       127         0         2         0
35         4       222         4         5         5
36        31        26         4         3         1
37         4       175         2         3         2
38        17       133         3         8         7
39         9       107         0         2         0
40         4        43         0         2         0

gather is not in dplyr, need to load tidyr as well – thermophile Oct 04 '16 at 18:40 — thermophile, Oct 04 '16 at 18:40

akrun · Answer 2 · 2016-10-04T15:52:38.960

We split the first column of grouping dataset ('gp1') by the second (gp1[,2]) to a list, loop through the list, subset the rows of the species dataset by matching its row names with the list elements, get the column sums of logical matrix (x1==0), check if that is greater than 0, compare the corresponding elements of each list element using & in Reduce, negate (!) the index to change TRUE to FALSE (and viceversa) to subset the columns of species dataset.

sp1[!Reduce(`&`,lapply(split(gp1[,1], gp1[,2]), function(x) {
                x1 <- sp1[match(x, row.names(sp1)),]
                colSums(x1==0)>0}))]
#    Otu000132 Otu000144 Otu000151 Otu000176 Otu000190
#30         0       191         2         1         1
#31        56        14         9         9         1
#32        30        58         4         0         1
#33        52       137         1         1         0
#34         1       127         0         2         0
#35         4       222         4         5         5
#36        31        26         4         3         1
#37         4       175         2         3         2
#38        17       133         3         8         7
#39         9       107         0         2         0
#40         4        43         0         2         0

this works, thanks. I selected the other one because understanding what's going on within an lapply is difficult — thermophile, Oct 04 '16 at 15:59

rocket1906 · Answer 3 · 2016-10-04T16:23:56.913

You could do it with dplyr or just with base functions as such:

species = merge(species, group, by.x=c("row.names"), by.y=c("V1"))

#Find the lowest values in each grouping
check = aggregate(species[,c("Otu000132", "Otu000144", "Otu000146", 
                   "Otu000147", "Otu000151", "Otu000162", "Otu000164", 
                   "Otu000174", "Otu000176", "Otu000186", "Otu000190")], 
                    by=list(species$V2), min)

#sum across the groupings
vars = apply(check, 2, function(x) sum(x))

#retain variables where sum > 1, indicating at least one grouping has full observations
vars = vars[vars!=0]

#extract the variable names
vars = names(vars)[-1]

#subset dataset to select variables identified above
out = species[vars]

out
#   Otu000132 Otu000144 Otu000151 Otu000176 Otu000190
#1          0       191         2         1         1
#2         56        14         9         9         1
#3         30        58         4         0         1
#4         52       137         1         1         0
#5          1       127         0         2         0
#6          4       222         4         5         5
#7         31        26         4         3         1
#8          4       175         2         3         2
#9         17       133         3         8         7
#10         9       107         0         2         0
#11         4        43         0         2         0

You have some any/all confusion. This sums each group for each column, or in other words, whether ANY values in the group are non-zero. OP wanted to know about groups where ALL values are non-zero, and then take columns with ANY groups matching that criteria. — jdobres, Oct 04 '16 at 16:02
You're right, thanks for catching it. I modified the code to address this. — rocket1906, Oct 04 '16 at 16:25

select columns from dataframe where groups of samples are nonzero

3 Answers3