Apply dcast() function to the list of dataframes

Question

I have a list of data frames which I show some of it below:

> str(ls.df.val.keep)

List of 87
 $ AT_VALUES_IMV_EU28_EXTRA  :'data.frame': 1554 obs. of  4 variables:
  ..$ PARTNER        : Factor w/ 3 levels "EU28_EXTRA","EU28_INTRA",..: 1 1 1 1 1 1 1 1 1 1 ...
  ..$ PRODUCT        : int [1:1554] 2207 2207 2207 2207 2207 2207 2207 2207 2207 2207 ...
  ..$ PERIOD         : Date[1:1554], format: "2010-01-01" "2010-02-01" "2010-03-01" "2010-04-01" ...
  ..$ INDICATOR_VALUE: num [1:1554] 1033828 291810 227820 104650 254214 ...
 $ AT_VALUES_IMV_EU28_INTRA  :'data.frame': 4380 obs. of  4 variables:
  ..$ PARTNER        : Factor w/ 3 levels "EU28_EXTRA","EU28_INTRA",..: 2 2 2 2 2 2 2 2 2 2 ...
  ..$ PRODUCT        : int [1:4380] 2207 2207 2207 2207 2207 2207 2207 2207 2207 2207 ...
  ..$ PERIOD         : Date[1:4380], format: "2010-01-01" "2010-02-01" "2010-03-01" "2010-04-01" ...
  ..$ INDICATOR_VALUE: num [1:4380] 1894626 2177029 1271173 994368 1746904 ...
 $ AT_VALUES_IMV_IN          :'data.frame': 6 obs. of  4 variables:
  ..$ PARTNER        : Factor w/ 3 levels "EU28_EXTRA","EU28_INTRA",..: 3 3 3 3 3 3
  ..$ PRODUCT        : int [1:6] 2208 2208 220890 220890 22089069 22089069
  ..$ PERIOD         : Date[1:6], format: "2010-09-01" "2012-07-01" "2010-09-01" "2012-07-01" ...
  ..$ INDICATOR_VALUE: num [1:6] 58 37 58 37 58 37
.
.
.
.

I would like to use the custom function dcast() to the elements of this list which are my 87 dataframes. The desired result should look like this for each data frame depending on the number of columns that it has:

   22082027 22082029 22082040 22082062 22082086 22082089 22083011 22083019 22083030 22083041 22083061 22083069 22083071 22083082 22084011
1        NA       NA       NA       NA       NA       NA   400675       NA     3103       NA      186       NA       NA       NA    22297
2        NA     2594       NA       NA       NA       NA   667309     3011       NA       NA       NA       NA       NA     1916       NA
3        NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA      263    93482
4        NA       NA       NA       NA       NA       NA   503155     4516       NA       NA       NA       NA       NA     5359       NA
5        NA      175       NA       NA       NA       NA   226936     4514       NA       NA       NA       NA    39389       NA       NA

I have tried various solutions and pieces of codes and non is working:

## First solution
all.cast <- list()
for (i in 1:length(ls.df.val.keep)) {
  all.cast[[i]] <- dcast(ls.df.val.keep[[i]], PERIOD ~ PRODUCT, value.var = "INDICATOR_VALUE")
}
ls.df.val.dcas <- list()
ls.df.val.dcas <- lapply(names(ls.df.val.keep), myfunc)

It produces error:

Error: value.var (INDICATOR_VALUE) not found in input
Called from: cast(data, formula, fun.aggregate, ..., subset = subset, fill = fill, 
    drop = drop, value.var = value.var)

Second solution

ls.df.val.dcas < - lapply(ls.df.val.keep, function(x) dcast(x, PERIOD ~ PRODUCT, value.var="INDICATOR_VALUE"))

The errors is:

Error in dim(ordered) <- ns : 
  dims [product 1] do not match the length of object [0]

Third solution

llply(ls.df.val.keep, .fun=function(x) dcast(x, PERIOD ~ PRODUCT, value.var="INDICATOR_VALUE"))

The error is:

Error in do.ply(i) : 
  task 9 failed - "dims [product 1] do not match the length of object [0]"
In addition: Warning message:
In setup_parallel() : No parallel backend registered

Fourth Solution (desperate one)

all.cast <- list()
for (i in 1:length(ls.df.val.keep)) {
  all.cast[[i]] <- dcast(ls.df.val.keep[[i]], PERIOD ~ PRODUCT, value.var = "INDICATOR_VALUE")
}

It gets error again:

Error in dim(ordered) <- ns : 
  dims [product 1] do not match the length of object [0]

Strange enough, if I put the name of each dataframe or just the element of the list e.g. ls.df.val.keep[[3]], they work! I have many other things that I want to do to this list using lapply but I am stucked in the first step! Is there something that I am not seeing?

EDITED.

I am adding a sample of my dataframe data as @Jake requested:

> head(ls.df.val.keep[[1]])
         PARTNER    PRODUCT     PERIOD             INDICATOR_VALUE
58136 EU28_EXTRA    2207     2010-01-01         1033828
58138 EU28_EXTRA    2207     2010-02-01         291810
58140 EU28_EXTRA    220710   2010-03-01         227820
58142 EU28_EXTRA    220710   2010-04-01         104650
58144 EU28_EXTRA    22071000 2010-09-01         254214
58146 EU28_EXTRA    22071000 2010-10-01         572446

.. .. ..

In the second solution, `val.dcas < - ` should be `val.dcas <- `. — lmo, Nov 15 '16 at 14:55
What do you get from `all(sapply(ls.df.val.keep, function(x) length(dim(x)) == 2 && all(dim(x) > 0)))`? — Pierre L, Nov 15 '16 at 14:59
@imo. Thx. It was a typo. I edited the question. The typo was not causing any error though. — Feraman, Nov 15 '16 at 15:22
It should return `TRUE`. You have an object in your list that is not a valid array. Change the first part of the above test to `which(sapply(ls.df.val.keep,.....)` and you'll see where the problem is. — Pierre L, Nov 15 '16 at 15:27
@Pierre Brilliant! It seems the 9th element is an empty dataframe! Cheers — Feraman, Nov 15 '16 at 15:36

Jake Kaupp · Accepted Answer · 2016-11-15T15:56:26.793

0

Pulling in Pierre's finding of empty data frames throwing the error, you can use purrr to achieve your goals.

library(purrr)
library(dplyr)
library(reshape2)

dcast_list <- map_if(ls.df.val.keep, negate(is_empty), ~dcast(.x, PERIOD ~ PRODUCT, value.var = "INDICATOR_VALUE"))

This will return a list of data frames, and keep the null data frames intact. If you want them removed, filter them out prior to the map.

edited Nov 15 '16 at 15:56

answered Nov 15 '16 at 15:03

Jake Kaupp

7,892
2
26
36

It makes one of the errors that I mentioned. Precisely this one: Error in dim(ordered) <- ns : dims [product 1] do not match the length of object [0] – Feraman Nov 15 '16 at 15:19
Is this still the case? I tested this using the data you provided, along with a null data frame in the list and it works. – Jake Kaupp Nov 18 '16 at 00:26
The code provided by you is fine and the problem was with one of my list arrays which was empty. Cheers – Feraman Nov 18 '16 at 16:40
Great, just wanted to know if you still needed help! – Jake Kaupp Nov 18 '16 at 16:40

Apply dcast() function to the list of dataframes

Second solution

Third solution

Fourth Solution (desperate one)

EDITED.

1 Answers1