8

I am just curious whether it is more efficient to store data in long or wide format regardless of the interpretative? I have used object.size() to determine the size in the memory but they do not differ significantly (the long being slightly more efficient in terms of size) and the value is only and estimate.

On top of the raw size, I am also wondering which of the format is more efficient in terms of being manipulated when used in modelling.

csgillespie
  • 59,189
  • 14
  • 150
  • 185
MKao
  • 281
  • 4
  • 11
  • To ask it in another way, is it more efficient to do matrix/data manipulation on a matrix of 1000 * 10 or 100 * 100? Thanks again. – MKao Nov 18 '11 at 11:05

2 Answers2

7

The memory usage of the two different matrixes should be identical:

> object.size(long <- matrix(seq(10000), nrow = 1000))
40200 bytes
> object.size(square <- matrix(seq(10000), nrow = 100))
40200 bytes

Any differences in efficiency will be dwarfed by the inefficiency in using R, so hardly need to be considered, if they are even measurable.

The situation is very different for a data.frame, since it is implemented as a list of vectors:

> object.size(as.data.frame(long))
41704 bytes
> object.size(as.data.frame(square))
50968 bytes

The time efficiency of this will depend on what exactly you want to do.

Michael Hoffman
  • 32,526
  • 7
  • 64
  • 86
  • Thanks Michael, I've done exactly the same test for both matrix and data.frame and came to the same conclusion. I want to know say the efficiency of these two different format when implemented by the underlying algorithm that deals with matrix (e.g. LAPACK) and also the algorithm used in various model. – MKao Nov 18 '11 at 11:33
  • Which algorithm? Provide some example code that you would like to compare the efficiency of. – Michael Hoffman Nov 18 '11 at 11:35
  • I am not comparing specific examples, but I am asking in general. Which format is in favored for most algorithm/manipulation within R. If you were to give advice to someone, which one would you recommend and why? – MKao Nov 18 '11 at 12:02
  • There is no answer "in general." Neither format is more favored for most algorithms within R. – Michael Hoffman Nov 18 '11 at 12:14
2

For a matrix there will be absolutely no difference. The same is true for a data.frame of that matrix. Reforming the shape of a matrix is merely assigning dimension attributes... for the most part.

If you are going to categorize that data in some way and add additional information then wide will usually be more efficient storage wise but long will generally be handled more efficiently. This isn't a necessary property of long format, that it's less space efficient, but generally you would have a compound variable description in the column names in wide that would be separated and given a new column, or multiple columns in long. Therefore, it will take up more space due to those redundancies. On the handling side it's easier to aggregate the long data or select specific cases for deletion than in a wide format that has multivariate column designations.

Long is also the best way (of these two) if the data are not perfectly rectangular (or cubic, etc).

John
  • 23,360
  • 7
  • 57
  • 83