Checking the variance (or elasticity) of a variable over time in panel data

Question

I have a panel dataset which is similar to the one below:

  Country Ccode Year Happiness Power ID
1  France    FR 2000      1000  1000 01
2  France    FR 2001      1200  1400 01
3  France    FR 2000      1400  1800 02
4  France    FR 2001      1600  2200 02
5      UK    UK 2000      1000  1000 03
6      UK    UK 2001      1000  1000 03
7      UK    UK 2000      1000  1000 04
8      UK    UK 2001      1000  1000 04

What I am actually interested in is to what degree a variable is elastic with respect to time. In calculating this elasticity I want to make use of the panel data.

This is how far I got:

library(tidyverse)
df <- df %>%
  arrange(ID, Year)
  group_by(ID) %>%
  mutate_if(is.numeric, funs(d = . - lag(.)))

But this just calculates the differences for each variable instead of the variance of a specific one.

Any suggestions on how to do this properly?

Does `summarize_if(is.numeric, var)` instead of the `mutate_if`-call provide the answer? — kath, Aug 15 '18 at 12:53
If I am correct, this will only give me the variance, not the variance over time, right? — Tom, Aug 15 '18 at 13:08
That's correct, for a variance over time you'd need to specify something like a moving window, or do you always only want to consider the current and the next value? — kath, Aug 15 '18 at 13:16
I'm going out of my mind. I ran the code like you suggested and it actually removes 9 columns instead of adding a variance column. Another time it said `Error in grouped_df_impl(data, unname(vars), drop) : Column panelid`is unknown` While there is quite obviously a column `panelid`. I have no clue what is going on today.. — Tom, Aug 15 '18 at 15:26
The `summarize`calculates the variance for each numeric column, so it doesn't add a column but replaces the values in the existing once... I'm still not 100% sure what you're looking for, so maybe an expected output would help — kath, Aug 15 '18 at 15:29
Ah okay, so I need to put in a different df.. Thank you for giving me back my sanity. This was the second time today something weird was happening haha. With regard to what I am looking for, I thought about it for a while, and I believe your answer should actually do what I want. When the dataframe is grouped by ID, I think I can just add all the variances per ID — Tom, Aug 15 '18 at 15:32
@kath Hey Kath, I am getting all NA's when I use your suggestion, or when I try: `df <- df %>% group_by(ID) %>% mutate_if(is.numeric, funs(var(., na.rm = TRUE)))` (because the dataset has NA's). Do you have any idea why? Also, is there a vignette available for this functionality? I have not been able to find it. — Tom, Aug 20 '18 at 16:34
`my_data %>% group_by(ID) %>% mutate_if(is.numeric, var, na.rm = T)` still works for me even with missing variables. The only reference I have is SO and https://dplyr.tidyverse.org/reference/summarise_all.html — kath, Aug 20 '18 at 17:06

score 0 · Answer 1 · answered Aug 15 '18 at 12:53

If I understand correctly, you want to calculate the variance of the numeric variables per year?

df %>%
  arrange(ID, Year) %>% 
  group_by(Year) %>%
  mutate_if(is.numeric, funs(d = var(.)))

        Country Ccode  Year Happiness Power    ID Happiness_d Power_d  ID_d
  <fct>   <fct> <int>     <int> <int> <int>       <dbl>   <dbl> <dbl>
1 France  FR     2000      1000  1000     1      40000. 160000.  1.67
2 France  FR     2001      1200  1400     1      80000. 320000.  1.67
3 France  FR     2000      1400  1800     2      40000. 160000.  1.67
4 France  FR     2001      1600  2200     2      80000. 320000.  1.67
5 UK      UK     2000      1000  1000     3      40000. 160000.  1.67
6 UK      UK     2001      1000  1000     3      80000. 320000.  1.67
7 UK      UK     2000      1000  1000     4      40000. 160000.  1.67
8 UK      UK     2001      1000  1000     4      80000. 320000.  1.67

I find it quite hard to explain but I would rather like to know if a variable changes over time. Is it elastic, or will it not change over time (fixed effect). Does that clarify anything? — Tom, Aug 15 '18 at 13:06

Checking the variance (or elasticity) of a variable over time in panel data

1 Answers1