R weekly averaging

Question

I have a set of daily values of gridded sea surface temperature for 34 years (12418 daily files x 4248 points) and pretend to calculate weekly values. I have almost succeeded following this post https://stackoverflow.com/a/15102394/709777. But there is some disagreement between dates and weeks. I can't find the point and I want to be sure I'm getting the right dates to calculate the weekly average.

I use this piece of my R script to read daily data and build a big data frame that contains all daily values from a single point in a column (12418 rows/days by 4248 columns/temperature)

# Paths
ruta_datos_diarios<-"/home/meteo/PROJECTES/VERSUS/DATA/SST/CSV/"
ruta_files<-"/home/meteo/PROJECTES/VERSUS/SCRIPTS/CLUSTER/FILES/"
ruta_eixida<-"/home/meteo/PROJECTES/VERSUS/OUTPUT/DATA/SEMANAL/"

# List of daily files
files <- list.files(path = ruta_datos_diarios, pattern = "SST-diaria-MED")

output <- matrix(ncol=4248, nrow=length(files))
fechas <- matrix(ncol=1, nrow=length(files))

for (i in 1:length(files)){
  # read data
  datos<-read.csv(paste0(ruta_datos_diarios,files[i],sep=""),header=TRUE,na.strings = "NA")
  datos<-datos[complete.cases(datos),]

  # Extract dates from daily file names
  yyyy<-substr(files[i],16,19)
  mm<-substr(files[i],20,21)
  dd<-substr(files[i],22,23)
  dates[i,]<-paste0(yyyy,"-",mm,"-",dd,sep="")

  output[i,]<-t(datos$sst)
}

datos.df<-as.data.frame(output)

# Build a dataframe with the dates  (day, week and year)
fechas<-as.data.frame(fechas)
fechas$V1<-as.Date(fechas$V1)
fechas$Week <- week(fechas$V1)
fechas$Year <- year(fechas$V1)

# Extract day of the week (Saturday = 6)
fechas$Week_Day <- as.numeric(format(fechas$V1, format='%w'))
# Adjust end-of-week date (first saturday from the original Date)
fechas$End_of_Week <- fechas$V1 + (6 - fechas$Week_Day)

# new dataframe from End_of_Week
fechas.semana<-fechas[!duplicated(fechas$End_of_Week),]
fechas.semana<-as.data.frame(fechas.semana)

colnames(fechas)<-c("Day","Week","Year","Week_Day","End_of_Week")
colnames(fechas.semana)<-c("Day","Week","Year","Week_Day","End_of_Week")

This is how I read my data and dates. To keep a short example I've saved a subset of the dataframe in this file temp-sst.csv (1000 obs. of 10 variables, including "Day","Week","Year","Week_Day","End_of_Week").

sst.dat <- read.csv("temp-dat.csv",header=TRUE)

# Join dates and SST values
sst.dat <- cbind(fechas, sst.dat)

# Build new dates data frame
fechas<-as.data.frame(sst.dat$Day)
colnames(fechas)<-c("Day")
fechas$Day<-as.Date(fechas$Day)
fechas$Week <- week(fechas$Day)
fechas$Year <- year(fechas$Day)
# Extract day of the week (Saturday = 6)
fechas$Week_Day <- as.numeric(format(fechas$Day, format='%w'))
# Adjust end-of-week date (first saturday from the original Date)
fechas$End_of_Week <- fechas$Day + (6 - fechas$Week_Day)

fechas.semana<-fechas[!duplicated(fechas$End_of_Week),]
fechas.semana<-as.data.frame(fechas.semana)

colnames(fechas)<-c("Day","Week","Year","Week_Day","End_of_Week")
colnames(fechas.semana)<-c("Day","Week","Year","Week_Day","End_of_Week")

# Weekly aggregation function from the referred post
media.semanal <- function(x, column){
  a<-aggregate(x[,column]~End_of_Week+Year, FUN=mean, data=x, na.rm=TRUE)
  colnames(a)<-c("End_of_Week","Year","SSTmean")
  return(a)
}

# Matrix to be populated by weekly function
SST.mat<-matrix(nrow=nrow(fechas.semana), ncol=length(sst.dat)-5)  # 5 son las columnas de fecha

for (j in 6:length(sst.dat)){   # comienza en 6 para evitar las columnas de fecha
b<-media.semanal(sst.dat,j)
SST.mat[,j-5]<-b$SSTmean
}

But here comes the problem. "b" dataframe from the loop has 145 rows while SST.mat and fechas.semana have only 144. I haven't found the point where this disagreement comes.

Any help would be greatly appreciated, I'm stuck here. Thanks

"_To keep a short example_" - instead of posting a link to a 1000*10 file on dropbox, you should provide a _minimal_, self-contained example. — Henrik, Dec 02 '16 at 14:47

Hack-R · Accepted Answer · 2016-12-02T15:48:12.070

You have a duplicate in one vale of b$End_of_Week.

First I noticed that there was no difference in the set membership:

setdiff(as.character(b$End_of_Week),as.character(fechas.semana$End_of_Week))

character(0)

Then I realized that had to be because of a duplicate and confirmed it like this:

table(table(as.character(b$End_of_Week))>1)

143    1 
FALSE  TRUE

Looking at the table shows the dupe is 1983-01-01.

It seems the root cause is that you aggregate by End_of_Week + Year where Year is unnecessary, since End_of_Week has the year in it as well and if you only aggregate by End_of_Week you get 144 instead of 145.

# Weekly aggregation function from the referred post
media.semanal <- function(x, column){
  a<-aggregate(x[,column]~End_of_Week, FUN=mean, data=x, na.rm=TRUE)
  colnames(a)<-c("End_of_Week","SSTmean")
  return(a)
}

# Matrix to be populated by weekly function
SST.mat<-matrix(nrow=nrow(fechas.semana), ncol=length(sst.dat)-5)  # 5 son las columnas de fecha

for (j in 6:length(sst.dat)){   # comienza en 6 para evitar las columnas de fecha
  b<-media.semanal(sst.dat,j)
  SST.mat[,j-5]<-b$SSTmean
}
dim(b)

144 2

R weekly averaging

1 Answers1