1

I am using xts to convert to hourly average data. I am starting with a year's worth of 10-minute data. Some hours have one 10-minute period (such as 'UTSP' in row 229) that is NA (missing).

For such hours, I would still like the average of the data that are available, however in the output I get NA for that variable for the hour.

Other hours may have no data (all data are missing). I want these completely missing hours to return NA, but where some data exist for an hour, I want that data to be used.

Here is a reproducible example of what I've been trying:

Lines <- "date,time,UTSP,UPM10,UPM25,UPM1,UWS,UWDT,PTSP,PPM10,PPM25,PPM1,PWS,PWDT
218,2014/10/15,22:00,9.7,4.9,4.66,1.54,6,152.56,102,53.6,33.71,10.34,NA,NA
219,2014/10/15,22:10,9.3,5.1,4.57,1.61,6.4,147.56,106.4,55.1,33.92,10.47,NA,NA
220,2014/10/15,22:20,8.9,5,4.7,1.55,6.4,147.56,108.3,54.8,33.19,10.53,NA,NA
221,2014/10/15,22:30,9.7,5.3,4.93,1.62,6.8,152.56,110.3,57.4,34.97,11.14,NA,NA
222,2014/10/15,22:40,9.1,5.2,4.76,1.54,6.8,152.56,118.9,62.3,37.58,11.63,NA,NA
223,2014/10/15,22:50,9.8,5.5,5.07,1.62,6.7,152.56,120.5,61.8,36.24,11.9,NA,NA
224,2014/10/15,23:00,11.1,5.6,5.2,1.59,6.4,152.56,108.6,57.1,34.93,11.66,NA,NA
225,2014/10/15,23:10,9.8,5.4,4.89,1.63,7.3,152.56,116,59.6,35.08,11.14,NA,NA
226,2014/10/15,23:20,9.1,5,4.95,1.63,7.1,152.56,122.6,63.8,38.28,12.17,NA,NA
227,2014/10/15,23:30,9.7,5.2,4.88,1.58,7.3,147.56,88.1,46.7,29.59,9.78,NA,NA
228,2014/10/15,23:40,9.2,5.2,4.79,1.66,7.1,152.56,92.4,48.8,30.11,9.69,NA,NA
229,2014/10/15,23:50,NA,NA,NA,NA,NA,NA,89.7,48.1,30.53,9.89,NA,NA
230,2014/10/16,00:00,9.8,5.5,5.03,1.6,7,147.56,91.2,47.5,30.09,9.38,NA,NA
231,2014/10/16,00:10,9.7,5.1,4.81,1.57,7.1,152.56,91.2,47.6,29.44,9.4,NA,NA
232,2014/10/16,00:20,9.9,5.4,5.09,1.61,7.4,147.56,91.1,48.3,29.78,9.23,NA,NA
233,2014/10/16,00:30,9.8,5.4,4.82,1.62,6.9,152.56,95.7,48.6,29.47,9.8,NA,NA
234,2014/10/16,00:40,10.6,5.7,4.99,1.58,6.8,147.56,91.3,47.9,29.57,9.94,NA,NA
235,2014/10/16,00:50,10.1,5.4,4.93,1.65,7,147.56,86.3,44.9,27.9,8.93,NA,NA"
conn <- textConnection(Lines)
dframe <- read.csv(conn)
close(conn)

library(xts) 
USP_TSP.xts <- xts(dframe$UTSP,
    as.POSIXct(paste(dframe$date,dframe$time), format="%Y/%m/%d %H:%M")) 
na.exclude(USP_TSP.xts)
ep <- endpoints(USP_TSP.xts,'hours') 
period.apply(USP_TSP.xts,ep,mean)

I have also tried several variations of na.contiguous, na.omit, na.action.

My resultant output always seems to be the same (excerpt):

                           [,1]
2014-10-15 22:50:00   9.4166667
2014-10-15 23:50:00          NA
2014-10-16 00:50:00   9.9833333

... with the value for 2014-10-15 hr 23 being NA, even though there were 5 out of 6 non-missing values

Also, I am calculating all the columns separately, then combining them later. Is there an easier way - like calculating all the columns at once?

Joshua Ulrich
  • 173,410
  • 32
  • 338
  • 418
lost in R
  • 55
  • 11
  • Could you format correctly your post? – Braiam Aug 01 '15 at 12:02
  • yes...I was formatting it as you commented. It is my very first post and I am working this all out. Sorry – lost in R Aug 01 '15 at 12:11
  • Your code works fine. You just need to assign `USP_TSP.xts <- na.exclude(USP_TSP.xts)`. If you merely call `na.exclude(USP_TSP.xts)` the output without NA's is printed, but not stored in any variable. – RHertel Aug 01 '15 at 14:40

2 Answers2

2

Calling na.exclude doesn't change the USP_TSP.xts object. You would need to assign the output of na.exclude to USP_TSP.xts to achieve that.

USP_TSP.xts <- na.exclude(USP_TSP.xts)

But if you want to process all the columns in the object at once, using na.exclude is going to remove all rows that have at least one column with a missing value.

xData <- xts(dframe[,-(1:2)],
    as.POSIXct(paste(dframe$date,dframe$time), format="%Y/%m/%d %H:%M")) 
na.exclude(xData)
#      UTSP UPM10 UPM25 UPM1 UWS UWDT PTSP PPM10 PPM25 PPM1 PWS PWDT
str(na.exclude(xData))
# An 'xts' object of zero-width

Instead, you should supply na.rm=TRUE to the call to mean inside the period.apply call. If you want to process all columns at the same time, you can use colMeans.

xDataMeans <- period.apply(xData, endpoints(xData, "hours"), colMeans, na.rm=TRUE)
xDataMeans
#                         UTSP    UPM10    UPM25  UPM1      UWS     UWDT
# 2014-10-15 22:50:00 9.416667 5.166667 4.781667 1.580 6.516667 150.8933
# 2014-10-15 23:50:00 9.780000 5.280000 4.942000 1.618 7.040000 151.5600
# 2014-10-16 00:50:00 9.983333 5.416667 4.945000 1.605 7.033333 149.2267
#                          PTSP    PPM10    PPM25      PPM1 PWS PWDT
# 2014-10-15 22:50:00 111.06667 57.50000 34.93500 11.001667 NaN  NaN
# 2014-10-15 23:50:00 102.90000 54.01667 33.08667 10.721667 NaN  NaN
# 2014-10-16 00:50:00  91.13333 47.46667 29.37500  9.446667 NaN  NaN
Joshua Ulrich
  • 173,410
  • 32
  • 338
  • 418
1

Your code works fine. You just need to assign USP_TSP.xts <- na.exclude(USP_TSP.xts). If you merely call na.exclude(USP_TSP.xts), then the output without NAs is printed, but it is not stored in any variable.

USP_TSP.xts <- na.exclude(USP_TSP.xts)
ep <- endpoints(USP_TSP.xts,'hours') 
period.apply(USP_TSP.xts,ep,mean) 
#                        [,1]
#2014-10-15 22:50:00 9.416667
#2014-10-15 23:40:00 9.780000
#2014-10-16 00:50:00 9.983333

Alternatively you can use period.apply(USP_TSP.xts,ep,mean, na.rm=T) if you don't want to modify the original xts object.

RHertel
  • 23,412
  • 5
  • 38
  • 64