0

Why does the following MSR code not replace the original column "Var1"?

rxDataStep(inData = input_xdf, outFile = input_xdf, overwrite = TRUE,
       transforms = list(Var1 = as.numeric(Var1)),
       transformVars = c("Var1")
       )
JimBoy
  • 597
  • 8
  • 18

2 Answers2

1

MSR doesn't allow you to overwrite a variable in place with a different variable type.

You have two options: Write to a different variable or write to a different file. I have added a bit of code that shows that both solutions work as stated in MRS 9.0.1. As stated in the comments, there is some point in earlier versions where this might not work. I am not totally sure where that point is, so the code should let you know.

input_xdf <- "test.xdf"
modified_xdf <- "test_out.xdf"

xdf_data <- data.frame(Var1 = as.character(1:10),
                       Var2 = 2:11,
                       stringsAsFactors = FALSE)

rxDataStep(inData = xdf_data,
           outFile = input_xdf,
           rowsPerRead = 5,
           overwrite = TRUE)

rxDataStep(inData = input_xdf, 
           outFile = input_xdf, 
           overwrite = TRUE,
           transforms = list(Var1b = as.numeric(Var1)),
           transformVars = c("Var1")
)

rxGetInfo(input_xdf, getVarInfo = TRUE, numRows = 5)

rxDataStep(inData = input_xdf, 
           outFile = modified_xdf,
           transforms = list(Var1 = as.numeric(Var1)),
           transformVars = c("Var1")
)

rxGetInfo(modified_xdf, getVarInfo = TRUE, numRows = 5)
Derek McCrae Norton
  • 844
  • 4
  • 17
  • 1
    Actually, your second option won't work (one of those annoying warts in RevoScaleR). I don't think there's any way to change the type of an existing variable, even if you write to a different file. – Hong Ooi Dec 17 '16 at 12:48
  • @HongOoi, That may have been true before, but is not true now. I tested before I posted. Both solutions work. I'll add some code to make it a working example. – Derek McCrae Norton Dec 19 '16 at 19:13
  • I'm getting mixed results. Email sent. – Hong Ooi Dec 20 '16 at 09:36
1

At the moment, RevoScaleR doesn't support changing the type of a variable in an xdf file (even if you write to a different file). The way to do it is to create a new variable, drop the old, and then rename the new variable to the old name.

I would suggest doing this with a transformFunc (see ?rxTransform for more information), so that you can create the new variable and drop the old, all in one step:

rxDataStep(inXdf, outXdf, transformFunc=function(varlst) {
    varlst$Var1b <- as.numeric(varlst$Var1)
    varlst$Var1 <- NULL
    varlst
}, transformVars="Var1")

# this is fast as it only modifies the xdf metadata, not the data itself
names(outXdf)[names(outXdf) == "Var1b"] <- "Var1"
Hong Ooi
  • 56,353
  • 13
  • 134
  • 187
  • What's the reason that RevoScaleR doesn't allow to manipulate an existing variable in place? – JimBoy Dec 18 '16 at 09:23
  • I'm not in the engineering team so I can't give a definitive answer, but it would be for performance reasons. – Hong Ooi Dec 18 '16 at 15:47