1

Getting knee deep with sparklyr/rsparkling, I have some spark tables with annoying column names and I would like to rename them. But I cannot seem to do it.

library(sparklyr)
library(rsparkling)
library(dplyr)
library(DBI)
sc <- spark_connect(master = "local")
iristbl <- copy_to(sc,iris, overwrite = TRUE)
colnames(iristbl) ##lists the columns
colnames(iristbl)=paste0('silly',1:5) ##i want to rename to silly names

colnames(iristbl)=paste0('silly',1:5) ##i want to rename to silly names

GIVES:

Error in dimnames(x) <- dn : 'dimnames' applied to non-array

Any suggestions?

desertnaut
  • 57,590
  • 26
  • 140
  • 166
Chris
  • 1,219
  • 2
  • 11
  • 21
  • 1
    Here `iristbl` is not a data frame, but a'pointer' to the Spark DataFrame. You need to use dplyr syntax to manipulate `iristbl`. Namely, for renaming columns you'd do something like `iristbl %>% rename(new_name=old_name)` or `iristbl %>% select(new_name=old_name)`, where `select` will keep only that column. The `rename` function is sometimes not cooperative though. – Oldřich Spáčil Nov 03 '17 at 10:54

0 Answers0