Sparklyr read database table to distributed DF

Question

Hi I am trying to figure out if there is a way to directly read a DB table to a sparkR dataframe. I have rstudio installed on an EMR cluster which has my hive metastore on it.

I know I can do the following:

library(sparklyr)
library(dplyr)
sc <- spark_connect(master = "local")
library(DBI)
query <- "select * from schema.table"
result <- dbGetQuery(sc, query) 
result_t <- copy_to(sc,result)

but is there a way to query directly into result_t?

have you tried `result_t <- tbl(sc, "schema.table")`? – kevinykuo Dec 01 '17 at 16:00 — kevinykuo, Dec 01 '17 at 16:00
yeah i did try that and it worked, sorry, forgot to update – user295944 Dec 04 '17 at 23:36 — user295944, Dec 04 '17 at 23:36

score 2 · Accepted Answer · answered Dec 04 '17 at 23:37

2

Like @kevinykuo suggested,

result_t <- tbl(sc, "schema.table")

answered Dec 04 '17 at 23:37

user295944

273
4
17

I found result_t <- dbReadTable(sc, "schema.table") as a faster option, but I am new to spark – James Crumpler Mar 10 '23 at 14:04

Sparklyr read database table to distributed DF

1 Answers1