Multiple date formats in tbl_spark in R. I want to change all the date formats to one format (dd-mm-yyyy)

Question

library(sparklyr)
sc <- spark_connect(master = "local")

date1 <- data.frame(DATE1 =c ("12-01-2020","2020-12-24",
  "14-May-2020","19/3/2020 14:32:15",
  "2020-09-08","05/05/2020"),
  NAME1 = c("A","B","C","D","E","F"))
tbl <- copy_to(sc, date1)

The end result date should be in 'dd-mmm-yyyy' format.

Tried this

 lubridate::parse_date_time(tbl %>% select(DATE1) , c('dmY', 'Ymd', 'dmYT'))

[1] NA NA Warning message: All formats failed to parse. No formats found.

score 1 · Accepted Answer · answered May 14 '20 at 09:55

1

This works in R, can you check if it does in sparklyr ?

as.Date(lubridate::parse_date_time(date1$DATE1, c('dmY', 'Ymd', 'dmYT')))
#[1] "2020-01-12" "2020-12-24" "2020-05-14" "2020-03-19" "2020-09-08" "2020-05-05"

answered May 14 '20 at 09:55

Ronak Shah

377,200
20
156
213

@Let'sCode and without `as.Date` works? `lubridate::parse_date_time(date1$DATE1, c('dmY', 'Ymd', 'dmYT'))` – Ronak Shah May 14 '20 at 09:57
The result is POSIXct of length 0. – Let's Code May 14 '20 at 10:02
I think in sparklyr, the '$' doesnt work for extract a column – Let's Code May 14 '20 at 10:03
@Let'sCode Try `lubridate::parse_date_time(date1 %>% pull(DATE1), c('dmY', 'Ymd', 'dmYT'))` – Ronak Shah May 14 '20 at 10:32

Multiple date formats in tbl_spark in R. I want to change all the date formats to one format (dd-mm-yyyy)

1 Answers1