4

I am trying to split column values separated by comma(,) into new rows based on id's. I know how to do this in R using dplyr and tidyr. But I am looking to solve same problem in sparklyr.

id <- c(1,1,1,1,1,2,2,2,3,3,3)
name <- c("A,B,C","B,F","C","D,R,P","E","A,Q,W","B,J","C","D,M","E,X","F,E")
value <- c("1,2,3","2,4,43,2","3,1,2,3","1","1,2","26,6,7","3,3,4","1","1,12","2,3,3","3")
dt <- data.frame(id,name,value)

R solution:

separate_rows(dt, name, sep=",") %>%
  separate_rows(value, sep=",")

Desired Output from sparkframe(sparklyr package)-

> final_result
   id name value
1   1    A     1
2   1    A     2
3   1    A     3
4   1    B     1
5   1    B     2
6   1    B     3
7   1    C     1
8   1    C     2
9   1    C     3
10  1    B     2
11  1    B     4
12  1    B    43
13  1    B     2
14  1    F     2
15  1    F     4
16  1    F    43
17  1    F     2
18  1    C     3
19  1    C     1
20  1    C     2
21  1    C     3
22  1    D     1
23  1    R     1
24  1    P     1
25  1    E     1
26  1    E     2
27  2    A    26
28  2    A     6
29  2    A     7
30  2    Q    26
31  2    Q     6
32  2    Q     7
33  2    W    26
34  2    W     6
35  2    W     7
36  2    B     3
37  2    B     3
38  2    B     4
39  2    J     3
40  2    J     3
41  2    J     4
42  2    C     1
43  3    D     1
44  3    D    12
45  3    M     1
46  3    M    12
47  3    E     2
48  3    E     3
49  3    E     3
50  3    X     2
51  3    X     3
52  3    X     3
53  3    F     3
54  3    E     3

Note-

  1. I have approx 1000 columns with nested values. so, I need a function which can loop in for each column.
  2. I know we have sdf_unnest() function from package sparklyr.nested. But, I am not sure how to split strings of multiple columns and apply this function. I am quite new in sparklyr.

Any help would be much appreciated.

zero323
  • 322,348
  • 103
  • 959
  • 935
Rushabh Patel
  • 2,672
  • 13
  • 34

1 Answers1

3

You have to combine explode and split

sdt %>% 
  mutate(name = explode(split(name, ","))) %>% 
  mutate(value = explode(split(value, ",")))

# Source:   lazy query [?? x 3]
# Database: spark_connection
      id name  value
   <dbl> <chr> <chr>
 1  1.00 A     1    
 2  1.00 A     2    
 3  1.00 A     3    
 4  1.00 B     1    
 5  1.00 B     2    
 6  1.00 B     3    
 7  1.00 C     1    
 8  1.00 C     2    
 9  1.00 C     3    
10  1.00 B     2   
# ... with more rows

Please note that lateral views have be to expressed as separate subqueries, so this:

sdt %>% 
  mutate(
    name = explode(split(name, ",")),
     value = explode(split(value, ",")))

won't work

zero323
  • 322,348
  • 103
  • 959
  • 935