How to extract column value to compare with rlike in spark dataframe

Question

I have the below code in spark 2.3 .It shows error since rlike can only evaluate a string but not a column field, So I want to extract the value in corresponding col(unique_handle_regex) column as string

val df3 = df1.join(df2).select("*").where(col(unique_handle) rlike col(unique_handle_regex))

please post your sample table and expected output. – Addy Jul 08 '20 at 16:31 — Addy, Jul 08 '20 at 16:31

Srinivas · Accepted Answer · 2020-07-09T03:36:54.673

1

Try below code.

val whereExpr = Seq("unique_handle","unique_handle_regex").mkString(" rlike ")
val df3 = df1.join(df2).select("*").where(whereExpr)

edited Jul 09 '20 at 03:36

answered Jul 08 '20 at 16:32

Srinivas

8,957
2
12
26

Hi Srinivas, thank you for your response.your solution works fine.You have hard-coded the column name in the query right is there any other solution without hard-coding it @Srinivas – minnu Jul 09 '20 at 03:30
Thanks .. is this solution not solved your issue ??.. if it solved please accept this answer .. – Srinivas Jul 09 '20 at 04:29

How to extract column value to compare with rlike in spark dataframe

1 Answers1