0

i am facing a strange issue , i am trying to display values of my JSON object, it works fine with select() but it dont work with selectExp(), i get a weird error, following in my implementation,

from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("JsonPractice").getOrCreate()
my_json_df = spark.range(1).selectExpr(
    """'{"sample_json":{"sample_json1":["1st_vale","2nd_val"]}}' as my_json_column""")
my_json_df.selectExpr(get_json_object(col("my_json_column"), "$.sample_json.sample_json1[1]")).show(2)
my_select_expr = get_json_object(col('my_json_column'), '$.sample_json.sample_json1')
my_json_df.selectExpr(my_select_expr).show()

I am getting following error

raise TypeError("Column is not iterable")
TypeError: Column is not iterable

noobie-php
  • 6,817
  • 15
  • 54
  • 101

1 Answers1

0

We don't need to specify col while using selectExpr

my_select_expr = "get_json_object(my_json_column, '$.sample_json.sample_json1')"
my_json_df.selectExpr(my_select_expr).show(10,False)

#or    
my_json_df.selectExpr("get_json_object(my_json_column,'$.sample_json.sample_json1')").show(10,False)
#+-----------------------------------------------------------+
#|get_json_object(my_json_column, $.sample_json.sample_json1)|
#+-----------------------------------------------------------+
#|["1st_vale","2nd_val"]                                     |
#+-----------------------------------------------------------+

UPDATE:

from pyspark.sql.functions import *
my_select_expr=get_json_object(col('my_json_column'),'$.sample_json.sample_json1')
my_json_df.select(my_select_expr).show(10,False)
#+-----------------------------------------------------------+
#|get_json_object(my_json_column, $.sample_json.sample_json1)|
#+-----------------------------------------------------------+
#|["1st_vale","2nd_val"]                                     |
#+-----------------------------------------------------------+
notNull
  • 30,258
  • 4
  • 35
  • 50
  • cant we evaluate `my_select_expr = "get_json_object(my_json_column, '$.sample_json.sample_json1')"` , without quotes? like this `my_select_expr = get_json_object(my_json_column, '$.sample_json.sample_json1')` ? – noobie-php Jul 09 '20 at 14:23
  • @noobie-php, That's possible with `.select` as select accepts `col,strings`, Please check the Updated answer! – notNull Jul 09 '20 at 14:30