0

I'm reading a JSON in pyspark and seeing the below issue.

Column name "change(me)" contains invalid characters, please use alias to rename it

I have tried to use withColumnRenamed but that does not seem to help

df = spark.read.option("multiline","true").json("json_file")
df = df.withColumnRenamed("change(me)", "change_me")

Here is my sample json

{
"1": {
    "task": [
        "wakeup",
        "getready"
    ]
},
"2": {
    "task": [
        "brush",
        "shower"
    ]
},
"3": {
    "task": [
        "brush",
        "shower"
    ]
},
"activites": ["standup", "play", "sitdown"],
"statuscheck": {
    "time": 60,
    "color": 1002,
    "change(me)": 9898
},
"action": ["1", "2", "3", "4"]
}

when I check for columns in my dataframe, I do not see change(me) but it still complains of invalid character

RData
  • 959
  • 1
  • 13
  • 33

0 Answers0