Encrypting values in a JSON array in data frame

Asked Oct 07 '21 at 16:39

Active Oct 23 '21 at 12:09

Viewed 73 times

So we have a struct with an array in a nested struct in a JSON object, so we have column JSON then there is an array type called address in the nested struct. We are looking to encrypt some of these fields using fernet. At moment I have only got this far:

df = df.withColumn("json", col("json").withField(arrayPath, transform("json." + arrayPath, lambda x : x.withField(field, encrypt("json." + arrayPath + "." + field, lit(encryptKey))))))

Arraypath is the path to the array struct and the field we want to encrypt is in the field variable. I get the error 'TypeError: encoding without a string argument' as I can't drill down to the string value in the encrypt function call, any suggestions?

edited Oct 23 '21 at 12:09

Alex Ott

80,552
8
87
132

asked Oct 07 '21 at 16:39

T.UK

what is `encrypt` function? if that's a non-pyspark function then you must make a UDF in order to use it – pltc Oct 11 '21 at 03:50
encrypt is a udf function that takes in a string and a key – T.UK Oct 11 '21 at 08:33
is `field` nullable? – pltc Oct 11 '21 at 15:20
no, basically a way I found around this issue is to explode array and put this in another dataframe to grab values and transform – T.UK Oct 13 '21 at 08:41

Encrypting values in a JSON array in data frame

0 Answers0