encryption decryption of streaming delta table in azure databricks

Question

My goal: my goal is to encrypt and decrypt streaming delta table in azure databricks in python.

I am trying to get the solution for encryption and decryption of streaming delta table in python so far I am able to achieve column level encryption however can I encrypt the entire table at once instead of encrypting column one by one please provide me with the relevant solution for the encryption and decryption of entire table

from pyspark.sql.functions import udf, lit, md5
from pyspark.sql.types import StringType
# Register UDF's
encrypt = udf(encrypt_val, StringType())
decrypt = udf(decrypt_val, StringType())
# Fetch key from secrets
# encryptionKey = dbutils.preview.secret.get(scope = "encrypt", key = "fernetkey")
encryptionKey = key
encryptionKey2 = key2

# Encrypt the data 
df = spark.table("EncryptTest")
encrypted = df.withColumn("ssn", encrypt("ssn",lit(encryptionKey)))\
              .withColumn("Address", encrypt("Address",lit(encryptionKey2)))
display(encrypted)
#Save encrypted data 
encrypted.write.format("delta").mode("overwrite").option("overwriteSchema", "true").saveAsTable("Encryption_Test_Table")

decrypted = encrypted.withColumn("ssn", decrypt("ssn",lit(encryptionKey)))\
              .withColumn("Address", decrypt("Address",lit(encryptionKey2)))
display(decrypted)

Answer 1

Step 1: Creating a table

Step 2: Inserting a record

Step 3: Installation and create a fernet key

Step 4: Example for fernet

Step 5: Create UDFs in encrypting a value and decrypting a value

Step 6: Using AzureKey valut and store it in secrets

encryptionKey = dbutils.secrets.get(scope = "test", key = "fernet") Refer below link

https://hevodata.com/learn/databricks-secret/#65

Step 7: Use the UDF in a dataframe to encrypt a column

Step 8: Use the UDF in a data frame to decrypted a column