1

My code used to work. Why does my code not work anymore? I updated to the newer Databricks runtime 10.2 so I had to change some earlier code to use pandas on pyspark.

# Drop customer ID for AutoML
automlDF = churn_features_df.drop(key_id)

# Write out silver-level data to autoML Delta lake
automlDF.to_delta(mode='overwrite', path=automl_silver_tbl_path)

The error I am getting is 'DataFrame' object has no attribute 'to_delta'

Climbs_lika_Spyder
  • 6,004
  • 3
  • 39
  • 53

1 Answers1

1

I was able to get it to work as expected using to_pandas_on_spark(). My working code looks like this:

# Drop customer ID for AutoML
automlDF = churn_features_df.drop(key_id).to_pandas_on_spark()

# Write out silver-level data to autoML Delta lake
automlDF.to_delta(mode='overwrite', path=automl_silver_tbl_path)
Climbs_lika_Spyder
  • 6,004
  • 3
  • 39
  • 53