0

In Synapse Delta tables are great for ETL because they allow merge statements on parquet files. I was wondering how to use delta table to our advantage for reading the data as well when we load Silver to Gold. is there a way, in Synapse, to read the 'latest' version of each key?

The link below is for an approach for Azure Databricks. I could not get it to work in a PySpark notebook. I was wondering if there is a similar approach for Synapse Delta

How to fetch the latest version number of a delta table

david
  • 7
  • 2
  • When you say ‘could not get it to work’ did you have a specific error? Ensure your Spark pool is using 3.3 and you created the table using the DELTA syntax. Delta is the default format in Databricks so it’s not always used explicitly in scripts. It does have to be explicit for a Synapse Notebook. – wBob Mar 28 '23 at 09:29

1 Answers1

0

Yes, I agree with wBob,

I reproduce the same thing in the environment with the azure synapse Delta table.

Try this Code to get latest version of a record:

from pyspark.sql import SparkSession

df1 = spark.read \
    .format("delta") \
    .option("startingVersion", "latest") \
    .option("readChangeFeed", "true") \
    .table('sample_table')

df1.show()

enter image description here

B. B. Naga Sai Vamsi
  • 2,386
  • 2
  • 3
  • 11