Is there any way to programmatically access the mechanism used in Databricks to assign Delta table identity columns, so that you can assign the values to a Dataframe pre-insert? Is this available in an io.delta class anywhere or is it something that happens internally in Databricks on insert?
Asked
Active
Viewed 72 times
0
wrschneider
- 17,913
- 16
- 96
- 176
-
Here's a Spark PR for this: https://github.com/apache/spark/pull/38823 – Kombajn zbożowy Jun 03 '23 at 21:25
-
@Kombajnzbożowy the PR shows how Spark would handle `generated always as ...` in general but I'm looking for the `identity` (ID generation) transformation specifically. – wrschneider Jun 04 '23 at 21:50