Is there a way to find out DLT pipeline run status like if pipeline failed or succeeded ?
I was looking into the API https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-api-guide.html#pipelinestateinfo
State in below doesn't have failed status
I'm looking to have report which shows pipeline run info, expectation info ( I was able to get this from event log ) .
UpdateStateInfo seems what you're after, doesn't it?
The current state of a pipeline update.
where state is described as follows:
The state of the update. One of QUEUED, CREATED, WAITING_FOR_RESOURCES, INITIALIZING, RESETTING, SETTING_UP_TABLES, RUNNING, STOPPING, COMPLETED, FAILED, or CANCELED.
I use Delta Live Tables CLI (aka databricks pipelines) to interact with Delta Live Tables API. Use --debug flag to find out what it does under the covers.
$ databricks pipelines get --pipeline-id 960da65b-c9df-4cb9-9456-1005ffe103a9 | jq '.latest_updates'
[
{
"update_id": "967094d6-6751-4d4c-94ea-40950a9e7f8e",
"state": "FAILED",
"creation_time": "2023-03-16T17:43:54.278Z"
},
{
"update_id": "2187cb69-8161-4013-816f-1be6a1a6728b",
"state": "COMPLETED",
"creation_time": "2023-03-10T15:02:08.598Z"
},
{
"update_id": "a8893445-aedd-4af8-a42b-80e64b43ef7a",
"state": "COMPLETED",
"creation_time": "2023-03-10T11:38:54.316Z"
},
{
"update_id": "509264a4-5589-4d40-a940-5859fe37142c",
"state": "COMPLETED",
"creation_time": "2023-03-10T10:43:26.921Z"
},
{
"update_id": "262e4c01-b563-4d8a-913a-a4c25e927682",
"state": "COMPLETED",
"creation_time": "2023-03-09T16:54:49.801Z"
}
]