0

Is there away to restrict access to a Delta Table based on a process or client id ?

Here is my scenario:

I have a streaming job that writes to a delta table, and sometimes the job fails due to concurrency issues or merge collision that are triggered by manual actions done by an Data Engineer.

My idea is that when somebody is trying to do a merge that is not the granted (Streaming app client) then the "Table" should not allow it unless the Streaming Job is paused or stopped!

Is there anything like this i can implement

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Up_One
  • 5,213
  • 3
  • 33
  • 65

1 Answers1

1

If you are using Databricks and your streaming job is scheduled using Databricks Workflows (previously known as Jobs), you can query the jobs api server first to check if the job is currently active by listing the runs of the streaming job and passing job_id and active_only=true as the payload of the request. If the response code is 200 and the response body is blank this will mean that the job is not running at the moment.

Vikas Saxena
  • 1,073
  • 1
  • 12
  • 21