1

I am new to Azure and Spark and request your help on writing the exception handling code for the below scenario.

I have written HQL scripts (say hql1, hql2, hql3) in 3 different notebooks and calling them all on one master notebook (hql-master) as,

val df_tab1 = runQueryForTable("hql1", spark)
val df_tab2 = runQueryForTable("hql2", spark)

Now I have the output of HQL scripts stored as dataframe and I have to write exception handling on master notebook where if the master notebook has successfully executed all the dataframes (df1_tab, df2_tab), a success status should get inserted into the synapse table job_status.

Else if there was any error/exception during the execution of master notebook/dataframe, then that error message should be captured and a failure status should get inserted into the synapse table.

I already have the INSERT scripts for success/failure message insert. It will be really helpful if you please provide a sample code snippet through which the exception handling part can be achieved. Thank you!!

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Shyam
  • 85
  • 2
  • 11

1 Answers1

2

basically, it's just a simple try/except code, something like this:

results = {}
were_errors = False
for script_name in ['script1', 'script2', 'script3']:
  try:    
    retValue = dbutils.notebook.run(script_name)
    results[script_name] = retValue
  except Exception as e:
    results[script_name] = "Error: {e}"
    were_errors = True

if were_errors:
  log failure # you can use data from results variable
else:
  log success
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • Thanks @Alex. Can you please share me the answer in scala format as I'm writing my code in scala ? Also, I've already run the hql scripts before the exception handling as val df_tab1 = runQueryForTable("hql_script_1", spark) & val df_tab2 = runQueryForTable("hql_script_2", spark).So retValue = dbutils.. will again execute them which is not necessary as I am already holding the output of hql1 and hql2 as dataframe (df_tab1, df_tab2). I just need to check whether those dataframes are successfully executed (or) not and based on the result of df_tab1, df_tab2, I should write exception handling. – Shyam Mar 17 '21 at 10:07
  • scala code would be the same, just change syntax to `try { execute logic } catch { case e: Exception => ... }` – Alex Ott Mar 17 '21 at 11:13
  • Here how to raise an exception. if raised where exactly the exception will be sent , can i simply raise a string or does it have to Exception instance – Surender Raja Jun 17 '22 at 04:29