0

I have the temp view being created in a loop. This temp view is used in subsequent queries. 

for row in manual_est_query_results_list:
    manual_est_query_results.createOrReplaceTempView("manual_estimates")

Sometimes, the size of manual_est_query_results_list list is around 500 rows and is resulting in 500 rows in the Spark UI, SQL Tab as shown below. The exec time is in millis but this is cluttering up the UI making it hard to navigate while debugging issues. What should I pay attention to if I want to move it out of the loop

enter image description here

Aravind Yarram
  • 78,777
  • 46
  • 231
  • 327
  • Can you elaborate on what is the usecase? Moving it out of the for loop would be much more efficient than appending `row` inside a for loop and creating a temp view inside the for loop. – Jayadeep Jayaraman Mar 16 '20 at 05:30
  • @JayadeepJayaraman that view i being queried in the loop but the query clause varies. That is pretty much about it. – Aravind Yarram Mar 16 '20 at 16:05

0 Answers0