Spark error when trying to load new View from Power BI

Question

I am using Spark cli service in Power Bi, it throwing the below error trying to load View from spark.

DataSource.Error: ODBC: ERROR [HY000] [Microsoft][Hardy] (35) Error from server: error code: '0' error message: 'org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2891.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2891.0 (TID 1227) (ip-XXX-XXX-XXX.compute.internal executor driver): java.io.FileNotFoundException: /tmp/blockmgr-51aefd41-4d64-49fb-93d0-10deca23cad3/03/temp_shuffle_39d969f9-b0af-4d4a-b476-b264eb18fd1c (No such file or directory) at java.io.FileOutputStream.open0(Native Method)

The view returns data in spark-sql cli:

New Tables are working fine in the refresh, the error happens only with the views.

I also verify the disk space, it is not full.

Answer 1

It seems it was bug in spark-core https://issues.apache.org/jira/browse/SPARK-36500

Others have similar issues: Spark - java IOException :Failed to create local dir in /tmp/blockmgr*

After a research, in my case the solution is to increase the executor memory. In the spark-defaults.conf

spark.executor.memory 5g

Then restart