1

i am new to spark and trying pyspark and spark-shell in both cases i am getting this error pyspark gives following error

ERROR storage.DiskBlockManager: Failed to create local dir in /tmp. Ignoring this directory. java.io.IOException: Failed to create a temp directory (under /tmp) after 10 attempts!

can any one please help me on this.

Ramesh Maharjan
  • 41,071
  • 6
  • 69
  • 97
akash sharma
  • 15
  • 1
  • 4
  • Did you check the permissions of the folder ? and if there is enough space in /tmp folder ? – Ahmed May 30 '18 at 10:13
  • actually permssions are good and space also looks good, like 42% free so i am confused. is there any way we can change /redirect the working folder path – akash sharma May 30 '18 at 10:35
  • Try with *spark.local.dir* conf : `export SPARK_JAVA_OPTS="-Dspark.local.dir=/newFolder..."` – Ahmed May 30 '18 at 12:32
  • Does this answer your question? [Spark - java IOException :Failed to create local dir in /tmp/blockmgr\*](https://stackoverflow.com/questions/41238121/spark-java-ioexception-failed-to-create-local-dir-in-tmp-blockmgr) – arghtype Nov 22 '19 at 23:21

1 Answers1

0

If there is enough space left on your folder, try

chmod ugo+rwx /tmp

My tmp folder is somehow deleted and after created a new one I fixed it by this

Litchy
  • 623
  • 7
  • 23