Spark - java IOException :Failed to create local dir in /tmp/blockmgr*

Question

I was trying to run a long running Spark Job. After few hours of execution, I get exception below :

Caused by: java.io.IOException: Failed to create local dir in /tmp/blockmgr-bb765fd4-361f-4ee4-a6ef-adc547d8d838/28

Tried to get around it by checking:

Permission issue in /tmp dir. The spark server is not running as root. but /tmp dir should be writable to all users.
/tmp Dir has enough space.

Did you solve this issue?? – James Wierzba Feb 01 '19 at 02:41 — James Wierzba, Feb 01 '19 at 02:41

score 4 · Accepted Answer · answered Dec 20 '16 at 09:11

4

Assuming that you are working with several nodes, you'll need to check every node participate in the spark operation (master/driver + slaves/nodes/workers).

Please confirm that each worker/node have enough disk space (especially check /tmp folder), and right permissions.

answered Dec 20 '16 at 09:11

Yaron

10,166
9
45
65

2

space is available on all nodes and /tmp is readable, writable. But still i get this issue in zeppelin.. – Prasanth Jaya Aug 08 '17 at 09:31
Did you figure out what the problem was? I have the same one now. – Andrey Cheptsov Apr 11 '18 at 08:35
@AndreyCheptsov - did you confirm that you have enough disk space as described in my answer? – Yaron Apr 11 '18 at 08:37
1

@Yaron yes, permissions and space were OK. What helped is a restart of the cluster. – Andrey Cheptsov Apr 12 '18 at 09:37
@Yaron Can you clarify what the "right permissions" are? I'm seeing this issue on AWS EMR (AWS managed spark cluster), and /tmp is actually an external file mount – James Wierzba Feb 01 '19 at 02:23
Permissions __look__ fine on all of my spark nodes (master and worker), and plenty of disk space...I am lost at what to do – James Wierzba Feb 01 '19 at 02:30

Ladislav Vrbsky · Answer 2 · 2019-04-25T20:58:08.540

Edit: The answer below did not eventually solve my case. It's because some subfolders spark (or some of its dependencies) was able to create, yet not all of them. The frequent necessity of creation of such paths would make any project unviable. Therefore I ran Spark (PySpark in my case) as an Administrator, which solved the case. So in the end it is probably a permission issue afterall.

Original answer:
I solved the same problem I had on my local Windows machine (not a cluster). Since there was no problem with permissions, I created the dir that Spark was failing to create, i.e. I created the following folder as a local user and did not need to change any permissions on that folder.

C:\Users\<username>\AppData\Local\Temp\blockmgr-97439a5f-45b0-4257-a773-2b7650d17142

Mohan Kumar Kannan · Answer 3 · 2019-05-23T01:52:11.183

0

After verifying all the permissions and user access.

I got the same issue when building the components in Talend studio and it resolved by providing the correct "/" in spark scratch directory (temp directory) in spark Configuration tab. This is required when building the jar in windows and running in Linux cluster.

edited May 23 '19 at 01:52

answered May 23 '19 at 01:37

Mohan Kumar Kannan

21
4

Spark - java IOException :Failed to create local dir in /tmp/blockmgr*

3 Answers3

Linked