0

I'm running a MapR Community Edition Hadoop cluster (M3).

Unfortunately, the HiveServer2 service crashes and, according the log file in /opt/mapr/hive/hive-0.13/logs/mapr/hive.log, there's a problem with permissions on the scratch directory:

2015-02-24 21:21:08,187 WARN  [main]: server.HiveServer2 (HiveServer2.java:init(74)) - Failed to create/change scratchdir permissions to 777: Could not create FileClient java.io.IOException: Could not create FileClient

I checked the settings for the scratch directory using hive -e 'set;' | grep scratch:

hive.exec.scratchdir=/user/mapr/tmp/hive/
hive.scratch.dir.permission=700

I notice that hive.scratch.dir.permission is set to 700 and the error message suggests that it wants to change this to 777. However, according to the filesystem, /mapr/my.cluster.com/user/mapr/tmp has 777 permissions and belongs to the mapr user.

mapr@hadoop01:/mapr/my.cluster.com/user/mapr/tmp$ ls -al
total 2
drwxr-xr-x  3 mapr mapr  1 Feb 22 10:39 .
drwxr-xr-x  5 mapr mapr  3 Feb 24 08:40 ..
drwxrwxrwx 56 mapr mapr 54 Feb 23 10:20 hive

Judging by the filesystem permissions, I would expect the mapr user to do whatever it wants with this folder and so don't understand the error message.

I'm curious to know if anyone's seen this before and, if so, how did you fix it?

Update:

I had a look at the source code, and notice some relevant comments just prior to the warning:

// When impersonation is enabled, we need to have "777" permission on root scratchdir, because
// query specific scratch directories under root scratchdir are created by impersonated user and
// if permissions are not "777" the query fails with permission denied error.

I added set the following properties in hive-site.xml:

<property>
  <name>hive.scratch.dir.permission</name>
  <value>777</value>
</property>

<property>
  <name>hive.exec.scratchdir</name>
  <value>/tmp/hive/</value>
</property>

... and created the /tmp/hive/ folder in HDFS with 777 permissions:

mapr@hadoop01:~$ hadoop fs -ls -d /tmp/hive
drwxrwxrwx   - mapr mapr          0 2015-02-27 08:38 /tmp/hive

Although this looked promising, I still got the same warning in hive.log.

Alex Woolford
  • 4,433
  • 11
  • 47
  • 80

1 Answers1

0

Update the permission of your /tmp/hive HDFS directory to set it to 777:

  hadoop fs -chmod 777 /tmp/hive

Or remove /tmp/hive , temporary files will be created anyway even when you delete them.

hadoop fs -rm -r /tmp/hive;  
rm -rf /tmp/hive
Yosser Abdellatif Goupil
  • 789
  • 3
  • 10
  • 26
  • Thanks @Yosr. I appreciate the advice. The `/tmp/hive` directory didn't exist in HDFS (probably because MapR uses slightly different paths). I added it anyway and did a `hadoop fs -chmod 777 /tmp/hive`. Unfortunately, it didn't resolve the issue. – Alex Woolford Feb 27 '15 at 19:37