Maintaining persistent HDFS in Google Cloud

Question

I am having my students use bdutil to create a Google Compute Engine cluster with persistent disks and HDFS as the default filesystem. We want to have persistent disks so that the students can work on projects over a period of weeks. However, HDFS doesn’t seem usable after I redeploy the cluster.

My question is really "How do I maintain a persistent HDFS filesystems across redeployments of a cluster?"

Here is what I have tried

Everything works fine on the initial deployment, which creates the persistent disk. I create a directory with the command

$ hadoop fs -mkdir /foo
$ hadoop fs –put foo.txt /foo/foo.txt
$ hadoop fs –cat /foo/foo.txt
foo

I then delete and redeply the cluster with DELETE_ATTACHED_PDS_ON_DELETE=false and CREATE_ATTACHED_PDS_ON_DEPLOY=false to preserve the persistent disks across the redeployment

When I ssh into the redeployed cluster, I can see the file I created

$ hadoop fs –ls /foo
Found 1 items
-rw-r--r--   3 mpcs supergroup          4 2014-10-01 13:16 /foo/foo.txt

However, any attempt to access the contents of the files fails:

$ hadoop fs –cat /foo/foo.txt
cat: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Zero blocklocations for /foo/foo.txt. Name node is in safe mode

Manually turning off safe code doesn't help

$ hadoop dfsadmin -safemode leave
Safe mode is OFF
$ hadoop fs –cat /foo/foo.txt
14/10/01 13:31:20 INFO hdfs.DFSClient: No node available for: blk_2908405986797013125_1002 file=/foo/foo.txt
14/10/01 13:31:20 INFO hdfs.DFSClient: Could not obtain blk_2908405986797013125_1002 from any node: java.io.IOException: No live nodes contain current block. Will get new block locations from namenode and retry...
*etc*

Any suggestions on how to create an HDFS store that can persist across redeploying a cluster would be greatly appreciated

Thanks,

Mike

score 3 · Accepted Answer · answered Oct 01 '14 at 22:37

Thanks for the detailed report! Indeed, it appears you've discovered a bug introduced a few releases back where bdutil-0.35.2/libexec/configure_hadoop.sh unfortunately clobbers the directory permissions on the datanode data dirs with a slightly overly-permissive setting of 775 vs the expected default of 755 (for Hadoop 1.2.1) or 700 (for Hadoop 2.4.1). This causes the datanodes to never recover on the reboot, printing:

2014-10-01 20:37:59,810 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in dfs.data.dir: Incorrect permission for /mnt/pd1/hadoop/dfs/data, expected: rwxr-xr-x, while actual: rwxrwxr-x
2014-10-01 20:37:59,811 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in dfs.data.dir are invalid.

A short-term workaround which you can run directly on the broken re-deployed cluster if you're using Hadoop 1.2.1 is to run:

./bdutil run_command -t all -- "chmod 755 /hadoop/dfs/data"
./bdutil run_command -t master -- "sudo -u hadoop /home/hadoop/hadoop-install/bin/stop-dfs.sh"
./bdutil run_command -t master -- "sudo -u hadoop /home/hadoop/hadoop-install/bin/start-dfs.sh"

As it turns out, Hadoop 2 actually fixed this by having the DataNode just go ahead and set the permissions that it requires if it doesn't already match:

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-common/2.2.0/org/apache/hadoop/util/DiskChecker.java#130

120  public static void More ...mkdirsWithExistsAndPermissionCheck(
121      LocalFileSystem localFS, Path dir, FsPermission expected)
122      throws IOException {
123    File directory = localFS.pathToFile(dir);
124    boolean created = false;
125
126    if (!directory.exists())
127      created = mkdirsWithExistsCheck(directory);
128
129    if (created || !localFS.getFileStatus(dir).getPermission().equals(expected))
130        localFS.setPermission(dir, expected);
131  }

Whereas Hadoop 1 just bails out:

https://github.com/apache/hadoop/blob/release-1.2.1/src/core/org/apache/hadoop/util/DiskChecker.java#L106

private static void checkPermission(Path dir, 
                                   FsPermission expected, FsPermission actual) 
throws IOException {
  // Check for permissions
  if (!actual.equals(expected)) {
    throw new IOException("Incorrect permission for " + dir + 
                          ", expected: " + expected + ", while actual: " + 
                          actual);
  }

}

We'll be fixing this in the next bdutil release with explicit settings for dfs.datanode.data.dir.perm but in the meantime, you can also patch the following hard-coded workaround with patch bdutil-0.35.2/libexec/configure_hdfs.sh tmpfix.diff:

43a44,46
>   # Make sure the data dirs have the expected permissions.
>   chmod 755 ${HDFS_DATA_DIRS}
>

Alternatively, if you're using bdutil with -e hadoop2_env.sh, then HDFS persistence should already work without any further modifications.

Maintaining persistent HDFS in Google Cloud

1 Answers1