1

I'm using Spark-2.4, I have a Kerberos enabled cluster where I'm trying to run a query via the spark-sql shell.

The simplified setup basically looks like this: spark-sql shell running on one host in a Yarn cluster -> external hive-metastore running one host -> S3 to store table data.

When I launch the spark-sql shell with DEBUG logging enabled, this is what I see in the logs: 

> bin/spark-sql --proxy-user proxy_user

...
DEBUG HiveDelegationTokenProvider: Getting Hive delegation token for proxy_user against hive/_HOST@REALM.COM at thrift://hive-metastore:9083
DEBUG UserGroupInformation: PrivilegedAction as:spark/spark_host@REALM.COM (auth:KERBEROS) from:org.apache.spark.deploy.security.HiveDelegationTokenProvider.doAsRealUser(HiveDelegationTokenProvider.scala:130)

This means that Spark made a call to fetch the delegation token from the Hive metastore and then added it to the list of credentials for the UGI. This is the piece of code in Spark which does that. I also verified in the metastore logs that the get_delegation_token() call was being made.

Now when I run a simple query like create table test_table (id int) location "s3://some/prefix"; I get hit with an AWS credentials error. I modified the hive metastore code and added this right before the file system in Hadoop is initialized (org/apache/hadoop/hive/metastore/Warehouse.java): 

 public static FileSystem getFs(Path f, Configuration conf) throws MetaException {
...
    try {
      // get the current user 
      UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
      LOG.info("UGI information: " + ugi);
      Collection<Token<? extends TokenIdentifier>> tokens = ugi.getCredentials().getAllTokens();
      // print all the tokens it has 
      for(Token token : tokens) {
        LOG.info(token);
      }
    } catch (IOException e) {
      e.printStackTrace();
    }
...
}

In the metastore logs, this does print the correct UGI information: 

UGI information: proxy_user (auth:PROXY) via hive/hive-metastore@REALM.COM (auth:KERBEROS)

but there are no tokens present in the UGI. Looks like Spark code adds it with the alias hive.server2.delegation.token but I don't see it in the UGI. This makes me suspect that somehow the UGI scope is isolated and not being shared between spark-sql and hive metastore. How do I go about solving this?

sbrk
  • 1,338
  • 1
  • 17
  • 25
  • please accept my condolences for having looked at the UGI code. There are some things humanity was Not Meant To See. UGI and the kerberos layers underneath qualify. – stevel Apr 23 '20 at 16:31

1 Answers1

1

Spark is not picking up your Kerberos identity -it asks each FS to issue some "delegation token" which lets the caller interact with that service and that service alone. This is more restricted and so more secure.

The problem here is that spark collects delegation tokens from every filesystem which can issue them -and as your S3 connector isn't issuing any, nothing is coming down.

Now, Apache Hadoop 3.3.0's S3A connector can be set to issue your AWS credentials inside a delegation token, or, for bonus security, ask AWS for session credentials and send only those over. But (a) you need a spark build with those dependencies, and (b) Hive needs to be using those credentials to talk to S3.

stevel
  • 12,567
  • 1
  • 39
  • 50