0

Within a bash script, I am invoking yarn application command in order to get the current applications running on a Cloudera Hadoop cluster secured by Kerberos. In case my application is not running, it is necessary to restart it:

spark_rtp_app_array=( $(yarn application --list -appTypes SPARK -appStates ACCEPTED,RUNNING | awk -F "\t" ' /'my_user'/ && /'my_app'/ {print $1}') )

Whenever the Kerberos ticket has ended I need to invoke kinit command, in order to renew that ticket before calling yarn application --list:

kinit -kt my_keytab_file.keytab my_kerberos_user

Otherwise, I could end with an authentication error which keeps repeating in an undefinite way with the following traces:

19/02/13 15:00:22 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS in\

itiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

19/02/13 15:00:22 WARN security.UserGroupInformation: PriviledgedActionException as:my_kerberos_user (auth:KERBEROS) cause:java.io\

.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechan\

ism level: Failed to find any Kerberos tgt)]
[...]

Is there any way of setting a maximum number of connection retries to YARN? The bash script is being executed in a cron task, so it should not be hung in any way.

aloplop85
  • 892
  • 3
  • 16
  • 40
  • 1
    My 2 cents : try a ticket renewal first (`kinit -R`) ; on failure, try a ticket re-creation from keytab ; on failure, abort. Otherwise you can assume you have a valid ticket. – Samson Scharfrichter Feb 26 '19 at 18:19
  • Why don't you add the `kinit` to the cronjob's bash script? – tk421 Feb 26 '19 at 22:35
  • Actually the kinit call is inside the script, before the call to yarn application. I would like to know if there exists any way to prevent it being hung. I will try @SamsonScharfrichter solution. – aloplop85 Feb 27 '19 at 07:53
  • Did you inspect all properties listed in `mapred-default` and `yarn-default`, e.g. `yarn.app.mapreduce.client.job.max-retries` and `.retry-interval` ? Cf. http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml – Samson Scharfrichter Feb 27 '19 at 12:10

0 Answers0