Unable to kill YARN job

Question

I have a simple Samza job, which I submit to our YARN cluster. The job allocates one single container and runs without any issues.

When trying to kill the job, however, both the AM and job containers are left running on the NM, even though the RM claims that the job was killed successfully:

$ yarn application -kill application_1461969364354_5761
Killing application application_1461969364354_5761
16/05/19 06:48:49 INFO impl.YarnClientImpl: Killed application application_1461969364354_5761

From the NM log, I'm able to see:

2016-05-19 06:48:50,051 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e34_1461969364354_5761_01_000002 transitioned from RUNNING to KILLING
2016-05-19 06:48:50,051 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e34_1461969364354_5761_01_000002
2016-05-19 06:48:50,060 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1461969364354_5761 transitioned from RUNNING to FINISHING_CONTAI
NERS_WAIT

The status never transitions out of FINISHING_CONTAINERS_WAIT and I had to kill -9 the container process.

I'm using Samza version 0.10.0 and YARN version Hadoop 2.6.0-cdh5.4.9.

Any idea?

UPDATE:

After some digging, I'm able to see this:

2016-05-20 03:14:59,497 INFO org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking finishApplicationMaster of class ApplicationMasterProtocolPBClientImpl over rm157 after 2326 fail over attempts. Trying to fail over immediately.
org.apache.hadoop.security.token.SecretManager$InvalidToken: appattempt_1463512986427_0017_000001 not found in AMRMTokenSecretManager.
        at sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
        at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
        at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.finishApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:94)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy16.finishApplicationMaster(Unknown Source)
        at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:378)
        at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.unregisterApplicationMaster(AMRMClientAsyncImpl.java:157)
        at org.apache.samza.job.yarn.SamzaAppMasterLifecycle.onShutdown(SamzaAppMasterLifecycle.scala:63)
        at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$run$5.apply(SamzaAppMaster.scala:133)
        at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$run$5.apply(SamzaAppMaster.scala:132)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.samza.job.yarn.SamzaAppMaster$.run(SamzaAppMaster.scala:132)
        at org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:104)
        at org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): appattempt_1463512986427_0017_000001 not found in AMRMTokenSecretManager.        
        at org.apache.hadoop.ipc.Client.call(Client.java:1468)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy15.finishApplicationMaster(Unknown Source)
        at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.finishApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:91)
        ... 15 more