0

When running HiveQL queries (with TEZ Execution engine) with success in Hue on Hortonworks, the process is still marked as "Running" under "Hue Job Browser" & "Hadoop ResourceManager UI". Is it supposed to be doing this waiting for new input or is it failing to terminate?

My setup:

  • HortonWorks HDP-2.3.2.9-2950 running on 3 RH Linux 7 machines
  • Hue 3.9.0
  • Hive 1.2.1.2.3
  • Oozie 4.2.0.2.3
Mogsdad
  • 44,709
  • 21
  • 151
  • 275
Eirik Y. Øra
  • 81
  • 1
  • 8
  • Which execution engine for Hive: MapReduce or TEZ? Because TEZ retains a lease on YARN containers for several seconds, to avoid unnecessary allocate/deallocate operations when executing multiple queries in sequence (cf. `tez.am.container.reuse.enabled`, `tez.am.container.idle.release-timeout-min.millis`, `tez.am.container.reuse.rack-fallback.enabled` etc. in *tez-site.xml*) – Samson Scharfrichter Jan 04 '16 at 17:27
  • Thank you Samson! Is it possible to tune this "timeout" anywhere? (I also wrote the final answer based upon your answer) – Eirik Y. Øra Jan 05 '16 at 13:35
  • You can change these `tez.am.container.*` properties statically, in *tez-site.xml*; or you can reset them dynamically in your Hive script e.g. `set tez.am.container.reuse.enabled =false ;` (but I'm not sure it would apply immediately to the containers already allocated, since the A.M. is already up and running) – Samson Scharfrichter Jan 05 '16 at 15:09

2 Answers2

0

As stated by Samson Scharfrichter:

The TEZ execution engine retains a lease on YARN containers for several seconds after query completion, to avoid unnecessary allocate/deallocate operations when executing multiple queries in sequence.

This will make the completed query job appear as running for some seconds after the final results have appeared in the Hive Query Frontend.

Thank you Samson Scharfrichter !

Eirik Y. Øra
  • 81
  • 1
  • 8
0

Decreasing the timeout for tez.session.am.dag.submit.timeout.secs can help. Time is in sec.

<property>
<name>tez.session.am.dag.submit.timeout.secs</name>
<value>10</value>
</property>

/etc/tez-site.xml file.

desaiankitb
  • 992
  • 10
  • 17