0

I had a recent Hive version upgrade from 1.2.1 to 2.3.0, along with a movement from HDP-EC2 to EMR.
Below is the shell script format using which i execute all the queries, 

ssh -T -i key.pem uname@99.99.99.99 <<EOF
hive <<EO1
     <Hive-Query>
EO1
EOF

In version 1.2.1, the running shell script used to stdout runtime log including the query being executed and the running mapper/reducer counts in realtime.
In EMR with hive version 2.3.0, with exactly the same script, log has nothing except query complete notification.

Expected Log 

Total jobs = 1
Launching Job 1 out of 1


Status: Running (Executing on YARN cluster with App id application_********_0373)

Map 1: -/-  Reducer 2: 0/36 
Map 1: 0/191    Reducer 2: 0/36 
Map 1: 0(+1)/191    Reducer 2: 0/36 
Map 1: 0(+22)/191   Reducer 2: 0/36 
Map 1: 0(+44)/191   Reducer 2: 0/36 
Map 1: 0(+62)/191   Reducer 2: 0/36 
Map 1: 1(+61)/191   Reducer 2: 0/36 
Map 1: 2(+61)/191   Reducer 2: 0/36 
Map 1: 2(+62)/191   Reducer 2: 0/36 
.
.
.
.
.
Moving data to directory hdfs://ip-***************

Current Log in EMR 

Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1592149437885_0947)

Moving data to directory hdfs://ip-***************

Any pointers on what change would have caused the log to disappear?

Note : log is there when executing the query directly in cluster. But when executed through ssh, log is missing.

Optimus Prime
  • 73
  • 12
  • Do you want to print a hive's actual log files or is the --verbose flag not enough? The SSH session should let both stdout and stderr through to the local terminal. – Simon Jun 04 '20 at 22:37
  • When we execute a hive sql directly, it shows the progress of mappers and reducers as it runs. I need to have it printed out realtime during the ssh execution too. verbose mode only prints the executing query, not the realtime log. – Optimus Prime Jun 04 '20 at 22:43

1 Answers1

0

You can use ssh -t or ssh -tt to run your command remotely.

fmx
  • 1