0

I am using Impyla for querying some results from HIVE, however, I met this problem:

From Impyla:

impala.error.OperationalError: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

and from HiveServer2:

WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = hduser_20180827031927_fdb148b0-725b-434c-a0f8-98b6843d4348
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Defaulting to jobconf value of: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

and my source code is:

from impala.dbapi import connect
import sys


dbName = sys.argv[1:][0]
query = sys.argv[1:][1]

conn = connect(host='192.168.0.10', port=10000, database=dbName , auth_mechanism='NOSASL' , use_ssl=True)
cursor = conn.cursor()
cursor.execute(query, configuration={'hive.exec.reducers.bytes.per.reducer': '100000', 'hive.auto.convert.join.noconditionaltask':'false','mapreduce.job.reduces':'1','hive.auto.convert.join':'false'})
returnData = []
for row in cursor:
    returnData.append(row[0])
pprint(returnData)

as you see I have added many configurations, but it does not work

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Truong Nguyen
  • 75
  • 2
  • 11
  • Show your query and the error from the yarn UI, not the client – OneCricketeer Aug 27 '18 at 06:04
  • 1
    @cricket_007 You save my day. Thank you very much, the fault belongs to yarn. When I check the Yarn interface, it does not show. Then I found that it has not started correctly. Now everything works fine. – Truong Nguyen Aug 27 '18 at 07:56

2 Answers2

0

From your error, there is no way to know what happened.

I'm not sure to enable debug logging in impyla, so you'll need to go to the YARN UI to find the query.

If YARN isn't running, I would think you'd get a more descriptive error such as "unable to submit job", though maybe that error is not being propagated from HiveServer

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
0

I was facing the same problem. After browsing thousands of pages on google I realized the error message that impyla returned had no help, so I started to check the hive logs. In the hive-site.xml file, I searched the "log" keyword, and luckily, I found this parameter:

hive.server2.logging.operation.log.location

so I went to this directory, and located the log message generated by my impyla code. The error message was as follows:

ERROR : Job Submission failed with exception 'org.apache.hadoop.security.AccessControlException(Permission denied: user=APP, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x

At this stage, everything was clear, the username I passed in my impyla code had no permission to do the query, so I changed to another username, and world returned back to peace:)

吴武辉
  • 3
  • 1
  • 2