0

I have created a UDF translateText(), which calls API to translate the given text and returns the correct result in select clause, but when I apply the INSERT INTO TABLE as given below:

INSERT OVERWRITE TABLE gl_staging_eve.header_text select header_text, translateText(header_text) from gl_staging_eve.header_text_only;

The query fails with below output:

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1543885299280_8715_1_00, diagnostics=[Task failed, taskId=task_1543885299280_8715_1_00_000040, diagnostics=[TaskAttempt 0 failed, info=[Container container_e14_1543885299280_8715_01_000042 finished with diagnostics set to [Container completed. ]], TaskAttempt 1 failed, info=[Container container_e14_1543885299280_8715_01_000043 finished with diagnostics set to [Container completed. ]], TaskAttempt 2 failed, info=[Container container_e14_1543885299280_8715_01_000087 finished with diagnostics set to [Container completed. ]], TaskAttempt 3 failed, info=[Container container_e14_1543885299280_8715_01_000137 finished with diagnostics set to [Container completed. ]]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:40, Vertex vertex_1543885299280_8715_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:019/02/12 06:54:58 [ATS Logger 0]: INFO hooks.ATSHook: Received post-hook notification for :bduser_20190212065438_6cf35096-aa74-48d1-a8ae-6d76b0049099 19/02/12 06:54:58 [main]: ERROR ql.Driver: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1543885299280_8715_1_00, diagnostics=[Task failed, taskId=task_1543885299280_8715_1_00_000040, diagnostics=[TaskAttempt 0 failed, info=[Container container_e14_1543885299280_8715_01_000042 finished with diagnostics set to [Container completed. ]], TaskAttempt 1 failed, info=[Container container_e14_1543885299280_8715_01_000043 finished with diagnostics set to [Container completed. ]], TaskAttempt 2 failed, info=[Container container_e14_1543885299280_8715_01_000087 finished with diagnostics set to [Container completed. ]], TaskAttempt 3 failed, info=[Container container_e14_1543885299280_8715_01_000137 finished with diagnostics set to [Container completed. ]]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:40, Vertex vertex_1543885299280_8715_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 19/02/12 06:54:58 [ATS Logger 0]: DEBUG security.UserGroupInformation: PrivilegedAction as:bduser (auth:SIMPLE) from:org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPosting(TimelineWriter.java:112)

Basically INSERT INTO TABLE is not working.

Farooque
  • 3,616
  • 2
  • 29
  • 41
  • 1
    Can specify what `translateText` is doing and what API it calls? when you run simple select query Hive doesn't submit the job to the cluster and does it locally. When doing INSERT INTO TABLE Hive does submit the job to the cluster. You might not have access to translation service you are calling from clusters' nodes, that's why it failed. – serge_k Feb 12 '19 at 10:38
  • I am calling https://api.cognitive.microsoft.com/sts/v1.0/issueToken API, but how do I know I do not have access from cluster?? – Farooque Feb 12 '19 at 11:31
  • Even I have commented the API call and in evaluate method I am just returninng the input I am getting, then also I am getting the same isssue – Farooque Feb 12 '19 at 11:33
  • 1
    It's hard to say what the problem is. Try to change the execution engine to MapReduce: `set hive.execution.engine=mr` and then execute the query again to see what errors it will output. – serge_k Feb 12 '19 at 13:04
  • When I changed the execution engine to MapReduce: set hive.execution.engine=mr; then my INSERT query ran successfully. – Farooque Feb 14 '19 at 13:17
  • @serge_k: What can we conclude from above action?? – Farooque Feb 15 '19 at 05:54
  • it's really hard to say why it didn't work for Tez, you might need to write some additional instructions in your UDF. Unfortunately, I'm not an expert in Tez. – serge_k Feb 15 '19 at 07:03

1 Answers1

1

In Hive 1.2.x by default hive.execution.engine is tez.

When I submitted the INSERT INTO TABLE query where SELECT was having UDF then query was failing.

But when I changed the execution engine to MapReduce: set hive.execution.engine=mr; then my INSERT query ran successfully.

Hope this help others as well!

Farooque
  • 3,616
  • 2
  • 29
  • 41