0

After configuring Hive with Tez, it works properly and the ACID table is created successfully. but when trying to run insert statement to insert record to this table, Tez failed withe below log

2023-08-04 09:48:26,265 [INFO] [main] |common.TezClassLoader|: Created TezClassLoader with parent classloader: sun.misc.Launcher$AppClassLoader@677327b6, thread: 1, system classloader: sun.misc.Launcher$AppClassLoader@677327b6
2023-08-04 09:48:27,355 [INFO] [main] |app.DAGAppMaster|: Creating DAGAppMaster for applicationId=application_1691002829482_0026, attemptNum=1, AMContainerId=container_1691002829482_0026_01_000001, jvmPid=83708, userFromEnv=hadoop, cliSessionOption=true, pwd=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1691002829482_0026/container_1691002829482_0026_01_000001, localDirs=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1691002829482_0026, logDirs=/opt/hadoop-3.1.1/logs/userlogs/application_1691002829482_0026/container_1691002829482_0026_01_000001
2023-08-04 09:48:28,487 [INFO] [main] |security.SecurityUtil|: Updating Configuration
2023-08-04 09:48:29,360 [INFO] [main] |app.DAGAppMaster|: Created DAGAppMaster for application appattempt_1691002829482_0026_000001, versionInfo=[ component=tez-dag, version=0.10.2, revision=72977b8720b2337ab0a0a3bf3b12e1c57900fa69, SCM-URL=scm:git:https://gitbox.apache.org/repos/asf/tez.git, buildTime=2022-07-08T16:21:10Z, buildUser=laszlobodor, buildJavaVersion=1.8.0_292 ]
2023-08-04 09:48:29,624 [INFO] [main] |app.DAGAppMaster|: 
/************************************************************
[system properties]
os.name: Linux
os.version: 3.10.0-1160.el7.x86_64
java.home: /opt/jdk1.8.0_202/jre
java.runtime.version: 1.8.0_202-b08
java.vendor: Oracle Corporation
java.version: 1.8.0_202
java.vm.name: Java HotSpot(TM) 64-Bit Server VM
java.class.path: /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1691002829482_0026/container_1691002829482_0026_01_000001:/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1691002829482_0026/container_1691002829482_0026_01_000001/hive-exec-3.1.2-50b0545693ae9392559584116a172ee32922e57ae9ea0eb7dc120290c217c2d5.jar:/tmp/hadoop-hadoop/nm-local-
user.name: hadoop
************************************************************/
2023-08-04 09:48:29,717 [INFO] [main] |counters.Limits|: Counter limits initialized with parameters:  GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2023-08-04 09:48:29,717 [INFO] [main] |counters.Limits|: Counter limits initialized with parameters:  GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=120
2023-08-04 09:48:29,751 [INFO] [main] |shim.HadoopShimsLoader|: Trying to locate HadoopShimProvider for hadoopVersion=3.3.1, majorVersion=3, minorVersion=3
2023-08-04 09:48:29,754 [INFO] [main] |shim.HadoopShimsLoader|: Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=3.3.1, majorVersion=3, minorVersion=3
2023-08-04 09:48:30,056 [INFO] [main] |app.DAGAppMaster|: AM Level configured TaskSchedulers: [0:TezYarn:null],[1:TezUber:null]
2023-08-04 09:48:30,056 [INFO] [main] |app.DAGAppMaster|: AM Level configured ContainerLaunchers: [0:TezYarn:null],[1:TezUber:null]
2023-08-04 09:48:30,056 [INFO] [main] |app.DAGAppMaster|: AM Level configured TaskCommunicators: [0:TezYarn:null],[1:TezUber:null]
2023-08-04 09:48:30,056 [INFO] [main] |app.DAGAppMaster|: Comparing client version with AM version, clientVersion=0.10.2, AMVersion=0.10.2
2023-08-04 09:48:31,885 [INFO] [main] |service.AbstractService|: Service org.apache.tez.dag.app.DAGAppMaster failed in state INITED
java.lang.ClassCastException: org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto cannot be cast to com.google.protobuf.Message
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:123)
    at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:965)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
    at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1731)
    at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1752)
    at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1749)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1764)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1760)
    at org.apache.tez.common.TezCommonUtils.getTezBaseStagingPath(TezCommonUtils.java:88)
    at org.apache.tez.common.TezCommonUtils.getTezSystemStagingPath(TezCommonUtils.java:146)
    at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:484)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
    at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2600)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2597)
    at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2384)
2023-08-04 09:48:31,893 [WARN] [main] |service.AbstractService|: When stopping the service org.apache.tez.dag.app.DAGAppMaster
java.lang.NullPointerException
    at org.apache.tez.dag.app.DAGAppMaster.initiateStop(DAGAppMaster.java:2111)
    at org.apache.tez.dag.app.DAGAppMaster.serviceStop(DAGAppMaster.java:2126)
    at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
    at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
    at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
    at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2600)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2597)
    at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2384)
2023-08-04 09:48:31,893 [ERROR] [main] |app.DAGAppMaster|: Error starting DAGAppMaster
java.lang.ClassCastException: org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto cannot be cast to com.google.protobuf.Message
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:123)
    at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:965)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
    at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1731)
    at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1752)
    at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1749)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1764)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1760)
    at org.apache.tez.common.TezCommonUtils.getTezBaseStagingPath(TezCommonUtils.java:88)
    at org.apache.tez.common.TezCommonUtils.getTezSystemStagingPath(TezCommonUtils.java:146)
    at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:484)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
    at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2600)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2597)
    at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2384)
2023-08-04 09:48:31,941 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked

Service org.apache.tez.dag.app.DAGAppMaster failed in state INITED i tried some solutions with no luck, could you please help me in that

Kaher
  • 3
  • 1
  • Did you try this https://stackoverflow.com/a/76641439/1332870 – CompEng Aug 05 '23 at 18:58
  • Yes, MR is working properly with Hive but Tez can assist in creating Hive DB and select from tables but insertion always fails. here is my tez-site.xml -------------------------------------------------- tez.lib.uris ${fs.default.name}/apps/tez/tez-0.10.2/share/tez.tar.gz tez.am.resource.memory.mb 3024 tez.am.java.opts Xmx2024m ------------------------------------------------- – Kaher Aug 07 '23 at 17:18
  • Try my soluyion not tar gz try open version – CompEng Aug 07 '23 at 18:01
  • Update, when i tried to test tez using below command hadoop jar /opt/tez-0.10.2/tez-examples-0.10.2.jar orderedwordcount /tmp/derby.log /tmp/output Tez took long time more than 15 minutes without finalizing the task. keep logging as below 023-08-07 10:25:48,434 INFO impl.YarnClientImpl: Submitted application application_1691427806408_0004 2023-08-07 10:25:48,469 INFO client.TezClient: The url to track the Tez AM: http://198.168.11.141:8088/proxy/application_1691427806408_0004/ 2023-08-07 10:26:03,257 INFO client.DAGClientImpl: DAG initialized: – Kaher Aug 07 '23 at 18:06
  • your solution gives below error .................................. Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/api/records/ApplicationId at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) – Kaher Aug 07 '23 at 18:39

0 Answers0