Stack Overflow
Stack Overflow

Questions tagged [apache-tez]

The Apache Tez project is aimed at building an application framework which allows for a complex directed-acyclic-graph of tasks for processing data.

The Apache Tez project is aimed at building an application framework which allows for a complex directed-acyclic-graph of tasks for processing data. It is currently built atop Apache Hadoop YARN

See Hive-on-Tez configuration properties.

192 questions
0
votes
0 answers

Hive: when it runs tez tasks and when not

I have a Hive installation with Tez as an execution engine. When I run simple queries (select * from table where col = value), it seems Hive doesn't rely on Tez at all. Explain looks like Plan optimized by CBO. 2 3 Stage-0 4 Fetch…
DanSab
  • 1
0
votes
0 answers

CAST string to BIGINT in hive not giving expected result

I am not able to understand output of query run on hive select count(*), count(col1) as count, count( distinct col1) as distinct, SUM (case when (cast(col1 as BIGINT) is null or cast(col1 as BIGINT) is not null )then 1 else 0 end) as…
Viren Agarwal
  • 1
  • 1
0
votes
1 answer

is there a Hive/hcat property to export partitioned tables to a single file instead of a directory?

I am trying to export a table with partitions. By default, it generates number of files based on the number of partitions. Is there a property I can set to merge the files, what is the performance consideration for making this change. Few of the…
wololo
  • 345
  • 2
  • 12
0
votes
0 answers

Hive query failed with org.apache.hadoop.hive.ql.exec.tez.TezTask java.lang.OutOfMemoryError: Java heap space

I have a Hive Table in ORC format, where: My table is partition by: Year + Month + Day. Total file size in HDFS: 10TB. Total records: 21 Billion records. Total data nodes in HDP: 8 I used Zeppelin notebook to query the table using…
Shuwn Yuan Tee
  • 5,578
  • 6
  • 28
  • 42
0
votes
1 answer

Is there an option to print Hive SQL runtime log while executing through SSH

I had a recent Hive version upgrade from 1.2.1 to 2.3.0, along with a movement from HDP-EC2 to EMR. Below is the shell script format using which i execute all the queries, ssh -T -i key.pem uname@99.99.99.99 <
Optimus Prime
  • 73
  • 12
0
votes
1 answer

modifying hive tez container size from IBM datastage with JDBC connector takes too long

In a mapr cluster using yarn and tez engine, we need to query hive data from datastage using jdbc connector. In some cases we need to increase tez container size due to data size. We do that in before sql statement in a parallel job, and then we…
AlvaroMatarranz
  • 1
  • 1
0
votes
1 answer

Hive : Tez how to increase AM Container memory

I am trying to run a hive query as set yarn.nodemanager.vmem-check-enabled=false; set hive.strict.checks.cartesian.product=false; select count(*) from db1.tb1 a where a.col1='2015-07-13' and a.col2='val' and exists ( select 1 from db2.tb2…
AbtPst
  • 7,778
  • 17
  • 91
  • 172
0
votes
2 answers

hive-on-tez mapper stuck in INITIALIZING with total number of containers being -1 when accessing data on S3/MinIO

I have a Hadoop+Hive+Tez setup from scratch (meaning I deployed it component by component). Hive is set up using Tez as execution engine. In its current status, Hive can access table on HDFS, but it can not access table stored on MinIO (using s3a…
Naitree
  • 1,088
  • 1
  • 14
  • 23
0
votes
1 answer

Hive on Tez: Could not find or load main class 400

I'm trying to configuring the Hive on Tez by following the formal document and the Tez installation guide. By overview my environment and configuration are as the following: - Apache Hadoop/YARN version 3.1.2 Apache Hive version 3.1.2 Apache Tez…
Charlee Chitsuk
  • 8,847
  • 2
  • 56
  • 71
0
votes
1 answer

Getting error in Hive : vertex failed failed due to Root_Input_init_failure, how much we can increase the memory while using Tez?

Getting error in Hive : vertex failed #some vertex name# Killed/failed due to Root_Input_init_failure
sreeram hadoop
  • 3
  • 3
0
votes
1 answer

How to run a query hive and get the applicationID via log

I'm writing a shell script that executes a hive command, writing the log and output information to two separate files: hive -S -f pdr_extrator.sql 2> pdr_extrator_log.txt | sed 's / [\ t] / | / g' 1> pdr_extrator_out.txt The log file, at the end of…
jsoldera
  • 13
  • 1
  • 5
0
votes
1 answer

Loading data to Hive using tez engine and ambari logs shows complete but logs hang without completing the Insert into those partitions

We are loading data into a hive table using tez execution engine. The jobs we use to load is in running state but on the ambari for the application id, we see it as succeeded and from the logs we could see it is stuck . It had around 90 partitions…
user10520244
  • 21
  • 3
0
votes
1 answer

TEZ doesn't work when tez.lib.uris==tez.tar.gz

I installed Tez 0.9.1 via Ambari 2.6.1. tez.tar.gz is successfully copped to hdfs. But while service check - I get error. Diving into logs by yarn logs -applicationId APP_ID I found Error: Could not find or load main class…
Andrey Emelyanenko
  • 11
  • 2
0
votes
3 answers

Hive query getting failed while trying to use TEZ engine

I am trying to execute the below query using the TEZ engine which got executed successfully in MR engine. set hive.execution.engine=tez;SET hive.exec.dynamic.partition = true; SET hive.exec.dynamic.partition.mode = nonstrict;SET…
Sourav Chatterjee
  • 159
  • 2
  • 15
0
votes
1 answer

Hive query failed in Tez engine but runing in MR mode

I have change my hive engine to Tez and want to run query using tez but query only executed with hadoop and hive users and when I change my user(user51) in beeline or Hue query got failed. But same query running fine with user51 when hive engine is…
Gabber
  • 7,169
  • 3
  • 32
  • 46
1 2 3
…
12 13