Use for questions specific to Apache Hadoop 3.0 features (i.e. Erasure Coding, YARN Timeline Service v2, Opportunistic Containers, 3+ NameNode fault-tolerance). For general questions related to Apache Hadoop use the tag [hadoop].
Questions tagged [hadoop3]
112 questions
0
votes
0 answers
Unable to run MapReduce program on Hadoop 3.3.4
I am newbie and have setup Hadoop 3.3.4 on my laptop. I have performed the configurations like setting HADOOP_HOME in .bash_profile, changes to mapred-site.xml, yarn-site.xml, core-site.xml and hdfs-site.xml. I am able to start dfs and yarn and they…
0
votes
0 answers
Invalid UTF-8 start byte 0xa0 (at char #720, byte #37)
I have installed hadoop-3.3.4 in Ubuntu 22.04.1 virtual machine using virtual box 7. I have configured the core-site.xml. But, it cannot run the format of namenode.
The code for core-site.xml is stipulated below
…
0
votes
1 answer
Continuous rolling temporal window query in hive3
I am trying to figure out how to query a table (Hive 3) of dates and values with a moving temporal window aggregation. In the example below, I want to collect every two-day window possible (in which non-terminal dates would each be used…
avikarto
- 101
- 5
0
votes
0 answers
Simple Hive Job with 8 GB csv data consumes all disk (around 500GB)
I have setup Hadoop cluster with 20GB RAM and 6 cores. I have around 8 GB data in 3 csv files and I have to join them. For this purpose, I have used Apache Hive for this. Hadoop, Hive are 3.x version.
Here is the Hive query
SELECT distinct…
Hafiz Muhammad Shafiq
- 8,168
- 12
- 63
- 121
0
votes
1 answer
Is it necessary to configure hadoop.tmp.directory in core-site.xml in hadoop-3.3.1?
As I am a beginner in the field of Big data, installed Hadoop 3.3.1 and Hive and uploaded data on Hive. I have some questions and confusions:
I have not configured hadoop.tmp.directory in core-site.xml but configured datanode and namenode dir paths…
Javeria
- 5
- 5
0
votes
1 answer
flume monitor hive log, and sink to hdfs error:Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoSuchMethodError
hadoop 3.3.1
Hive2.3.9
flume 1.9.0
what I want is:when hive updates log, flume monitors hive log,flume sink hive log changes to hdfs.but I can't get log in hdfs.
I have downloaded and copy below jars to…
Venus
- 1,184
- 2
- 13
- 32
0
votes
1 answer
how to jdbc to hive using java on ubuntu?
Ubuntu 16.04.1 LTS
hadoop 3.3.1
Hive 2.3.9
I have a java file:
public class HiveCreateDb {
private static String driverName = "org.apache.hive.jdbc.HiveDriver";
public static void main(String[] args) throws SQLException {
//…
Venus
- 1,184
- 2
- 13
- 32
0
votes
1 answer
hive 2.3.9 can not start because: Couldn't create directory /tmp/hive/a44bc0d7-d01b-4c14-ae34-62a34d0b4827_resources
Hive 2.3.9
Ubuntu16.04.3-64bit
I have started hadoop:
hadoop@ubuntu:~/hadoop/sbin$ ./start-all.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [ubuntu]
Starting resourcemanager
Starting nodemanagers
and have…
Venus
- 1,184
- 2
- 13
- 32
0
votes
1 answer
hadoop 3.3.1 fs -mkdir No such file or directory
$ hadoop fs -mkdir /home/hadoop/hadoop_input
mkdir: `hdfs://localhost:9000/home/hadoop': No such file or directory
even I create /home/hadoop/hadoop_input folder,the error still exists.
Venus
- 1,184
- 2
- 13
- 32
0
votes
1 answer
what is the value of arguments in command hadoop fs
hadoop 3.3.1
I am learning Hadoop,and find a command
hadoop fs -ls
What is the possible value of ?
Venus
- 1,184
- 2
- 13
- 32
0
votes
2 answers
hadoop on macOS initiating secondary namenode fails due to ssh connection refused
I've successfully gone through initiating single-node in a pseudo-distributed mode described in https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html#Pseudo-Distributed_Operation, under Window's wsl2…
김기영
- 123
- 9
0
votes
0 answers
Could not able to connect Hive Metastore from intellij on local machine
After looking at all the solutions, I am asking this question.I have been trying to create a view from dataframe.I am able to connect dataframe but not able to create a view on top of that. I am not able to connect to local metastore using the…
Mirshad VI
- 1
- 1
0
votes
1 answer
Hadoop aws and spark incompatibilities
I've a very strange dependency errors:
I've this simple scala code:
val spark: SparkSession = SparkSession.builder()
.master("local[1]")
.appName("HDFStoAWSExample")
.getOrCreate()
spark.sparkContext
…
Matteo Notaro
- 31
- 5
0
votes
1 answer
Unable to access localhost:7070 due to error in Apache Kylin
Good afternoon, I am having problems when trying to access http://:7070. I installed the following programs: apache-hive-3.1.2-bin , kylin-3.1.1-bin-hadoop3 , kafka_2.12-2.5.0 , hadoop-3.3.0 , and hbase-2.3.4
I tend to think it has to do with some…
0
votes
1 answer
Unable to start Hadoop Tools start-all.sh showing nothing
Freshly installed Hadoop 3.3 version on Mac system and trying a simple standalone system. And was trying to start all the services but i see there is no logs from the system.
core-site.xml
…
narengs7
- 67
- 1
- 9