Use for questions specific to Apache Hadoop 3.0 features (i.e. Erasure Coding, YARN Timeline Service v2, Opportunistic Containers, 3+ NameNode fault-tolerance). For general questions related to Apache Hadoop use the tag [hadoop].
Questions tagged [hadoop3]
112 questions
0
votes
1 answer
Cannot create directory on hadoop through hadoop web console
I have set up hadoop environment using a Linux VMWare image. I am able to create file and folder using Linux terminal
But when I use the web interface to do the same, I get error:
Permission denied: user=dr.who, access=WRITE,
…
FLICKER
- 6,439
- 4
- 45
- 75
0
votes
1 answer
Problem accessing files from S3 using spark on remote Yarn cluster
I'm trying to import csv files from S3 using spark-shell (val df=spark.read.csv("s3a://xxxxxx") )
the spark-shell client is connected to a remote yarn cluster.
It failed with java.lang.VerifyError, however, when I launch spark-shell from the same…
user1488211
- 55
- 1
- 5
0
votes
1 answer
yarn usercache dir not resolved properly when running an example application
I am using Hadoop 3.2.0 and trying to run a simple application in a docker container and I have made the required configuration changes both in yarn-site.xml and container-executor.cfg to choose LinuxContainerExecutor and docker runtime.
I use the…
Vinay Kashyap
- 31
- 3
0
votes
0 answers
How can a Hadoop MapReduce Job be executed from a Java program?
I am trying to integrate a Hadoop MapReduce Job as part of a Hadoop Job Dispatching software-client (GUI) that I am developing as a personal project.
I now have two files: the first being the client, and the second being a correctly functioning…
Vilitaria
- 107
- 9
0
votes
1 answer
Hadoop 3.1.1 showing only one live datanode
I installed hadoop 3.1.1 on 3 virtual machines with VMware on Ubuntu. When I run hdfs namenode -format and start-all.sh then jps works correctly on my master and two slaves nodes.
However, with the command hdfs dfsadmin -report, I can see only one…
Juan
- 184
- 1
- 4
- 16
0
votes
1 answer
Failed to initialize schema for HiveServer2 in Apache Hive 3.0.0 on Cygwin (Windows 10)
I already had a Hadoop 3.0.0 cluster consisting of 2 machine: 1 namenode + RM and 1 datanode. I tried to install Apache Hive 3.0.0 by following this document.
When I run schematool -dbType derby -initSchema --verbose on Cygwin, an exception was…
phqb
- 351
- 3
- 14
0
votes
1 answer
Hadoop 3.1.1 issue on start HDFS service
I am configuring Hadoop 3.1.1 on mac. However, once I complete all the steps I got the Permission denied on starting namenode, datanode and secondary namenode.
Also,
(unable to load native-hadoop library for your plateform.....using builtin-java…
Hatim TM
- 21
- 3
0
votes
0 answers
hadoop start namenode error root account is not found
when i have run bin/hdfs namenode -format,i want to run sudo sbin/start-dfs.sh,but it throw a error that root account is not found.
Starting namenodes on [localhost]
ERROR: Refusing to run as root: root account is not found. Aborting.
Starting…
Y.xp_rwb
- 1
- 2
0
votes
1 answer
JPS command shows only JPS
I installed hadoop and tried to run it. The terminal shows that everything has been started but when i run jps command it shows only jps. I am new to ubuntu and we need to use for academics can anyone help me run it.
I installed java using sudo…
Shivam...
- 409
- 1
- 8
- 21
0
votes
1 answer
hadoop3 can't find .nm-local-dir.usercache.hadoop.appcache. when doing pi test
I'am trying to setup an hadoop3 cluster on a local computer network, in small scale for starting one master node and two workers node.
I think I manage to have something that should work, following this tutorial configure hadoop 3.1.0 in multinodes…
RomainL.
- 997
- 1
- 10
- 24
0
votes
0 answers
Hadoop 3.0.0 No lib/native folder. WARN util.NativeCodeLoader
I was following https://amodernstory.com/2014/09/23/installing-hadoop-on-mac-osx-yosemite/ to install hadoop on my mac for local small tasks.
When hstart/hstop, getting "WARN util.NativeCodeLoader: Unable to load native-hadoop library for your…
sue
- 43
- 4
0
votes
1 answer
Hadoop 3.0 erasure coding: impact on file read performance?
I am trying to understand the impact erasure coding could have on read performance of a file.
Before that a brief summary of Hadoop 3.0 erasure coding using the Reed-Solomon method. If a file split into k blocks are encoded into p parity blocks,…
samshers
- 1
- 6
- 37
- 84
0
votes
1 answer
How do i set up HBase with HDFS 3.1.0?
HDFS 2.7 is default version for HBase 2.0.0.
For HBase Stable version, it is, 2.5.
I just started HDFS cluster with version 3.1.0. How do I make HBase to use this?
I get hsync error message.
EDIT
I am understanding that I have to replace all these…
Rock
- 13
- 7
0
votes
1 answer
Run HDFS pseudo mode in a docker container
I'm trying to run a HDFS under pseudo mode in a docker container, configured with this page: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html#Pseudo-Distributed_Operation, but I didn't use start-all.sh…
user3593261
- 560
- 4
- 17
0
votes
1 answer
Partitioning Not Working in Hive 2.3.0
I have created table as follows:
create table emp (
> eid int,
> fname string,
> lname string,
> salary double,
> city string,
> dept string )
> row format delimited fields terminated by ',';
then to enable partitioning…
Juned Ansari
- 5,035
- 7
- 56
- 89