Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [hdfs]

For questions regarding the Hadoop distributed file system (HDFS) which is part of the Apache Hadoop project.

71 questions
3
votes
2 answers

Disable The Under Replicated Blocks Alert in Cloudera Manager

I have a single server Hbase cluster that I am only using as the sink end of HBase replication. Therefore I don't want to replicate any blocks within this cluster (since the source has replicated blocks I don't feel I need it). I would like to…
Kyle Brandt
  • 83,619
  • 74
  • 305
  • 448
3
votes
1 answer

btrfs: Looking for experience on btrfs error modes on bad sectors / read errors / medium erros

While running a Hadoop cluster that uses HDFS (so data is already replicated 3x) we experience often issues with ext4 because some bad sectors (the filesystem is unable to read or write to them) causing ext4 to remount the filesystem read-only.. So…
kei1aeh5quahQu4U
  • 445
  • 5
  • 22
3
votes
1 answer

Setting up permissions in Cloudera Hadoop 4 (CDH4)

I would like to set up HDFS permissions in CDH4, with the following requirements: Everyone can read everything from all HDFS directories Each user can only write to his user directory on HDFS Except a special user who can write everywhere This is…
yby
  • 175
  • 2
  • 6
2
votes
1 answer

Running HDFS with only 1 data node - appending fails

I'm trying to test some services that require HDFS using Docker Compose. Since the services being tested, namenode, and data node(s) will all be running on the same physical machine (dev laptop), it would be nice to reduce the memory usage by only…
Robert Fraser
  • 141
  • 6
2
votes
2 answers

Can HDFS be installed on ZFS or BTRFS hard disks?

I am new to hadoop and confused about how HDFS works with ZFS or BTRFS. Can physical drives be mounted using ZFS and then have HDFS installed on top of ZFS? Or can HDFS be installed directly?
Greg
  • 1,657
  • 5
  • 27
  • 38
2
votes
0 answers

"The volume for a file has been externally altered ..." and setting file sizes on write to NFS from Windows

I have set up NFS access to HDFS. I can connect to the default root share from both Windows and Linux clients and all clients can read from the share. Writing from the Linux client is also ok. When I try to copy a file to the share from a Windows…
Bell
  • 121
  • 1
  • 5
2
votes
2 answers

Unable to convert HDFS from non-HA to HA

Introduction Aim: Convert HDFS from non-HA to HA. Method: According to this documentation it should be possible to convert HFDS from non- to HA by implementing following configuration: /etc/hadoop/conf/hdfs-site.xml …
030
  • 5,901
  • 13
  • 68
  • 110
2
votes
4 answers

HDFS datanode startup fails when disks are full

Our HDFS cluster is only 90% full but some datanodes have some disks that are 100% full. That means when we mass reboot the entire cluster some datanodes completely fail to start with a message like this: 2013-10-26 03:58:27,295 ERROR…
mbac
  • 21
  • 1
  • 2
2
votes
1 answer

Hadoop hdfs namenode is throwing an error

Full list of error: hb@localhost:/etc/hadoop/conf$ sudo service hadoop-hdfs-namenode start * Starting Hadoop namenode: starting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-localhost.out 12/09/10 14:41:09 INFO namenode.NameNode:…
Keval Domadia
  • 587
  • 5
  • 14
2
votes
2 answers

Deleting temp directory from HDFS

Is there a smart way of deleting old files from the hdfs /tmp directory? (Just to make sure, I am not talking about the unix FS /tmp)
Istvan
  • 2,582
  • 3
  • 22
  • 29
2
votes
2 answers

Hadoop DataNode is giving me an incompatible namespace ID

When I run the start-all.sh script from my master node, some of my DataNodes fail to start; the log file reports a Java IOException: Incompatible Namespace IDs in /tmp/$MY_USER_NAME.
ILikeFood
  • 399
  • 1
  • 5
  • 12
2
votes
1 answer

Does changing the default HDFS replication factor from 3 affect mapper performance?

Have a HDFS/Hadoop cluster setup and am looking into tuning. I wonder if changing the default HDFS replication factor (default:3) to something bigger will improve mapper performance, at the obvious expense of increasing disk storage used? My…
liamf
  • 392
  • 4
  • 10
1
vote
1 answer

List all files in hdfs directory

Due to some error at one component, files in HDFS got accumulated and the number is huge i.e 2123516. I want to list all files and want to copy their name in one file but when I run the following command, it gives Java heap space error. hdfs dfs -ls…
innervoice
  • 21
  • 7
1
vote
0 answers

HDFS NFS gateway read Input/output error

I have enabled the HDFS NFSv3 gateway on our HDFS cluster through official documentation. Everything works well except for one Ubuntu 16.04 server machine. The following is the kernel, mount and machine's sysctl -a output information. root@Linux:~$…
lordofire
  • 21
  • 3
1
vote
1 answer

Hadoop: Failed to start backup node, bad state: DROP_UNTIL_NEXT_ROLL

I have created a small Hadoop cluster setup with 1 NameNode and 1 DataNode to get hands-on. below is my configuration files: Core-site.xml fs.defaultFS …
Dipak
  • 111
  • 2
1
2
3 4 5