Stack Overflow
Stack Overflow

Questions tagged [webhdfs]

WebHDFS is a REST API that supports the complete FileSystem interface for HDFS (Hadoop Distributed File System)

WebHDFS is a REST API that supports the complete FileSystem interface for HDFS (Hadoop Distributed File System). This Api is used to establish a connection to the Hadoop Data Lake from a third-party tool such as SSIS: Using WebHDFS to connect Hadoop Data Lake to SSIS

268 questions
0
votes
1 answer

Reading file using webhdfs

I'm reading file residing on HDFS using curl(webhdfs) but problem is with data I'm getting HTTP headers also , How can I get only file data withought headers ? I'm using below command. If I'm removing "i"' option no output is given curl i -L…
Chhaya Vishwakarma
  • 1,407
  • 9
  • 44
  • 72
0
votes
1 answer

Using kerberized webhdfs/hdfs with JAVA API

I have a Hadoop cluster with Kerberos enabled I want to put files on HDFS using a windows/linux machine outside the cluster. Hadoop admin team have provided me with username to access hadoop and keytab file, how should I use them in my java code? I…
Chhaya Vishwakarma
  • 1,407
  • 9
  • 44
  • 72
0
votes
1 answer

Is there any possibility to get a notification when a certain file/ directory is deleted in HDFS?

I want to get a notification when a certain file/ directory has been deleted in HDFS. It would also be possible to generate a kind of log file with timestamp, affected file/ directory and so on. I need this due to traceability requirements. Is there…
Pigger
  • 11
  • 1
0
votes
1 answer

fluent and webhdfs filename with 197001011

I run td-agent on ubuntu 14.04 with the follow configuration: type tail format json path /path/tomcat/logs/file-input.log tag bhc.hdfs pos_file /var/td-agent/file.pos …
critical
  • 135
  • 1
  • 1
  • 7
0
votes
1 answer

WebHDFS append endpoint not behaving as documented

I'm following this documentation about my version of webHDFS (2.6), and trying to append to a file in the system that already exists. The issue that's confusing me is that the docs say I will receive a redirect to the appropriate datanode after…
zjm555
  • 781
  • 1
  • 5
  • 14
0
votes
1 answer

Copy data from local disk to HDFS in a federation of HDFS

I use two hosts (hadoop-coc-1 and hadoop-coc-2) to try to configure a Federation of HDFS in them. Therefore, I configured the $ cat etc/hadoop/hdfs-site.xml in both hosts (hadoop-coc-1, and hadoop-coc-2). Both HDFS are running properly with the…
xeon123
  • 819
  • 1
  • 10
  • 25
0
votes
1 answer

No copy method in WebHDFS

WebHDFS provides hooks for "rename", "delete" and other basic file manipulations. but here is no API hook for copy. Would someone mind explaining it or any good workarounds? Thanks!
Brian Dolan
  • 3,086
  • 2
  • 24
  • 35
0
votes
1 answer

Is there an equivalent of move in webhdfs

I would like to move one or more files from one path to another using webhdfs. I am using hadoop 1.3. Does such a REST call exist
user3304825
  • 107
  • 2
  • 12
0
votes
1 answer

Hadoop: Multinode cluster only recognizes 2 live nodes out of 3 data nodes

I have setup mutlinode hadoop with 3 datanodes and 1 namenode using virtualbox on Ubuntu. My host system serves as NameNode (also datanode) and two VMs serve as DataNodes. My systems are: 192.168.1.5: NameNode (also datanode) 192.168.1.10:…
Santosh Mishra
  • 74
  • 9
0
votes
1 answer

How to check HDFS SafeMode status through WebHDFS?

Is there any RestAPIs available to check the safemode status of HDFS? Currently using the below command to get the status? hadoop dfsadmin -safemode get Thanks
Kalai
  • 319
  • 1
  • 4
  • 21
0
votes
1 answer

Hadoop filesystem with kerberose

I am new to hadoop filesystem. I didn't find any helpful link on google related to Hadoop FileSystem. I want to authenticate using Kerberose while using Hadoop FileSystem. Sample Code : Path src = new Path("C:\\testing\\a\\a.avro"); Path…
user608020
  • 313
  • 4
  • 15
0
votes
1 answer

Node.js and HDFS

I need to read information about node.js and HDFS. I work with Centos. I have a file in HDFS and I want to read and print it in console. I wrote a node.js program to write a file and it works. But when I want to print an HDFS file, it doesn't work.…
Carlota
  • 1,239
  • 3
  • 29
  • 59
0
votes
1 answer

Hadoop Rest API for upload / download

I am trying to perform upload/download a file from Hadoop cluster, using a C# app, but I couldn't find the APIs for Upload and download from the documentation. So can you please let me know how to upload and download files from Hadoop using…
Kalai
  • 319
  • 1
  • 4
  • 21
0
votes
1 answer

Upload file to HDFS in chunks

I would like to use the gem 'webhdfs' to upload a file in chunks to HDFS. I see in the documentation there is a create method but I'm not sure how to use that and upload a large file in chunks. Has anyone tried this?
Jackson
  • 6,391
  • 6
  • 32
  • 43
0
votes
1 answer

Windows WebHDFS Client to Cloudera Hadoop

We have a Windows application which is communicating fine via the WebHDFS Client (In the Incubator phase) http:/ /hadoopsdk.codeplex.com/wikipage?title=WebHDFS%20Client&referringTitle=Home to a Cloudera Hadoop installation. The next phase is to…
xadzax
  • 21
  • 2
1 2 3
…
17
18