Stack Overflow
Stack Overflow

Questions tagged [webhdfs]

WebHDFS is a REST API that supports the complete FileSystem interface for HDFS (Hadoop Distributed File System)

WebHDFS is a REST API that supports the complete FileSystem interface for HDFS (Hadoop Distributed File System). This Api is used to establish a connection to the Hadoop Data Lake from a third-party tool such as SSIS: Using WebHDFS to connect Hadoop Data Lake to SSIS

268 questions
0
votes
1 answer

Trash config for hdfs not working

After adding following config in core-site.xml for hdfs, it does not create /user/X/.Trash folder when using webhdfs delete API. fs.trash.interval 10080 …
Phagun Baya
  • 2,127
  • 1
  • 18
  • 27
0
votes
1 answer

webhdfs rest api throwing file not found exception

I am trying to open a hdfs file that is present on cdh4 cluster from cdh5 machine using webhdfs from the command line as below: curl -i -L "http://namenodeIpofCDH4:50070/webhdfs/v1/user/quad/source/JSONML.java?user.name=quad&op=OPEN" I am getting…
JavaUser
  • 33
  • 9
0
votes
0 answers

HDFS file paths in HBase

We have a source of files each of from few MB to few GB in size. Each file is uniquely named and could be mapped to a person. However person information comes from different sources but it is not in the file system. Now, we have a requirement to…
user3600073
  • 1,773
  • 3
  • 18
  • 21
0
votes
1 answer

HDFS error + Incomplete HDFS URI, no host: hdfs://l27.0.0.1:9000

I run flume on hadoop with command flume-ng agent -c conf -f conf/flume-twitter.conf -n TwitterAgent and Now i am not able to save the twitter data. It shows error. I posted the error below. Can you have any idea ?? 2016-01-29 15:48:56,979…
srk
  • 599
  • 1
  • 6
  • 17
0
votes
0 answers

Can i install Apache Hue for Hadoop in Windows?

I am working on Hadoop but searching for a better open source HDFS explorer. Ended up with Apache Hue. For Linux there are manuals but what about for Windows?
srk
  • 599
  • 1
  • 6
  • 17
0
votes
0 answers

Multiple response parsing in python

I am using curl command to access hadoop(webhdfs) and for http response parsing i am using python. But after firing curl command ,multiple responses are being returned. curl -i …
Shashi
  • 2,686
  • 7
  • 35
  • 67
0
votes
1 answer

Checksum computation in WebHdfs

When file is ingested using "hdfs dfs -put" client computes checksum and sends both input data+checksum to Datanode for storing. How does this checksum calculatio/validation happen when File is read/write using WebHdfs ? how data integrity is…
Chhaya Vishwakarma
  • 1,407
  • 9
  • 44
  • 72
0
votes
1 answer

Create a hdfs file using pywebhdfs get an error: [Errno -2] Name or service not known

Some information about my dev env: openstack: juno hadoop: 2.4.1 pywebhdfs: 0.4.0 I create a hadoop cluster through openstack sahara client API, then I want to create a file to HDFS using pywebhdfs (for launching jobs on the created hadoop cluster).…
Robert.Chiang
  • 1
  • 3
0
votes
2 answers

Hadoop Webhdfs Delete option over Amazon EMR failed

i'm trying to see if delete option works over webhdfs : http://ec2-ab-cd-ef-hi.compute-1.amazonaws.com:14000/webhdfs/v1/user/barak/barakFile.csv?op=DELETE&user.name=hadoop but i get an error: {"RemoteException":{"message":"Invalid HTTP GET…
2Big2BeSmall
  • 1,348
  • 3
  • 20
  • 40
0
votes
1 answer

Cant acces from host to cloudera WEBHDFS (Guest)

I have read about Web Hdfs in all this links link1 link2 link3 checking with telnet - the port is OPEN made the needed settings in cloudera ( virtual box) to make webhdfs enable testing directly on cloudera works: [cloudera@quickstart ~]$ curl -i -L…
2Big2BeSmall
  • 1,348
  • 3
  • 20
  • 40
0
votes
2 answers

webhdfs always redirect to localhost:50075

I have a hdfs cluster (hadoop 2.7.1), with one namenode, one secondary namenode, 3 datanodes. When I enable webhdfs and test, I found it always redirect to "localhost:50075" which is not configured as…
feiyuw
  • 119
  • 1
  • 8
0
votes
1 answer

namenoderpcaddress is not specified

I am trying this command on my hadoop cluster (Cloudera 5.4.1) curl -i http://webhdfs:50075/webhdfs/v1/user/admin/test/test.txt?user.name=admin&op=OPEN&namenoderpcaddress=namenode:8022; echo Here I have confirmed that webhdfs, namenode machine…
Knows Not Much
  • 30,395
  • 60
  • 197
  • 373
0
votes
1 answer

Hadoop Java Client API messes up my fsURI

I try to access HDFS in Hadoop Sandbox with the help of Java API from a Spring Boot application. To specify the URI to access the filesystem by I use a configuration parameter spring.hadoop.fsUri. HDFS itself is protected by Apache Knox (which to me…
Tarmo
  • 3,851
  • 2
  • 24
  • 41
0
votes
1 answer

webhdfs two steps upload a file

I build a hadoop cluster with 4 machines: {hostname}: {ip-address} master: 192.168.1.60 slave1: 192.168.1.61 slave2: 192.168.1.62 slave3: 192.168.1.63 I use HttpFS upload a file to hdfs with restful way, there contains two steps to finish the…
Tenchael
  • 1
0
votes
1 answer

How to read hdfs file using webhdfs from a secure node?

I would like to make a webhdfs call to retrive a file from secure node using Java. Here is what I am doing 1. login using keytab (works fine). I am getting a kerberos ticket 2. Now when I try to invoke the below code, what it does is, it tries to…
Biscoot
  • 1
1 2 3
…
17 18