Stack Overflow
Stack Overflow

Questions tagged [webhdfs]

WebHDFS is a REST API that supports the complete FileSystem interface for HDFS (Hadoop Distributed File System)

WebHDFS is a REST API that supports the complete FileSystem interface for HDFS (Hadoop Distributed File System). This Api is used to establish a connection to the Hadoop Data Lake from a third-party tool such as SSIS: Using WebHDFS to connect Hadoop Data Lake to SSIS

268 questions
0
votes
1 answer

Possible encrypted secure comminication to WebHDFS via HTTPS?

I'm using Cloudera Hadoop environment. I've been familiar with WebHDFS over the past few days. I'm using it to perform various HDFS operations through REST API. But, now i need to secure the communication to WebHDFS i.e using SSL/HTTPS. Is this…
Saurabh Gokhale
  • 53,625
  • 36
  • 139
  • 164
0
votes
1 answer

Distcp to webHDFS brings down job tracker

We are having a strange issue while trying to pump large amount of data using distcp to another cluster . To give out some numbers, Data size = 320GB Number of Mappers = ~70 Total number of nodes in our cluster 82 src cluster : hadoop…
learner
  • 885
  • 3
  • 14
  • 28
0
votes
2 answers

Permissions error on webhdfs

I'm working on using the REST interface to Hadoop's HDFS as a convenient way to store files over the network. To test I installed hadoop on my mac (10.8.5) following these…
Mike Miller
  • 136
  • 1
  • 5
0
votes
1 answer

How to Authenticate WebHDFS with c#

I have been attempting to upload files using c# into Hadoop using the WebHDFS REST API. This code works fine: using (var client = new System.Net.WebClient()) { string result = client.DownloadString("http:/…
user2732948
  • 1
  • 1
0
votes
1 answer

Is there a difference between webhdfs:// and hdfs:// in terms of underlying protocol?

Right now I am ingesting logs via HttpFS, which is falling apart at the sheer volume. I cannot run JVM on my edge for performance reasons. So I was thinking of using the native protocol for ingest (I've heard it's either IPC or raw network socket…
Carl Sagan
  • 982
  • 1
  • 13
  • 34
0
votes
1 answer

Need help in finding the location of the replicated file on my hdfs cluster

I am using Webhdfs for storing a file in my hdfs cluster. In the conf files I have mentioned the replication factor as 2. Using the Webhdfs api, the first PUT request tells us location of the datanode to put the file on and using the address in the…
anonymous123
  • 1,271
  • 6
  • 19
  • 43
-1
votes
1 answer

How to get specific key/value from HDFS via HTTP or JAVA API?

How can I get the value of one or more keys in HDFS via HTTP or JAVA api from remote client? For example, the file below has a million keys and values. I just want to get the values of the 'phone' and 'toys' keys. MyFile: book, 5 notebook, 5 phone,…
nhkb_55
  • 3
  • 1
-1
votes
1 answer

Failed to retrieve data from /webhdfs/v1/?op=LISTSTATUS: Server Error on macOS Monterey

I have installed Hadoop and able to access localhost Hadoop interface. When I try to upload files the interface gives me the error "Failed to retrieve data from /webhdfs/v1/?op=LISTSTATUS: Server Error".
Mikhail A
  • 1
-1
votes
1 answer

Erreur: HTTPConnectionPool(host='dnode2', port=9864): Max retries exceeded with url: /webhdfs

I'm trying to read a file on my hdfs server in my python app deployed with docker, during dev, I don't have any problem, but in prod there are this error : Erreur: HTTPConnectionPool(host='dnode2', port=9864): Max retries exceeded with…
Eboua Osée
  • 19
  • 3
-1
votes
1 answer

Build a dynamic UI for HDFS with REST API

For scientific purposes we have to build a UI for a Hadoop Data Lake. I found out that Hadoop offers a REST API for the WebHDFS. Unfortunately, the standard WebHDFS is not usable for the enduser. I would like to use the Hadoop REST API to program a…
madik_atma
  • 787
  • 10
  • 28
-1
votes
1 answer

How to put file from local computer to HDFS

Is there any way to push files from local compute to HDFS. I have tried to send GET request to port 50070 but It always shows Authentication required. Please help me ! I am quite new to HDFS
Nga Vu
  • 1
  • 3
-1
votes
3 answers

Identify Version of the Remote Hadoop Cluster

How can the version of remote Hadoop cluster be identified through api/web-services? i.e. whether the cluster is 1.x type or 2.x type? Is there any API/web-service available for the same? I have researched webhdfs? hadoop Filesystem but could not…
Pankaj Khattar
  • 111
  • 1
  • 10
-2
votes
1 answer

I resarch about HDFS failures. For this I need to HDFS logs . Where can I download the logs?

I resarch about HDFS failures. For this I need to HDFS logs . Where can I download the logs ?
Mehdi Medadian
  • 7
  • 1
  • 3
1 2 3
…
17
18