Python: Cann't connect to HDFS files

Asked Feb 01 '18 at 10:31

Active Feb 01 '18 at 19:06

Viewed 1,028 times

I have already tried many different ways but non of them does not work. For example, the followng way failed with the error "The system cannot find the file specified."

Example:

import pyarrow as pa
fs = pa.hdfs.connect('192.168.100.45', 20500, 'hdfs')

path='/user/hive/warehouse/sample_07/sample_07'
with fs.open(path, 'r') as f:
    print(f)

Error

FileNotFoundError: [WinError 2] The system cannot find the file specified

HDFS File System IMAGE:

enter image description here

Full error print screen:

enter image description here

edited Feb 01 '18 at 19:06

asked Feb 01 '18 at 10:31

AntnR

Please show your hdfs filesystem to prove that it exists – OneCricketeer Feb 01 '18 at 11:46
`fs.open` is unable to raise `FileNotFoundError`. Can you please post a complete stack trace – Wes McKinney Feb 01 '18 at 13:23
Wes McKinney: full error attached – AntnR Feb 01 '18 at 19:08
Did you manage to resolve this? I am facing exactly the same issue. T – MrfksIV May 04 '18 at 11:27
No I didn't. I began to work with Hive & Impala DB tables, not HDFS files. – AntnR May 07 '18 at 11:16
struggling with the same issue. It seems like you have to set some environment variables before you can actually use the function [source](https://github.com/apache/arrow/blob/master/python/doc/source/filesystems.rst) – zar3bski May 24 '18 at 11:33

Python: Cann't connect to HDFS files

0 Answers0