How to Create HDFS file

Question

I know it is possible to create directory HDFS with python using snakebite

But I am looking to create a file on HDFS directory

Please share code you've tried so far – OneCricketeer Sep 22 '22 at 19:08 — OneCricketeer, Sep 22 '22 at 19:08

score 0 · Answer 1 · answered Sep 22 '22 at 20:20

0

You can use touchz to create an empty file on HDFS...

I see rename command in the docs, which might accept local source with a remote HDFS path for uploading files, but there is no put or copyFromLocal CLI command, so I don't think Snakebite can do what you want.

I do see put for the minicluster.

Overall, you may have better luck with pyspark or hdfs depending on your needs.

answered Sep 22 '22 at 20:20

OneCricketeer

179,855
19
132
245

I don't want to use hadoop native cmd like put, touch ..... – Zak_Stack Sep 23 '22 at 09:20
want I want is python script to create new hdfs file on a hadoop directory – Zak_Stack Sep 23 '22 at 09:21
1) snakebite touchz does create a file 2) None of those links are to native cmd. Look at the linked hdfs library, then., if you don't want to create an empty file, and would rather upload existing local data to hdfs – OneCricketeer Sep 23 '22 at 13:38

score 0 · Accepted Answer · edited Sep 23 '22 at 13:39

0

using subprocess

from subprocess import Popen, PIPE 

(ret, out, err)= run_cmd(['hdfs', 'dfs', '-touchz', '/directory/filename'])

edited Sep 23 '22 at 13:39

OneCricketeer

179,855
19
132
245

answered Sep 23 '22 at 10:20

Zak_Stack

103
8

You said you didn't want to use native cmd commands. You may as well use a bash script instead of Python, then – OneCricketeer Sep 23 '22 at 13:39
I used native hadoop cmd but with python – Zak_Stack Oct 05 '22 at 15:23

How to Create HDFS file

2 Answers2

Linked