Creating a file in HDFS using python

Question

I am trying to create a file/directory in HDFS using python. To be clear, I am running a Hadoop streaming job with mapper written in Python. This mapper is actually trying to create a file in HDFS. I read that there are several Python frameworks to do this, but my interest is to go for Hadoop streaming. So, is there any way in Hadoop streaming to accomplish this?.

score 1 · Answer 1 · answered Nov 19 '19 at 19:13

You Can run command HDFS in script python

import sys, subprocess

def run_cmd(args_list):
        proc = subprocess.Popen(args_list, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
        (output, errors) = proc.communicate()
        if proc.returncode:
                raise RuntimeError('Error run_cmd')
        return (output, errors)

And run

(out, errors)=run_cmd(['hdfs','dfs','-mkdir','%s' %apth_HDFS_to_create_folder])

score 0 · Answer 2 · answered Sep 22 '22 at 16:24

0

there is no way to create file with python script, but it's possible to create directory using pydoop or snakebit

see : https://www.geeksforgeeks.org/creating-files-in-hdfs-using-python-snakebite/

answered Sep 22 '22 at 16:24

Zak_Stack

103
8

No way to create file? https://hdfscli.readthedocs.io/en/latest/quickstart.html#reading-and-writing-files – OneCricketeer Sep 22 '22 at 20:21
yes it is possible to create file using: (ret, out, err)= run_cmd(['hdfs', 'dfs', '-touchz', filename]) – Zak_Stack Sep 30 '22 at 13:34
Yes, but no. It's possible with `pip install hdfs` **not subprocess** - https://pypi.org/project/hdfs/ – OneCricketeer Sep 30 '22 at 16:31
it's not about that – Zak_Stack Oct 03 '22 at 09:44

score 0 · Answer 3 · edited Sep 30 '22 at 20:23

0

Solution using supprocess inspired by this answer in the "Create HDFS file" question.

from subprocess import Popen, PIPE 

(ret, out, err) = run_cmd(['hdfs', 'dfs', '-touchz', '/directory/filename'])

edited Sep 30 '22 at 20:23

phwt

1,356
1
22
42

answered Sep 23 '22 at 13:52

Zak_Stack

103
8

score 0 · Answer 4 · answered Sep 30 '22 at 13:35

it is possible to create file using:

#define run commande function which run hadoop native linux cmd 
def run_cmd(args_list):
        """
        run linux commands
        """
        # import subprocess
        print('Running system command: {0}'.format(' '.join(args_list)))
        proc = Popen(args_list, stdout=PIPE, stderr=PIPE)
        s_output, s_err = proc.communicate()
        s_return =  proc.returncode
        return s_return, s_output, s_err 

(ret, out, err)= run_cmd(['hdfs', 'dfs', '-touchz', filename])

Please edit your [other answer](https://stackoverflow.com/a/73828803/2308683)(s) rather than post multiple different ones — OneCricketeer, Sep 30 '22 at 16:30

Creating a file in HDFS using python

4 Answers4