Using subprocess for accessing HBase

Question

I'm trying simple commands to access HBase through subprocess in Python. The following code gives me the wrong output:

import subprocess
cmd=['hbase','shell','list']
subprocess.call(cmd)

Instead of giving me the list of tables in HBase, I get the following output:

 Usage: hbase [<options>] <command> [<args>]
 Options:
  --config DIR    Configuration direction to use. Default: ./conf
  --hosts HOSTS   Override the list in 'regionservers' file

Commands:
Some commands take arguments. Pass no args or -h for usage.
shell           Run the HBase shell
hbck            Run the hbase 'fsck' tool
snapshot        Create a new snapshot of a table
snapshotinfo    Tool for dumping snapshot information
wal             Write-ahead-log analyzer
hfile           Store file analyzer
zkcli           Run the ZooKeeper shell
upgrade         Upgrade hbase
master          Run an HBase HMaster node
regionserver    Run an HBase HRegionServer node
zookeeper       Run a Zookeeper server
rest            Run an HBase REST server
thrift          Run the HBase Thrift server
thrift2         Run the HBase Thrift2 server
clean           Run the HBase clean up script
classpath       Dump hbase CLASSPATH
mapredcp        Dump CLASSPATH entries required by mapreduce
pe              Run PerformanceEvaluation
ltt             Run LoadTestTool
version         Print the version
CLASSNAME       Run the class named CLASSNAME

How do I give the subprocess command?

I'm pretty sure that hbase shell is in fact expecting an interactive shell. — orion78fr, Oct 20 '16 at 08:27

score 0 · Answer 1 · answered Oct 19 '16 at 17:31

0

If you need to access HBase from Python I strongly suggest you looks at the happybase modules.

I have been using them in production for the past 4 years - and they have simplified our ETL tasks.

Out of the box they are Python 2.X, but with a few minutes work - you can upgrade them to Python 3 (useful if you data is UTF-8)

answered Oct 19 '16 at 17:31

Tim Seed

5,119
2
30
26

Can happybase be used to rename a table? The only thing I found in the documentation is how to read/write to HBase – anonymous Oct 19 '16 at 19:10
I have read in the Docs that this is possible in the Connection - but I fail to see any methods that allow this. Sorry – Tim Seed Oct 20 '16 at 01:47
I have read in the Docs that this is possible in the Connection - but I fail to see any methods that allow this. From a quick look at the code - happy base is using the thrift interface (http://wiki.apache.org/hadoop/Hbase/ThriftApi) but this interface does not include snapshot, clone_snapshot and delete_snapshot (steps I am guessing you are using for your rename). So not a defect in the happy base layer - more of a thrift limitation. – Tim Seed Oct 20 '16 at 01:54
What version of HBase do you have? I have Hbase 1.2, it seems like Happybase has problem with hbase 1.2... – yuan0122 Jan 05 '17 at 00:04

score 0 · Answer 2 · answered Oct 24 '16 at 18:51

0

Figured out a way. I created a shell file that contained commands like:

echo 'list' | hbase shell -n

Ran this in Python.

answered Oct 24 '16 at 18:51

anonymous

405
8
22

score 0 · Answer 3 · answered Feb 12 '21 at 13:20

The below command worked for me.

import subprocess
print(subprocess.check_output(
                "echo \"list_regions \'table_name\'\" | hbase shell", shell=True))

Or

command = "echo \"list_regions \'table_name\'\" | hbase shell"
output,error  = subprocess.Popen(
                    command, universal_newlines=True, shell=True,
                    stdout=subprocess.PIPE, 

stderr=subprocess.PIPE).communicate()

print(output)
print(error)

Basically this is equivalent to the below

hbase_command | hbase shell

Using subprocess for accessing HBase

3 Answers3