how to get lot size of data from hbase by happybase at the same time?

Question

I'm trying to get data from hbase like this:

key = pd.read_sql('select key from table',hive_engine)
table = connection.table('games_ut')

res = {}
n = 0
for key in table.key:
    res[str(key)] = table.row(b'{key}'.format(key=key))
    n += 1
    if n % 100000 == 0:
        print(str(n) + " has been read,need a sleep!")
        sleep(0.5)

sleep(0.5) means have a rest. As you can see codes running well but too slow and gave us a lot of pressure on the cluster.Because the length of key is more than 40 million.

So,I want to know is there any way for me to get data lot size at the same time.I tried to use table.rows(),but fauild..... I am new and really how to generate lot size data at the same time by looping.

without profiling it's unclear what is slow. storing so much data in a dict can also be slow, for instance. — wouter bolsterlee, Jan 12 '17 at 08:01

score 0 · Answer 1 · answered Jan 12 '17 at 14:43

0

After my try......I know how to do.

Just use rows and pass a list as key...

answered Jan 12 '17 at 14:43

宋国庆

93
2
10

how to get lot size of data from hbase by happybase at the same time?

1 Answers1