12

I'm running Hbase 1.0.1/Hadoop 2.5.2. I'm trying to run a scan on a table but I'm getting RPC timeouts.

I've changed the Hbase RPC timeout to 2 minutes which I can confirm frm the UI...

<property>
  <name>hbase.rpc.timeout</name>
  <value>120000</value>
  <source>hbase-site.xml</source>
</property>

... but my client is still timing out after 60s...

Caused by: java.io.IOException: Call to xxxxxxx/172.16.5.13:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=2968, waitTime=60001, operationTimeout=60000 expired.
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1235)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:31751)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
    ... 6 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=2968, waitTime=60001, operationTimeout=60000 expired.
    at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1177)
    ... 12 more

I've tried changing the cache block size but this doesn't seem to make any difference.

Is there some other timeout that I'm missing. There are a lot of rows in the table (millions) although the scan only returns 10's of thousands but the problem only seems to be with a specific set of regions.

JohnnyD
  • 477
  • 1
  • 4
  • 19
  • 1
    I have the same problem, did u get to solve it? Thanks – Marco Jul 29 '15 at 04:11
  • 1
    I gave up in the end. The volume of data we were writing started to exceed the throughput of our HBase platform. As raw event data was of secondary value compared to our event counters we decided to stop logging it in HBase. We're now writing it to text files as JSON records which we can still run map reduce jobs if need be and leaving HBase for more critical data. – JohnnyD Jul 29 '15 at 19:01

5 Answers5

14

When the server receives a scan RPC request, a time limit is calculated to be half of the smaller of two values: hbase.client.scanner.timeout.period and hbase.rpc.timeout (which both default to 60000 milliseconds, or one minute). This is why after setting 2 min your scan is getting timed out in 60 seconds.

When the time limit is reached, the server returns the results it has accumulated up to that point. This result set may be empty. If your usage pattern includes that scans will take longer than a minute, you can increase these values.

To make sure the timeout period is not too short, you can configure hbase.cells.scanned.per.heartbeat.check to a minimum number of cells that must be scanned before a timeout check occurs. The default value is 10000. A smaller value causes timeout checks to occur more often.

Below links might be helpful to configure timeouts for Hbase scan:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_installing_manually_book/content/best-practices-timeouts-phoenix.html

https://www.cloudera.com/documentation/enterprise/5-5-x/topics/admin_hbase_scanner_heartbeat.html#concept_xsl_dz1_jt

Gaur
  • 280
  • 3
  • 8
8

Based on the first answers. I configured the properties hbase.client.scanner.timeout.period and hbase.rpc.timeout before to open connection with the HBase Cluster:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;

Configuration conf = HBaseConfiguration.create();

conf.set("hbase.rpc.timeout", "1800000");
conf.set("hbase.client.scanner.timeout.period", "1800000");
joaolsouzajr
  • 326
  • 3
  • 4
4

Try this when you create connection in client side instead of in hbase-site.xml:

conf.set("hbase.rpc.timeout", "1800000")
Scott Xing
  • 41
  • 4
0

This one should works.

hbase org.apache.hadoop.hbase.mapreduce.Export -Dhbase.client.scanner.timeout.period=600000 tbname /path/to/hdfs
world watera
  • 5
  • 1
0

For the opterationTimeout, what you need to change is this one:

  <property>
    <name>hbase.client.operation.timeout</name>
    <value>10000</value>
  </property>
Liqun Chen
  • 69
  • 4