0

I've just upgraded to HortonWorks HBase client 1.1.2.2.4.2.0-258 from version 0.98. All works fine locally, but under production load after a couple of minutes I'm starting to have TableNotFoundException:

Caused by: org.apache.hadoop.hbase.TableNotFoundException
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1264)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1162)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:152)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)

The behaviour seems random and unpredictable, e.g. repeating the same request just works (i.e. no exception is thrown and data are successfully retrieved)

I was trying to understand what was changed from 0.98 to 1.1 and the only significant thing I found was https://issues.apache.org/jira/browse/HBASE-16973

Playing around with those values didn't help.

Any thing else I should take into account? Any pointers are highly appreciated!

Thanks!

Vyacheslav
  • 1,186
  • 2
  • 15
  • 29
  • in local setup, the table you were using was created after updating the hbase-client or aleady existed one you are using? – Rahul Jan 03 '18 at 13:26
  • existing one. Actually I connected my local app to the production HBase. – Vyacheslav Jan 03 '18 at 13:47
  • please check all the regionserver's in production systems ip/hostname are reachable from your local system. if not please update host file in your local system – Rahul Jan 04 '18 at 05:12

1 Answers1

0

The problem was in the way how we used HConnection object, there were several instances of them (which, I agree, is not the right way to use it, IIUC). In the new version of the hbase client, there are also several "hconnection" threads for meta lookup per HConnection. All of this accumulated leading to the extremely high number of threads that JVM were not able to handle. After using a single HConnection instance, the problem went away and everything works as expected.

Vyacheslav
  • 1,186
  • 2
  • 15
  • 29