Recently my team is using hbase-indexer on CDH for indexing hbase table column to solr . When we deploy hbase-indexer server (which is called Key-Value Store Indexer) and begin testing. We found a situation that when we put data to hbase (We are using apache phoenix, a SQL layer above hbase) frequently, the hbase-indexer process will auto exit.We have checked the log and found ERROR log of Zookeeper Session Expired like this :
2016-04-18 12:17:50,340 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 31481ms for sessionid 0x2541e69d8a2001a, closing socket connection and attempting reconnect
2016-04-18 12:17:50,446 WARN com.ngdata.hbaseindexer.util.zookeeper.StateWatchingZooKeeper: Disconnected from ZooKeeper
2016-04-18 12:17:51,202 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server slave1/192.168.27.166:2181. Will not attempt to authenticate using SASL (unknown error)
2016-04-18 12:17:51,204 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.27.167:59529, server: slave1/192.168.27.166:2181
2016-04-18 12:17:51,211 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2541e69d8a2001a has expired, closing socket connection
2016-04-18 12:17:51,211 ERROR com.ngdata.hbaseindexer.util.zookeeper.StateWatchingZooKeeper: ZooKeeper session expired, shutting down.
2016-04-18 12:17:51,228 INFO org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:11060
2016-04-18 12:17:51,336 INFO com.ngdata.hbaseindexer.supervisor.IndexerSupervisor: IndexerWorker.EventWorker interrupted.
2016-04-18 12:17:51,448 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2541e69d8a20020 closed
2016-04-18 12:17:51,448 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-04-18 12:17:51,462 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 44594
2016-04-18 12:17:51,463 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=44594: stopping
2016-04-18 12:17:51,473 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2016-04-18 12:17:51,473 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2016-04-18 12:17:51,488 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper
2016-04-18 12:17:51,488 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.uniquekey.StringUniqueKeyFormatter
2016-04-18 12:17:51,514 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2541e69d8a2001f closed
2016-04-18 12:17:51,515 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-04-18 12:17:51,515 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 47364
2016-04-18 12:17:51,516 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=47364: stopping
2016-04-18 12:17:51,518 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper
2016-04-18 12:17:51,518 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2016-04-18 12:17:51,519 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2016-04-18 12:17:51,519 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.uniquekey.StringUniqueKeyFormatter
2016-04-18 12:17:51,527 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2541e69d8a2001e closed
2016-04-18 12:17:51,527 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-04-18 12:17:51,528 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 49605
2016-04-18 12:17:51,528 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=49605: stopping
2016-04-18 12:17:51,530 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2016-04-18 12:17:51,530 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2016-04-18 12:17:51,531 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper
2016-04-18 12:17:51,531 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.uniquekey.StringUniqueKeyFormatter
2016-04-18 12:17:51,539 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-04-18 12:17:51,539 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2541e69d8a2001c closed
2016-04-18 12:17:51,540 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 39464
2016-04-18 12:17:51,540 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=39464: stopping
2016-04-18 12:17:51,546 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2016-04-18 12:17:51,547 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2016-04-18 12:17:51,547 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper
2016-04-18 12:17:51,547 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.uniquekey.StringUniqueKeyFormatter
The software env is :
- CDH5.4
- HBase1.0
- Phoenix4.6
- Hbase-Indexer (hbase-solr-1.5-cdh5.4.2)
The java heapsize of hbase-indexer is configured to 1GB .
Is there anyone who meet this situation ?
