3

I use the following line to upsert data into Hbase using Phoenix.

upsert into fraud_xxx_y.detection_details (call_date_hour, msisdn,
insert_time, call_time, source, shutdown_file,imsi, imei, cell_id, lac, 
vendor, model, calls, cdt_duration ) values ( '2017032121' , 
'1117037044082', to_date('2017-03-21 21:38:32.752'), '2017-03-21 03:13:58','GKP','SbGSM_20170321172419_269.txt','621300303944920','null','50007'
,'0','null','null',108,269)

The same upsert statement is used for 2 different databases, db1 and db2, and both have the same table detection_details.

We see that for db1, only for the table detection_details the upsert is delayed by an hour each time, while for db2, the upsert goes through immediately.

The only differences in the 2 processes is for db1, the upserts are close to 100K within a 10 min period, while for db2 the upserts are 10K.

It seems that for db1, it keeps waiting, for about an hour with data in some cache, and at the end of the hour, it wakes up and puts all data in the table. We have not configured any cache for the operation, so the delay is a real mystery. Appreciate any insight into this.

sparkDabbler
  • 518
  • 2
  • 7
  • 20
  • you might want to look at your region server and see if there is any hotspotting - also how are you telling if the data is delayed? eg look at the actual upsert timing or subsequent queries over a filtered time window... I'd look at the clock on the db2 and db2 – Paul Bastide Mar 24 '17 at 11:43

0 Answers0