We are trying to update the hbase.rpc.timeout setting on our Apache Phoenix cluster, as we are having larger queries (creating indices or querying on non-indexed values) time out.
We updated the hbase-site.xml file with the new value forhbase.rpc.timeout from AWS and I SSHed into the servers and confirmed that the change was made in /etc/conf/hbase-site.xml, however the queries are still timing out and the timeout still seems to be set to the default:
org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Wed Nov 27 20:03:37 UTC 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60113: Call to ip-000-00-00-00.us-east-2.compute.internal/000-00-00-00.:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=108, waitTime=60010, rpcTimetout=60000 row '' on table 'INF_RES_ALT' at region=INF_RES_ALT,\x04,1574872147026.917a9e6e4a8d12e5c1dfb71aed9b4ee3., hostname=ip-000-00-00-00.us-east-2.compute.internal,16020,1574261382479, seqNum=1456
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1299)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1240)
at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:805)
at sqlline.BufferedRows.<init>(BufferedRows.java:37)
at sqlline.SqlLine.print(SqlLine.java:1660)
at sqlline.Commands.execute(Commands.java:833)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:813)
at sqlline.SqlLine.begin(SqlLine.java:686)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:291)
Any idea what we're doing wrong?
