How to re-establish lost KDB connection in Java

Question

In my Java application I am establishing KDB connection using

private void initConnection() throws KException, IOException {
    conn = new c(host, port);
    conn.tz = TimeZone.getTimeZone(CONNECTION_TIMEZONE);
}

Then I use conn for inserting data asynchronously in tables successfully.

KDB server is restarted once in week and at that time as connection is reset by server I am getting an exception java.net.SocketException.
Using the above exception I can again re-establish lost connection but this is not a decent solution as I lose batch etc. and I don’t want connection exception to be thrown at the time of insertion.

Instead, is there a call-back/listener/Exception method in KDB API where API can inform java application upfront if a connection is closed by server / lost?

Tschallacka · Accepted Answer · 2018-01-29T15:04:08.287

2

Why don't you just do a

SELECT 1;

on your connection.

If it's open it'll run.
If it's closed it should throw an error because of the failing network connection.

You'd run this test only in the moments that your kbd server is scheduled for restart.

Personally, on the scheduled moments of restarting the kbd database I'd dump the data into an sqlite database for holding, and then when the frame for rebooting is over and connection is re-established dump the contents of the sqlite database into the kbd database.

edited Jan 29 '18 at 15:04

answered Jan 29 '18 at 14:12

Tschallacka

27,901
14
88
133

Are you suggesting to exeute SELECT 1 before every insert ? I am dealing with millions of record in a short span of time and will not prefer to execute SELECT 1 every time before insert. – AmbGup Jan 29 '18 at 14:44
Well, your server is reset once a week on a fixed time assume. So just run these tests in the minutes your server needs to get back up and ignore on the rest of the time. `public boolean safeToRun() { if(ConnectionManager.isInUnstableMoment()) {return select 1;} return true; }` Personally i'd choose to have an sqllite database to load data in in the unstable minutes, and hten when the database is back up empty the sqlite database into the main database. – Tschallacka Jan 29 '18 at 14:58
thanks this is good as a solution and in the meantime instead of SELECT 1 I am introducing a sync method to check if connection is live with query conn.getConn().k("1"); – AmbGup Jan 29 '18 at 15:26

score 1 · Answer 2 · answered Jan 31 '18 at 03:18

If connection is lost, I would expect conn object to throw IOException or its successor, when you try to hit KDB.

Ie, you should catch IOException and

invoke initConnection method to re-establish connection
re-execute query

Would be nice to set reasonable threshold of retries either. Otherwise, if KDb is down, the code will keep trying to connect to it.

score 0 · Answer 3 · answered Mar 10 '20 at 18:45

This answer is related (and possibly adds on) to Anton Dovzhenko's answer above, but hopefully the below helps you as well.

If we take a look at kx's documentation for reconnecting to a q process automatically, we can observe the following code snippet from ReconnectionExample.java below.

//initiate reconnect loop (possibly within a catch block).
while (true) {
  try {
    System.err.println("Connection failed - retrying..");
    //Wait a bit before trying to reconnect
    Thread.sleep(5000);
    qConnection = qConnFactory.getQConnection();
    System.out.println("Connection re-established! Resuming..");
    //Exit loop
    break;
  } catch (IOException | KException e1) {
    //resume loop if it fails
    continue;
  }
  …
}

That would be one way to try and reconnect; within a catch block. However, to answer your questions in greater detail:

To alert the rest of your application, do so in the catch block. One way to go about doing this is to invoke another call back method to alert that the connection is now down.
To reconnect, do this as well in your catch block as well. Be careful to only do this at a specified interval, though. The reason why I mention that is because when a Java application tries to reconnect, it will create a new c object.

This should help your application handle reconnecting more efficiently, and notifying any other dependencies appropriately.

How to re-establish lost KDB connection in Java

3 Answers3