How to delegate user query to a separate thread in KDB/Q

Question

Is there a way in KDB/Q to delegate user request to a separate thread? In my server, I have an internal calculation loop that runs at frequent intervals and takes time to finish. While it is running, all user queries are blocked.

How do I delegate calculation to an entirely separate thread? The only way to achieve multithreading I see is using peach/.Q.fc but that doesn't help in this case as I want to process user requests and internal loop in parallel.

score 1 · Answer 1 · edited Dec 04 '17 at 14:55

1

It sounds to me like you're using a process as a kind of hub here, both as a gateway and an analytics engine - I would suggest sending the internal loop out to a separate process with an asynchronous call and having it in turn asynchronously update anything that you need updated by it back on the main thread, although knowing more about the nature of the loop would be helpful in order to pre-identify any problems with doing so.

See here for more information on IPC in general.

edited Dec 04 '17 at 14:55

Thomas Smyth - Treliant

4,993
6
25
36

answered Apr 05 '17 at 16:02

Paul Kerrigan

465
5
12

its not feasible to spawn a new Q process as there are many variables, tables and other in memory globals that need to be shared. SO I guess there is really no way to delegate a task in Q to a separate thread other than using peach/.Q.fc to paralellize a calculation. – user236215 Apr 06 '17 at 13:50
Basically yes - you can't query a busy process. If you need to perform both the loop and the user query functionality as a requirement then it will be necessary for you to revisit your architecture and come up with some way to share data amongst your processes - this could be a pub/sub model, or the use of data on disk/shared memory segments. Depending on the nature of the loop, you may be able to send all the necessary data directly to the process performing the timed processing and update the main/user-accessible process without having any data globally available on the timed process. – Paul Kerrigan Apr 11 '17 at 13:00

score 0 · Answer 2 · edited Dec 04 '17 at 14:51

0

Sounds like you want to use negative port mode. There are restrictions on what can be done by the slave threads, but they're not too dissimilar to those of peaching anyway.

edited Dec 04 '17 at 14:51

Thomas Smyth - Treliant

4,993
6
25
36

answered Jun 16 '17 at 18:27

Simon Major

291
2
7

How to delegate user query to a separate thread in KDB/Q

2 Answers2