2

How would a q implementation of Windows' waitfor look like?

I have a q process that feeds rows of a table, one at a time, to another process (say tickerplant).

feed:{h`.u.upd,x;};
feed each tbl;

I'd like to feed the next row if any one of the conditions is met:

  1. the process receives the signal (next or ready) from the other process to do so;

OR

  1. time waiting for signal runs out (timeout is specific to each row of tbl and will be provided as a column of tbl)

Think of tbl as containing the list of events that are to be published sequentially either when the rest of CEP/tickerplant is ready OR when the next event is due (timeout measured as deltas timespan between the last published event and the next one, i.e. update timeout:((1 _deltas time),0Wp) from tbl), whichever happens sooner.

I've tried while[.z.N<prevtstamp+timeout;wait()];feed x but while blocks the process from listening to async messages from the other process AFAIK.

Other solutions considered:

Checking for signals with .z.ts is too slow as \t can't go below 1ms precision.

Polling the tickerplant for next(ready) signal continuously from within the while loop would slow down the tickerplant.

One solution is to maintain i index of the current row of tbl, separate the feeder out into two processes each handling one condition separately and polling for the current i index. This sounds slow, compared to eaching rows.

Daniel Krizian
  • 4,586
  • 4
  • 38
  • 75
  • I'm not sure if there are any available `system` calls that can help you but IMO q isn't the right place to do this. Perhaps a bit of C that gives you this functionality and you can call within `feed` to check status? – Manish Patel Sep 08 '16 at 10:57
  • Curious as to your use case - why not just send the messages over IPC as they arrive (or in batches) and leave it up to the consuming process to work through the queue? That way the client application is always processing the next available message when it is ready to do so. Are you having a problem with slow consumers? – user2242865 Sep 09 '16 at 03:36
  • @user2242865 the use case is backtest(replay) of historical data, rather than real-time streaming. The concern is known as look-ahead bias: slower consumers should not be able to act on information from the future - i.e. from faster consumers. Say the recorded sequence of external events in `tbl` is A, B and we apriori know that internally generated events C and D occurred after them in overall ABCD order. If a slow process consumes A and produces C (updates the tickerplant which in turn publishes it), and a fast process consumes B and produces D, the observed order would be incorrectly ABDC. – Daniel Krizian Sep 09 '16 at 08:25

1 Answers1

0
  1. KDB is a single thread application so any waitFor alike things will do nothing more than while loop.

  2. 1ms precision is pretty high already...And KDB is NOT design for a real time stuff. So even 1ms can be easily missed if the process is busying on something else. If u need higher precision than that.. U might need C or even Driver level help to improve precision.

  3. Maybe U can consider a better design to avoid the real time requirement :)

Ming
  • 358
  • 1
  • 10
  • I've gone with separating out the play/pause logic into three processes: (1.)feeder polls (2.)switch which state is switching back and forth between play and pause by async messaging from (3.)tickerplant depending on its subscribers` busyness. `waitfor` is in essence `while` statement as you said and as can be seen here: https://github.com/danielkrizian/poetiq/blob/backtest-unit/code/tick/backtestfeed.q#L27 – Daniel Krizian Sep 22 '16 at 17:09