Parallel Processing in kdb

Question

When we use -s, main thread delegate some work to slave threads.

I'm running below code on the session with 3 slave thread: q -s 3

select from t where date within 2019.01.18 2019.01.20

Trying to understand how q used threads internally for above query.
Does kdb internally use each child thread to fetch data for a date and then main thread combine the data at the end?
something like:

main thread looking for slave threads and assigning work
    slave 1:  t1: select from t where date within 2019.01.18
    slave 2:  t2: select from t where date within 2019.01.19
    slave 3:  t3: select from t where date within 2019.01.20
    main thread: t1,t2,t3

What work is done by each slave thread and what work is done by main thread?

score 3 · Answer 1 · answered Jun 24 '19 at 09:57

Using slave threads invokes a map and reduce framework in which each thread carries out their assigned task(s) and returns copies of their results for the main thread to then aggregate.

In your specific case if the table you're querying from is partitioned then the query would effectively be parallelized. As it would be pulling in all columns the query would be I/O bound which means you wouldn't see much/any speedup.

score 1 · Accepted Answer · answered Jun 24 '19 at 09:10

1

If your table is segmented and you use multiple threads, then partitions will be processed parallel. More details: https://code.kx.com/q4m3/14_Introduction_to_Kdb+/#1442-segmentation-vs-partitions and https://code.kx.com/v2/kb/partition/

answered Jun 24 '19 at 09:10

Ferenc Bodon

310
4
12

score 1 · Answer 3 · answered Jun 24 '19 at 20:27

1

try this {select from t where date=x} peach (date where date within 2019.01.18 2019.01.20)

it should be faster

answered Jun 24 '19 at 20:27

Alex Zamsky

21
2

Parallel Processing in kdb

3 Answers3