kdb/q how to drop entries from lists

Question

I am trying to concatenate several csv's with identical columns.

getDataFromCsv :{[fn];
    if[not () ~key hsym fn; data: ("zSzzSISSIIIIIffffff"; enlist "\t") 0:fn;
    ... do stuff...
    :data];}

getFiles:{[dates;strat];root:"/home/me/data_";:{x: `$x, ssr[string y; "."; ""], ".csv"}[root] each dates;}


getData:{[dates;strat];`tasks set ([]c:());files:getFiles[dates;strat];:getDataFromCsv each files;}

doing this I get a list of tables with some entries empty where there was no file

[0] = ([] c1;c2;c2 ...
[1] = ([] c1;c2;c2 ...
[2] = ([] c1;c2;c2 ...
[3] = ([] c1;c2;c2 ...
[4] = ([] c1;c2;c2 ...
[5] = ::
[6] = ([] c1;c2;c2 ...

With those entries, I cant raze the list to get a table including all entries. How can I drop those empty entries?

score 8 · Accepted Answer · edited Feb 20 '18 at 08:42

You can remove from the list where the type is not 98h for a quick fix, assuming there are no other data types contained in the list:

q)r
::
+`a`b`c!(41 48 29;2 8 6;5 8 5)
+`a`b`c!(41 48 29;2 8 6;5 8 5)
+`a`b`c!(41 48 29;2 8 6;5 8 5)
q)raze @[r;where 98h=type each r]
a  b c
------
41 2 5
48 8 8
29 6 5
41 2 5
48 8 8
29 6 5
41 2 5
48 8 8
29 6 5

This also assumes that all columns are the same from each output. If they're not, you can use a uj to merge columns:

q)t:r,enlist ([] d:1 2 3; e:3 4 5)
q)(uj/)@[t;where 98h=type each t]
a  b c d e
----------
41 2 5
48 8 8
29 6 5
41 2 5
48 8 8
29 6 5
41 2 5
48 8 8
29 6 5
       1 3
       2 4
       3 5

score 5 · Answer 2 · answered Feb 20 '18 at 09:28

Personally...

Where you have "...do stuff..." or ":data", I'd simply check the count of data or add a similar check. If the count=0 then return an empty list '()' rather than the generic null '(::)' that you return in your function currently.

The generic null is the problem here and that's what you are looking to fix.

Example below...

// example returning generic null
q){if[x~0;:(::)];([]2?10)}each 1 0 3
+(,`x)!,4 4
::
+(,`x)!,7 9
q)raze {if[x~0;:(::)];([]2?10)}each 1 0 3
(,`x)!,6
(,`x)!,2
::
(,`x)!,9
(,`x)!,2

// put a check in against 'data' to return an empty list if count=0 or similar
q){if[x~0;:()];([]2?10)}each 1 0 3
+(,`x)!,3 2
()
+(,`x)!,1 8
// your raze works now
q)raze {if[x~0;:()];([]2?10)}each 1 0 3
x
-
3
1
7
2

kdb/q how to drop entries from lists

2 Answers2