kdb - sending large CSV from local to server

Question

All, I'm having some trouble sending a large csv-based tables (6 cols x 1mm+ rows) table to my server. I can bring tables back from my server with no issues. I've tried a few methods but presently I'm trying to read the csv and send it over as a string to build the table on the server. Perhaps I'm overlooking something very simple when assigning variables in the server string?

In trying to find a solution, I have been testing .Q.fs as it reads rows and inserts to a table thinking that I could just send a string and rebuild the table on the server. https://code.kx.com/q/kb/loading-from-large-files/#using-qfs

tbl: ("SIISSS";enlist",") 0:`:/data/local.csv; //how I normally load tables locally

Since I am unable to sort out how to send a tbl directly to the server, I was thinking of reading and sending as a string

//build table locally and insert records
 tbl:([]AAA:`first_row;BBB:1?10;CCC:2;DDD:`first_row;EEE:`first_row;FFF:`first_row); //create table tbl;
.Q.fs[{`tbl insert flip (cols tbl)!("SIISSS";",")0:x}]`:./Data/local.csv;   //load locally saved data;
tbl: delete from tbl where AAA =`first_row; //clean up by removing dummy data;

However, I'm not able to transfer this data to my server. I assume its my syntax assigning variables. I thought I would just read a file as text to send over and then save as a local csv on the server though I prefer not to save files there.

//try to send csv data from local to server
\p 5042;
h:hopen `:xxx.xxx.xx.xx:5042;
h "tbl:([]AAA:`first_row;BBB:1?10;CCC:2;DDD:`first_row;EEE:`first_row;FFF:`first_row)";  //create table on server

//TRYING TO READ TABLE AND SEND DATA TO SERVER 
csv_string: raze read0[./Data/local.csv];  //<--- read the file in as a razed string
h "data:" , csv_string ;  //                 <-----no luck assigning variable
h "save `/data/csv_string.csv";//            <-----save file to server directory to reference later

If I can get the file saved as a csv on the server then at lease I can just reference it.

h ".Q.fs[{`tbl insert flip (cols tbl)!("SIISSS";",")0:x}]`:/data/csv_string.csv";  //
h "tbl: delete from tbl where AAA =`first_row"; //clean up by removing dummy data;
hclose h

Answer 1

If you can hold the entire csv/table in memory client side. You could use the h(func;args) notation for ipc. Edit - corrected link: https://code.kx.com/q/learn/startingkdb/ipc/

Either use assign apply in the root namespace:

h({@[`.;`tbl;:;x]};([]a:1 2 3;b:`a`b`c))

or global assign:

h({tbl::x};localTblVariable)

Answer 2

Matt's suggestion is the best way to go when assigning variables on the server side.

With this, you can define variables from the local process (IE the table/csv in this case) in the server process.

// local process
q)h:hopen 23444
q)show local:("SIISSS";enlist",")0:`:local.csv
AAA       BBB CCC DDD       EEE       FFF
-----------------------------------------------
first_row 8   2   first_row first_row first_row
q)h({@[`.;`data;:;x]};local)
`.

// server process
q)\p
23444i
q)data
AAA       BBB CCC DDD       EEE       FFF
-----------------------------------------------
first_row 8   2   first_row first_row first_row
q)save`:data.csv
`:data.csv
q)("SIISSS";enlist",")0:`:data.csv
AAA       BBB CCC DDD       EEE       FFF
-----------------------------------------------
first_row 8   2   first_row first_row first_row