0

I see a lot of info regarding serializing tables on kdb but is there a suggested best practice on getting functions to persist on a kdb server? At present, I and loading a number of .q files in my startup q.q on my local and have duplicated those .q files on the server for when it reboots.

As I edit, add and change functions, I am doing so on my local dev machine in a number of .q files all referencing the same context. I then push them one-by-one sending them to the server using code similar to below which works great for now but I am pushing the functions to the server and then manually copying each .q file and then manually editing the q.q file on the server.

\p YYYY;
h:hopen `:XXX.XXX.XX.XX:YYYY;
funcs: raze read0[`$./funcs/funcsAAA.q"]; 
funcs: raze read0[`$./funcs/funcsBBB.q"]; 
funcs: raze read0[`$./funcs/funcsCCC.q"]; 
h funcs;

I'd like to serialize them on the server (and conversely get them when the system reboots. I've dabbled with on my local and seems to work when I put these in my startup q.q

`.AAA set get `:/q/AAAfuncs
`.BBB set get `:/q/BBBfuncs
`.CCC set get `:/q/CCCfuncs

My questions are:

  1. Is there a more elegant solution to serialize and call the functions on the server?
  2. Clever way to edit the q.q on the server to add the .AAA set get :/q/AAAfuncs
  3. Am I thinking about this correctly? I recognize this could be dangerous in a prod enviroment

ReferencesKDB Workspace Organization

Jason_L
  • 427
  • 1
  • 3
  • 12

1 Answers1

3

In my opinion (and experience) all q functions should be in scripts that the (production) kdb instance can load directly using either \l /path/to/script.q or system"l /path/to/script.q", either from local disk or from some shared mount. All scripts/functions should ideally be loaded on startup of that instance. Functions should never have to be defined on the fly, or defined over IPC, or written serialised and loaded back in, in a production instance.

Who runs this kdb instance you're interacting with? Who is the admin? You should reach out to the admins of the instance to have them set up a mechanism for having your scripts loaded into the instance on startup.

An alternative, if you really can't have your function defined server side, is to define your functions in your local instance on startup and then you send the function calls over IPC, e.g.

system"l /path/to/myscript.q"; /make this load every time on startup

/to have your function executed on the server without it being defined on the server
h:hopen `:XXX.XXX.XX.XX:YYYY;
res:h(myfunc1;`abc);

This loads the functions in your local instance but sends the function to the remote server for evaluation, along with the input parameter `abc

Edit: Some common methods for "loading every time on startup" include:

  1. Loading a script from the startup command line, aka
q myscript.q -p 1234 -w 10000

You could have a master script which loads subscripts.

  1. Load a database or script directory contains scripts from the startup command line, aka
q /path/to/db -p 1234 -w 10000

Jeff Borror mentions this here: https://code.kx.com/q4m3/14_Introduction_to_Kdb%2B/#14623-scripts and here: https://code.kx.com/q4m3/14_Introduction_to_Kdb%2B/#14636-scripts

  1. Like you say, you can have a q.q script in your QHOME
terrylynch
  • 11,844
  • 13
  • 21
  • Thanks again Terry, this is helpful and I appreciate your experience with implementing. We are our admins so we have full control fortunately. Our server is currently the Dev server as we are rapidly developing and updating functions to process new data and I am testing them locally and pushing to our Dev server. At some point we will have a freeze to become our Prod environment. – Jason_L Apr 01 '21 at 19:32
  • 1
    Also, thanks for sharing the folder folder load method, I forgot about that one. I have been using the the list script of subscripts method but the folder idea .Again, we are fortunate to have whitespace to work with as we our the admins of our KDB instance......at least for now. – Jason_L Apr 01 '21 at 19:37
  • 1
    Got it. Then yes, you should load function scripts on startup of the instance using one of the methods above. For dev/testing you could of course (re)load scripts on the fly and/or use an IDE/editor to redefine functions on the fly. But end goal should be for the functions to stabilse into a script that gets loaded once. – terrylynch Apr 01 '21 at 20:15
  • One last follow up..... from an experienced point of view...is serialization used for anything other than data and tables (splayed tables too)? It piqued my interested to see if I could load an entire "family" of functions by simply loading their context name.....i.e. I had ~50-100 functions in the .AAA context name saved at startup I would simply load them by loading it using `.AAA set get `:/data/root to load all the functions in `.AAA context at once...or at least that is how I thought about leveraging it and got a simple example working. – Jason_L Apr 01 '21 at 21:19
  • Understood that the using \l ./root/functions.q in the q.q is the suggested method (and how I've been doing it to date.) Thank you again! – Jason_L Apr 01 '21 at 21:20
  • 2
    I don't fully understand the first part of your question but if you want to load `AAA.q` into the `.AAA` namespace, at the start of the q file you can have `\d .AAA`. All function defined after this will be `.AAA.func` e.g. `f:{x+y}` will be `.AAA.f` – Matt Moore Apr 01 '21 at 22:07
  • 1
    Agree with Matt here - functions can be separated into namespaces within a script using `\d` context switching. I would strongly advise against your method of serialising a function space and loading it back it - this gives you zero visibility into the functions as it won't be human-readable in the unloaded state, it gives you no ability to manage and track code changes (e.g. via git), it gives you no way to search through code (.e.g if you wanted to search for all functions which use a particular table). Using scripts solves all of these problems – terrylynch Apr 02 '21 at 09:28
  • Thank you both again Matt and Terry. Yes, didn't see much value in reading or opening the serialized file outside of KDB. Regarding the namespace, I was testing a batch of functions under their own context for example `.aaa.func1:{x+y}` `aaa.func2:{x*y}` to group functions by datatype...ie data source 1 uses .aaa functions, data source 2 uses .bbb and so on. key `.aaa would display all the associated functions for each context. Not sure I see the value of this based on this discussion other than organization so appreciate the input. – Jason_L Apr 02 '21 at 13:38
  • I'll continue to stick with loading the folders at runtime using the q.q file. Thanks again! – Jason_L Apr 02 '21 at 13:47