0

I have a recurring problem that I have to track a data structure over time and pass it via GraphQL, for example, this could be a timeline of a list and its changes:

[]
[1]
[1 2]
[-1 1 2]
[-1 2]
[2]
[]

I'm looking for a library that allows me to describe this data structure and its changes and automatically find the right graphql representation to only publish the diffs?

For example it would break down the above structure into:

start empty
add 1
add 2
insert -1 in the beginning
etc.

in GraphQL terms.

Has somebody tackled this before?

Ideally ReasonML (front-end) and Python (back-end) but looking for any examples.

Peteris
  • 3,548
  • 4
  • 28
  • 44
  • any db, versionable if history required, subscriptions - you can publish insert/remove messages as diffs – xadm Apr 21 '20 at 10:16
  • Trying to avoid using a database for this purpose. It's a simple need of carrying an evolving persistent data structure over REST via graphql without having to publish all its copies. – Peteris Apr 21 '20 at 10:30
  • "persistent" for stateless server usually means using some storage (db/ file) between requests ... you can publish diffs (in memory state) using graphql subscriptions, any mutation (graphql change request) can be published as message to subsrcibers – xadm Apr 21 '20 at 10:39
  • More on persistent data structures here: https://en.wikipedia.org/wiki/Persistent_data_structure - not referring to storage persistance in this case. – Peteris Apr 21 '20 at 11:34
  • immutable.js ? mutation can result in a new object while subscription (published) message can contain only diff (but more like copy of mutation action and payload - like redux) ... to be used on clientside with immutable... ;) – xadm Apr 21 '20 at 11:38
  • Yep just like immutable.js. I'm looking for a way to automatically take these iterated data structures and build a graphql schema that shows them as inserts/removes/edits and then rebuilds on the other side using something like immutable.js. – Peteris Apr 21 '20 at 14:08

1 Answers1

1

You can use graphql subscriptions (see apollo docs for details).

At start query for current state. Both sides (server and client) will have equal data. Both can use the same lib, f.e. immutable.js

Client invokes subscription, starts to listen to 'onListChange' messages.

On server insertElement, removeElement, updateElement resolvers are doing the same:

  • processes mutation on backend state;
  • encodes insert (or remove or update) as actionType and required data (mutation arguments, newly created id, version number) as payload;
  • sends this ('onListChange') message to subscribers using publish.

Client:

  • receives a message;
  • decodes message and updates/mutates own state (processes the same mutation as on server);
  • updates local version number.

As the result client should have the same state.

Client can detect missing messages (version numbers passed within payload compared with local version indicator) and force [re-]query for current(entire) state.

Update

If server side changes:

  • are not driven by mutations (f.e. other app operates on the same db);
  • are not a stream of changes but simply new states (new state just arrives/is read from external source);

... - you don't have a chance to inject your logic into process applying change - you should have a running monitoring process to:

  • detect state changes;
  • convert state differences into set of 'patches' (f.e. using jiff);
  • publish patches as message to subscribers.

Client decodes (messages are a stream of) patches into mutations of local state (to be in sync with server state) as earlier. As above - apply changes if versions matches or requery entire state.

Graphql is only a communication channel (queries, messages) - it has nothing to processes (and data, on both sides) needed for this task.

xadm
  • 8,219
  • 3
  • 14
  • 25
  • Hi xadm, I don't think it has to be one query per diff. We have a lot of different state changes, so sending a new query for each is not feasible. Also I'm familiar with how to do this manually - and we are doing it right now, but looking for a way to *automatically* encode timelines of data structures into timelines of diffs. – Peteris Apr 21 '20 at 17:13
  • 1
    then this is not a graphql specific question ... it's a general programming/data structures//algorithm question ..... https://github.com/cujojs/jiff can be good? – xadm Apr 21 '20 at 17:44
  • I was actually looking for a GraphQL specific solution, because I didn't want to have to specify a custom schema for these variables. – Peteris Apr 22 '20 at 09:51
  • Thanks - I understand how to do the individual steps but was looking for an abstraction that handles this, may have to build this library myself. – Peteris Apr 22 '20 at 16:07