2

i have a very big json that has a very small changes each second.

im using asp.net core 3.1 and signalr core. and clients are using browsers running pure js and jquery.

right now for every change im re sending all the json again. i know that its not a good way.

what i want to do is just sending the Changes to client and then patch the changes to the main JSON in javascript.

by google search i understood that its a JSON diff/patch or diff/Merge operation, but i did not found anything on how to implement this.

so, how can i achieve a diff/pach opration in signalr ?

J_m and j_m_lit maybe 100 keys.

the data struncture

[
   {
      "id":0,
      "stdid":3808003,
      "lid":10533,
      .
      .
      (40 root keys)
      .
      .
      "J_ps":null,
      "J_ms":null,
      "J_m":{
         "SA_H2 0":{
            "c":"SA_H2",
            "a":"0",
            "b":0,
            "r":"0",
            "w":0,
            "en":"2nd",
            "o":{
               "2:0":{
                  "c":"2:0",
                  "v":1.3,
                  "pv":0.0,
                  "b":0
               },
               "Tie":{
                  "c":"Tie",
                  "v":3.2,
                  "pv":0.0,
                  "b":0
               }
            }
         },....
       
      },
      "J_m_Lit":{
         "OE 0":{
            "c":"OE",
            "a":"0",
            "b":0,
            "r":"0",
            "w":0,
            "en":"Total ",
            "o":{
               "xxx":{
                  "c":"Odd",
                  "v":1.833,
                  "pv":0.0,
                  "b":0
               },
               "zzz":{
                  "c":"Even",
                  "v":1.833,
                  "pv":0.0,
                  "b":0
               }
            }
         },  ....

      },
    
      "J_fre":{
         "tq":[
            0,
            1
         ],
         "m1":[
            23,
            17
         ]
      },
     
   }
]
Mrjm
  • 121
  • 2
  • 10
  • "by google search i understood that its a JSON diff/patch or diff/Merge operation" no, it's a **data** diff/merge problem. The serialization format you use to transfer the data from the server to the client is irrelevant. To address your problem, can you give a little information about the data structure and the amount/structure of the changes? Like is this just a huge list of a fairly simple/flat model where occasionally some fields change? Or is it a highly nested model where there may be changes on any level that need to be transmitted. – Thomas Sep 29 '20 at 16:27
  • actually its a big array of objects with the max depth of 4.its not flat and the changes might be on any depth. i cant give u the exact data, but i will edit the question with the idea of data structure. – Mrjm Sep 29 '20 at 16:41
  • the most precise but also most cumbersome way would be to diff the values and then send something like `{id:0, "J_m.SA_H2 0.a":1,"J_m.SA_H2 0.b":2, "J_m_Lit.OE 0.o.xxx": {...}}` to the frontend. Depending on how extensive the changes are you could also entire blocks or even the entire row; still better than sending the entire list of rows. In the frontend it should be trivial to get the entry by id and then `for(let key in objectWithChanges){ const path = key.split("."), value = objectWithChanges[key]; ... }` traverse the path and set the value (wether it is a primitive value or an object) – Thomas Sep 29 '20 at 17:28

1 Answers1

2

Yes, I think you are on the right track.

JSON-Patch (RFC6902), at its core, tries to solve the very problem you facing here: partial updates. JSON Patch offers an atomic update request that patches your JSON using a series of "add", "remove", "replace", "move" and "copy" operations. To quote one of the authors of the JSON Patch RFC, Mark Nottingham (IETF):

This has a bunch of advantages. Since it’s a generic format, you can write server- and client-side code once, and share it among a number of applications. You can do QA once, and make sure you get it right. Furthermore, your API will become less complex, because it has less URI conventions, leading to more flexibility and making it easier to approach for new developers. Using this approach will also make caches operate more correctly; since modifications to a resource will “travel” through its URL, rather than some other one, the right stored responses will get invalidated.

ASP.NET Core offers JSON-Patch support. JSON Patch can be added by installing Microsoft JSON Patch library (Nuget) from the Package Manager console:

Install-Package Microsoft.AspNetCore.JsonPatch

Then you apply patches using the JsonPatchDocument class and its methods like ApplyTo, Add, Copy, Move, Remove, Replace.

In an ASP.NET Core Web API context, we can also make use of the Microsoft.AspNetCore.Mvc.NewtonsoftJson NuGet package.

And then use the PUT and PATCH methods to update an existing resource with JSON Patch operations.

This MSDN article has some good explanations and a sample project.

In a JavaScript/Node.js project, you could also choose from a larger list of JS implementations, notably fast-json-patch.

Resources:

Side note: Another option is JSON Merge Patch (RFC7396), which can be used to send partial updates as well but works more like an actual diff/patch that simply contains the changes instead of using mutating operations. In this respect, JSON Merge Patch is simpler but more limited than JSON Patch, e.g. you cannot set a key to null (since null means remove), merging only works with objects {..}, arrays [..] can only be replaced as a whole, and merging never fails, could cause side-effects and is therefore not transactional. This renders Merge Patch (overly) simplistic and impractical for certain real-world applications. There are more related projects named like JSON-Diffpatch, etc that take similar approaches. Arguably, JSON-Patch is more flexible but both have advantages and disadvantages.

Community
  • 1
  • 1
wp78de
  • 18,207
  • 7
  • 43
  • 71