How to perform large batch operation in cosmos DB

Question

We are building an analytic engine for our clients. We have choose Cosmos DB. The reason for this is to store unstructured data. For the moment for one client we have around 300000 records.

Sample analytic item

{
    "adjustments": [],
    "payments": [],
    "amountReceivable": 0,
    "amountReceivablePatient": 0,
    "amountReceivableInsurance": 0,
    "id": "1_10",
    "clientID": 1,
    "chargeID": "10",
    "encounterID": "29310660",
    "patientID": "232362",
    "provider": "eeee",
    "financialClass": "kkk",
    "insurancePrimary": "jjj",
    "dateOfServiceBegin": "2019-08-19T18:30:00Z",
    "location": "test location","chargeTotal": 272,
    "chargeInsurancePrimary": 272,
    "chargePatient": 0
}

To show cumulative details in the dashboards we have created another collection called summary. This will hold the monthly and yearly pre calculated data. This is the collection we are going to use for our client application.

sample summary collection item

{
    "id": "630b20678a4745c4881b4d5cf07bcf08",
    "clientId": 1,
    "location": "test location",
    "provider": "eee",
    "periodicity": "monthly",
    "date": "2019-10-01T00:00:00",
    "financialClass": "kkk",
    "carrier": "jjj",
    "totalChargeByDoE": 488,
    "totalChargeInsuranceByDoE": 488,
    "totalChargePatientByDoE": 0,
    "totalChargeByDoS": 488,
    "totalChargeInsuranceByDoS": 488,
    "totalChargePatientByDoS": 0,
    "totalAdjustmentByDoE": 407,
    "totalAdjustmentInsuranceByDoE": 407,
}

The problem we are facing for the moment is calculating the cumulative records for the first time.Currently we are getting all the data from analytic collection and populating a list from that. Then calculating the monthly and yearly wise and then inserting to summary container. This is very slow and consuming more RUs. This issue is only for the existing data. We have implemented the change feed processor to handle real time data. Is there any way to reduce this process time and RUs? Thanks in advance.

Hello. Please provide a sample of what you do in your .net backend? Which version of cosmos SDK are you using? — Kiril1512, Mar 06 '20 at 10:24

score 0 · Accepted Answer · answered Mar 23 '20 at 10:03

Based on your description,you want to calculate history data to implement Group By month or year feature and upsert into summary collection,if i don't misunderstanding.

You could use Group By which is supported by .net sdk and js sdk and Aggregate feature like SUM in sql query.No need to load all data and calculate by yourself.However,if your calculate logic is complex, it seems you can't avoid it.

In addition,when you are ready for the summary collection to add data,you could consider using bulk executor which is more efficient. Or you could save the data and generate json file so that you could import data with migration tool.

Thanks. yes currently i am calculating the summary and generating the json. then i am using the migration tool to push to summary collection. This process i need to do only onetime so this is ok for me. — Niroj, Mar 23 '20 at 15:27

How to perform large batch operation in cosmos DB

1 Answers1