2

Seeing a flood of Object reference not set to an instance of an object exceptions when client tries to query cosmos.

Upon checking deeper the exception message has details like below:

{
   "CPU Load History":{
      "CPU History":"(2021-10-06T19:01:18.6834408Z 82.390), (2021-10-06T19:01:28.5562209Z 89.873), (2021-10-06T19:01:42.1291501Z 93.642), (2021-10-06T19:01:52.6528941Z 90.542), (2021-10-06T19:02:02.7588327Z 90.015), (2021-10-06T19:02:16.2446310Z 91.149)"
   }
}

"IsClientCpuOverloaded":true

"TransportException":"A client transport error occurred: The request timed out while waiting for a server response. (Time: 2021-10-06T19:03:07.1867280Z, error code: ReceiveTimeout [0x0010]

Exception while executing function: ParseData Object reference not set to an instance of an object.{"name":"Typed FeedIterator ReadNextAsync","id":"xxx-xx-xx-x-xxxx","caller info":{"member":"OperationHelperWithRootTraceAsync","file":"ClientContextCore.cs","line":219},"start time":"07:02:25:448","duration in milliseconds":52503.1979,"data":{"Client Configuration":{"Client Created Time Utc":"2021-10-06T18:14:08.4529348Z","NumberOfClientsCreated":1,"User Agent":"cosmos-netstandard-sdk/3.19.0|3.19.1|02|X64|Microsoft Windows xx.x.xxxx|.NET Core 3.1.18|N|F 00000001|","ConnectionConfig":{"gw":"(cps:50, urto:10, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: False, pr: ReuseUnicastPort)","other":"(ed:False, be:True)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[])"}},"children":[{"name":"MoveNextAsync","id":"xxxx-xx-x-fs-xxxx","caller info":{"member":"MoveNextAsync","file":"CrossPartitionRangePageAsyncEnumerator.cs","line":113}

{"ResponseTimeUTC":"2021-10-06T19:03:07.1875054Z","ResourceType":"Document","OperationType":"Query","LocationEndpoint":"xxxx.azure.com","StoreResult":{"ActivityId":"xxxx-xx-xx-x-xx","StatusCode":"Gone","SubStatusCode":"Unknown","LSN":-1,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":null,"CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsClientCpuOverloaded":true,"IsValid":false,"StorePhysicalAddress":xxxxx/","RequestCharge":0,"BELatencyInMs":null,"TransportException":"A client transport error occurred: The request timed out while waiting for a server response. (Time: 2021-10-06T19:03:07.1867280Z, activity ID: xxx-xx-x-xx-xx, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: xxxx/, connection: xx.xx.xx.xx:63676 -> xx.xx.xx.xxx:14190, payload sent: True, CPU history: (2021-10-06T18:25:38.6098171Z 100.000), (2021-10-06T18:25:53.9808590Z 95.716), (2021-10-06T18:25:58.9272146Z 59.098), (2021-10-06T18:26:15.3087856Z 91.388), (2021-10-06T18:26:18.6269463Z 96.000), (2021-10-06T18:26:28.8584330Z 85.611), CPU count: 4)

José Pedro
  • 1,097
  • 3
  • 14
  • 24
Raghavi Ravi
  • 55
  • 12
  • 2
    FYI please be mindful of tags: `[cosmos]` is unrelated to Cosmos DB (and the tag description calls this out explicitly). – David Makogon Oct 06 '21 at 23:28

1 Answers1

1

This is an error fixed in 3.20.0: https://learn.microsoft.com/en-us/azure/cosmos-db/sql/sql-api-sdk-dotnet-standard#-3200---2021-06-21

Image of the change log entry with the fix

Please update to the latest SDK version to avoid it.

Having said that, this is happening during Service Unavailable errors, please follow https://learn.microsoft.com/en-us/azure/cosmos-db/troubleshoot-service-unavailable

From your details, it seems CPU is hot and you might be incurring in thread starvation: https://learn.microsoft.com/en-us/azure/cosmos-db/sql/troubleshoot-dot-net-sdk-request-timeout#high-cpu-utilization

Your snapshot of CPU is showing both problems:

Issues with CPU diagnostics

Matias Quaranta
  • 13,907
  • 1
  • 22
  • 47