0

We were trying to do a bench test for our cadence setup (currently running on 2 EC2 instances: m5a.xlarge) Have observed a lot of cadence client errors:

cadence_service:cadence_matching,operation:historyclientrecordactivitytaskstarted

As of now we haven't added any dynamic config. Please let s know if you have any idea what might be the issue/ any dynamic config needs to be added regarding this/ how we can debug further enter image description here

Alessandro Togni
  • 680
  • 1
  • 9
  • 24
Chandrakanth Ganji
  • 11
  • 3

1 Answers1

0

This metric just tells what API failed on client side. You want to check the persistence error metrics and group by operation. Then you can check what errors you are getting from the server side. In M3 we are using a query like following:

fetch name:persistence_errors 
   | removeEmpty 
   | transformNull 0 
   | sum operation 
   | scaleToSeconds 1 
   | aliasByTags operation

Or better, you can search server logs (most probably they will show up in history or in frontend service) to get more detail. This should be the quickest way.

Ender
  • 176
  • 1
  • 6