I have written one method which creates an EventHubProducerClient using the credentials from DefaultAzureCredential. In DefaultAzureCredential, I am using options DefaultAzureCredentialResourceIdOptions with managed identity resource id. My code runs on Azure VMSS and there are many independent requests which try to create the event hub producer client and send the data to event hub. Some time(very less amount) I see the error in method "producerClient.sendBatch(batch)" as CredentialUnavailableError. But after retry, that error goes away.
I suspect that there is some limit which prevents from DDOS and due to which the request to get the token using DefaultAzureCredential is getting throttled, but I am not sure.
How can I write code more effectively to not get this error?
One more type of Messaging error I saw is - The request with message_id "b89ad345-b6eb-d24c-b245-49b5ad99fab5" to "$cbs" endpoint timed out. Please try again later.
This one also goes away on retry inside my code. How to fix this so that it wont come in future?